Merge branch '0.11'

Merge pull request #3133 from sylvestre/master
2021-12-02 18:56:19 +01:00 · 2021-11-03 16:01:00 +01:00 · 2021-10-25 12:10:28 +02:00 · 2021-10-23 17:06:13 +02:00 · 2021-10-17 19:53:10 +02:00 · 2021-09-19 18:54:23 +02:00
63 changed files with 971 additions and 285 deletions
@@ -0,0 +1 @@
+ChangeLog linguist-language=Markdown
@@ -1,49 +0,0 @@
-_We will be very grateful, if your problem was described as completely as possible,
-enclosing excerpts from logs (if possible within DEBUG mode, if no errors evident
-within INFO mode), and configuration in particular of effected relevant settings
-(e.g., with ` fail2ban-client -d | grep 'affected-jail-name' ` for a particular
-jail troubleshooting).
-Thank you in advance for the details, because such issues like "It does not work" 
-alone could not help to resolve anything!
-Thanks! (remove this paragraph and other comments upon reading)_
-
-### Environment:
-
-_Fill out and check (`[x]`) the boxes which apply. If your Fail2Ban version is outdated, 
-and you can't verify that the issue persists in the recent release, better seek support 
-from the distribution you obtained Fail2Ban from_
-
- Fail2Ban version (including any possible distribution suffixes):
- OS, including release name/version: 
- [ ] Fail2Ban installed via OS/distribution mechanisms
- [ ] You have not applied any additional foreign patches to the codebase
- [ ] Some customizations were done to the configuration (provide details below is so)
-
-### The issue:
-
-_Summary here_
-
-#### Steps to reproduce
-
-#### Expected behavior
-
-#### Observed behavior
-
-#### Any additional information
-
-### Configuration, dump and another helpful excerpts
-
-#### Any customizations done to /etc/fail2ban/ configuration
-```
-```
-
-#### Relevant parts of /var/log/fail2ban.log file:
-_preferably obtained while running fail2ban with `loglevel = 4`_
-
-```
-```
-
-#### Relevant lines from monitored log files in question:
-
-```
-```
@@ -0,0 +1,70 @@
+---
+name: Bug report
+about: Report a bug within the fail2ban engines (not filters or jails)
+title: '[BR]: '
+labels: bug
+assignees: ''
+
+---
+
+<!--
+  - Before reporting, please make sure to search the open and closed issues for any reports in the past.
+  - Use this issue template to report a bug in the fail2ban engine (not in a filter or jail).
+  - If you want to request a feature or a new filter, please use "Feature request" or "Filter request" instead.
+  - If you have rather some question, please open or join to some discussion.
+
+  We will be very grateful, if your problem was described as completely as possible,
+  enclosing excerpts from logs (if possible within DEBUG mode, if no errors evident
+  within INFO mode), and configuration in particular of effected relevant settings
+  (e.g., with ` fail2ban-client -d | grep 'affected-jail-name' ` for a particular
+  jail troubleshooting).
+  Thank you in advance for the details, because such issues like "It does not work" 
+  alone could not help to resolve anything!
+  Thanks! 
+  (you can remove this paragraph and other comments upon reading)
+-->
+
+### Environment:
+
+<!--
+  Fill out and check (`[x]`) the boxes which apply. If your Fail2Ban version is outdated, 
+  and you can't verify that the issue persists in the recent release, better seek support 
+  from the distribution you obtained Fail2Ban from
+-->
+
+- Fail2Ban version <!-- including any possible distribution suffixes --> : 
+- OS, including release name/version : 
+- [ ] Fail2Ban installed via OS/distribution mechanisms
+- [ ] You have not applied any additional foreign patches to the codebase
+- [ ] Some customizations were done to the configuration (provide details below is so)
+
+### The issue:
+
+<!-- summary here -->
+
+#### Steps to reproduce
+
+#### Expected behavior
+
+#### Observed behavior
+
+#### Any additional information
+
+
+### Configuration, dump and another helpful excerpts
+
+#### Any customizations done to /etc/fail2ban/ configuration
+<!-- put your configuration excerpts between next 2 lines -->
+```
+```
+
+#### Relevant parts of /var/log/fail2ban.log file:
+<!-- preferably obtained while running fail2ban with `loglevel = 4` -->
+<!-- put your log excerpt between next 2 lines -->
+```
+```
+
+#### Relevant lines from monitored log files:
+<!-- put your log excerpt between next 2 lines -->
+```
+```
@@ -0,0 +1,35 @@
+---
+name: Feature request
+about: Suggest an idea or an enhancement for this project
+title: '[RFE]: '
+labels: enhancement
+assignees: ''
+
+---
+
+<!--
+  - Before requesting, please make sure to search the open and closed issues for any requests in the past.
+  - Use this issue template to request a feature in the fail2ban engine (not a new filter or jail).
+  - If you want to request a new filter or failregex, please use "Filter request" instead.
+  - If you have rather some question, please open or join to some discussion.
+-->
+
+#### Feature request type
+<!--
+  Please provide a summary description of the feature request.
+-->
+
+#### Description
+<!--
+  Please describe the feature in more detail.
+-->
+
+#### Considered alternatives
+<!--
+  A clear and concise description of any alternative solutions or features you've considered.
+-->
+
+#### Any additional information
+<!--
+  Add any other context or screenshots about the feature request here.
+-->
@@ -0,0 +1,59 @@
+---
+name: Filter request
+about: Request a new jail or filter to be supported or existing filter extended with new failregex
+title: '[FR]: '
+labels: filter-request
+assignees: ''
+
+---
+
+<!--
+  - Before requesting, please make sure to search the open and closed issues for any requests in the past.
+  - Sometimes failregex have been already requested before but are not implemented yet due to various reasons.
+  - If there are no hits for your concerns, please proceed otherwise add a comment to the related issue (also if it is closed).
+  - If you want to request a new feature, please use "Feature request" instead.
+  - If you have rather some question, please open or join to some discussion.
+-->
+
+### Environment:
+
+<!--
+  Fill out and check (`[x]`) the boxes which apply.
+-->
+
+- Fail2Ban version <!-- including any possible distribution suffixes --> : 
+- OS, including release name/version : 
+
+#### Service, project or product which log or journal should be monitored
+
+- Name of filter or jail in Fail2Ban (if already exists) : 
+- Service, project or product name, including release name/version : 
+- Repository or URL (if known) : 
+- Service type : 
+- Ports and protocols the service is listening : 
+
+#### Log or journal information
+<!-- Delete unrelated group -->
+
+<!-- Log file -->
+
+- Log file name(s) : 
+
+<!-- Systemd journal -->
+
+- Journal identifier or unit name : 
+
+#### Any additional information
+
+
+### Relevant lines from monitored log files:
+
+#### failures in sense of fail2ban filter (fail2ban must match):
+<!-- put your log excerpt between next 2 lines -->
+```
+```
+
+#### legitimate messages (fail2ban should not consider as failures):
+<!-- put your log excerpt between next 2 lines -->
+```
+```
@@ -1,7 +1,8 @@
 Before submitting your PR, please review the following checklist:

 - [ ] **CHOOSE CORRECT BRANCH**: if filing a bugfix/enhancement
-      against 0.9.x series, choose `master` branch
+      against certain release version, choose `0.9`, `0.10` or `0.11` branch,
+      for dev-edition use `master` branch
 - [ ] **CONSIDER adding a unit test** if your PR resolves an issue
 - [ ] **LIST ISSUES** this PR resolves
 - [ ] **MAKE SURE** this PR doesn't break existing tests
@@ -6,6 +6,39 @@
 Fail2Ban: Changelog
 ===================

+ver. 1.0.1-dev-1 (20??/??/??) - development nightly edition
+-----------
+
+### Compatibility:
+* potential incompatibility by parsing of options of `backend`, `filter` and `action` parameters (if they
+  are partially incorrect), because fail2ban could throw an error now (doesn't silently bypass it anymore).
+* to v.0.11:
+  - due to change of `actioncheck` behavior (gh-488), some actions can be incompatible as regards
+    the invariant check, if `actionban` or `actionunban` would not throw an error (exit code 
+    different from 0) in case of unsane environment.
+
+### Fixes
+* readline fixed to consider interim new-line character as part of code point in multi-byte logs
+  (e. g. unicode encoding like utf-16be, utf-16le);
+* `action.d/ufw.conf`:
+  - fixed handling on IPv6 (using prepend, gh-2331, gh-3018)
+  - application names containing spaces can be used now (gh-656, gh-1532, gh-3018)
+* `filter.d/drupal-auth.conf` more strict regex, extended to match "Login attempt failed from" (gh-2742)
+
+### New Features and Enhancements
+* `actioncheck` behavior is changed now (gh-488), so invariant check as well as restore or repair
+   of sane environment (in case of recognized unsane state) would only occur on action errors (e. g.
+   if ban or unban operations are exiting with other code as 0)
+* better recognition of log rotation, better performance by reopen: avoid unnecessary seek to begin of file
+  (and hash calculation)
+* file filter reads only complete lines (ended with new-line) now, so waits for end of line (for its completion)
+* `action.d/ufw.conf` (gh-3018):
+  - new option `add` (default `prepend`), can be supplied as `insert 1` for ufw versions before v.0.36 (gh-2331, gh-3018)
+  - new options `kill-mode` and `kill` to drop established connections of intruder (see action for details, gh-3018)
+* `filter.d/nginx-http-auth.conf` - extended with parameter mode, so additionally to `auth` (or `normal`) 
+   mode `fallback` (or combined as `aggressive`) can find SSL errors while SSL handshaking, gh-2881
+
+
 ver. 0.11.2 (2020/11/23) - heal-the-world-with-security-tools
 -----------

@@ -189,7 +222,7 @@ Yes, Hrrrm...
 ### New Features
 * new replacement tags for failregex to match subnets in form of IP-addresses with CIDR mask (gh-2559):
  - `<CIDR>` - helper regex to match CIDR (simple integer form of net-mask);
-  - `<SUBNET>` - regex to match sub-net adresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional);
+  - `<SUBNET>` - regex to match sub-net addresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional);
 * grouped tags (`<ADDR>`, `<HOST>`, `<SUBNET>`) recognize IP addresses enclosed in square brackets
 * new failregex-flag tag `<F-MLFGAINED>` for failregex, signaled that the access to service was gained
  (ATM used similar to tag `<F-NOFAIL>`, but it does not add the log-line to matches, gh-2279)
@@ -278,6 +278,7 @@ to tune it.  fail2ban-regex -D ...  will present Debuggex URLs for the regexs
 and sample log files that you pass into it.

 In general use when using regex debuggers for generating fail2ban filters:
+
 * use regex from the ./fail2ban-regex output (to ensure all substitutions are
 done)
 * replace <HOST> with (?&.ipv4)
@@ -2,7 +2,7 @@
                        / _|__ _(_) |_  ) |__  __ _ _ _  
                       |  _/ _` | | |/ /| '_ \/ _` | ' \ 
                       |_| \__,_|_|_/___|_.__/\__,_|_||_|
-                       v0.11.0.dev1            20??/??/??
+                       v1.0.1.dev1            20??/??/??

 ## Fail2Ban: ban hosts that cause multiple authentication errors

@@ -34,6 +34,7 @@ this case, you should use that instead.**

 Required:
 - [Python2 >= 2.6 or Python >= 3.2](https://www.python.org) or [PyPy](https://pypy.org)
+- python-setuptools, python-distutils or python3-setuptools for installation from source

 Optional:
 - [pyinotify >= 0.8.3](https://github.com/seb-m/pyinotify), may require:
@@ -46,11 +47,11 @@ Optional:

 To install:

-    tar xvfj fail2ban-0.11.0.tar.bz2
-    cd fail2ban-0.11.0
+    tar xvfj fail2ban-1.0.1.tar.bz2
+    cd fail2ban-1.0.1
    sudo python setup.py install
   
-Alternatively, you can clone the source from GitHub to a directory of Your choice, and do the install from there. Pick the correct branch, for example, 0.11
+Alternatively, you can clone the source from GitHub to a directory of Your choice, and do the install from there. Pick the correct branch, for example, master or 0.11

    git clone https://github.com/fail2ban/fail2ban.git
    cd fail2ban
@@ -89,11 +90,11 @@ fail2ban(1) and jail.conf(5)  manpages for further references.
 Code status:
 ------------

-* travis-ci.org: [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=0.11)](https://travis-ci.org/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=0.10)](https://travis-ci.org/fail2ban/fail2ban?branch=0.10) (0.10 branch) 
+* travis-ci.org: [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=master)](https://travis-ci.org/fail2ban/fail2ban?branch=master) / [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=0.11)](https://travis-ci.org/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=0.10)](https://travis-ci.org/fail2ban/fail2ban?branch=0.10) (0.10 branch) 

-* coveralls.io: [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=0.11)](https://coveralls.io/github/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=0.10)](https://coveralls.io/github/fail2ban/fail2ban?branch=0.10) / (0.10 branch)
+* coveralls.io: [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=master)](https://coveralls.io/github/fail2ban/fail2ban?branch=master) / [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=0.11)](https://coveralls.io/github/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=0.10)](https://coveralls.io/github/fail2ban/fail2ban?branch=0.10) / (0.10 branch)

-* codecov.io: [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=0.11)](https://codecov.io/gh/fail2ban/fail2ban/branch/0.11) (0.11 branch) / [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=0.10)](https://codecov.io/gh/fail2ban/fail2ban/branch/0.10) (0.10 branch)
+* codecov.io: [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=master)](https://codecov.io/gh/fail2ban/fail2ban/branch/master) / [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=0.11)](https://codecov.io/gh/fail2ban/fail2ban/branch/0.11) (0.11 branch) / [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=0.10)](https://codecov.io/gh/fail2ban/fail2ban/branch/0.10) (0.10 branch)

 Contact:
 --------
@@ -18,21 +18,46 @@ before = firewallcmd-common.conf

 [Definition]

-actionstart = ipset create <ipmset> hash:ip timeout <default-ipsettime> <familyopt>
+actionstart = <ipstype_<ipsettype>/actionstart>
              firewall-cmd --direct --add-rule <family> filter <chain> 0 <actiontype> -m set --match-set <ipmset> src -j <blocktype>

-actionflush = ipset flush <ipmset>
+actionflush = <ipstype_<ipsettype>/actionflush>

 actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 <actiontype> -m set --match-set <ipmset> src -j <blocktype>
             <actionflush>
-             ipset destroy <ipmset>
+             <ipstype_<ipsettype>/actionstop>

-actionban = ipset add <ipmset> <ip> timeout <ipsettime> -exist
+actionban = <ipstype_<ipsettype>/actionban>

 # actionprolong = %(actionban)s

+actionunban = <ipstype_<ipsettype>/actionunban>
+
+[ipstype_ipset]
+
+actionstart = ipset create <ipmset> hash:ip timeout <default-ipsettime> <familyopt>
+
+actionflush = ipset flush <ipmset>
+
+actionstop = ipset destroy <ipmset>
+
+actionban = ipset add <ipmset> <ip> timeout <ipsettime> -exist
+
 actionunban = ipset del <ipmset> <ip> -exist

+[ipstype_firewalld]
+
+actionstart = firewall-cmd --direct --new-ipset=<ipmset> --type=hash:ip --option=timeout=<default-ipsettime> <firewalld_familyopt>
+
+# TODO: there doesn't seem to be an explicit way to invoke the ipset flush function using firewall-cmd
+actionflush = 
+
+actionstop = firewall-cmd --direct --delete-ipset=<ipmset>
+
+actionban = firewall-cmd --ipset=<ipmset> --add-entry=<ip>
+
+actionunban = firewall-cmd --ipset=<ipmset> --remove-entry=<ip>
+
 [Init]

 # Option:  chain
@@ -56,6 +81,12 @@ ipsettime = 0
 # banaction = %(known/banaction)s[ipsettime='<timeout-bantime>']
 timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0)

+# Option: ipsettype
+# Notes.: defines type of ipset used for match-set (firewalld or ipset)
+# Values: firewalld or ipset
+# Default: ipset
+ipsettype = ipset
+
 # Option: actiontype
 # Notes.: defines additions to the blocking rule
 # Values: leave empty to block all attempts from the host
@@ -75,14 +106,16 @@ multiport = -p <protocol> -m multiport --dports <port>

 ipmset = f2b-<name>
 familyopt =
+firewalld_familyopt =

 [Init?family=inet6]

 ipmset = f2b-<name>6
 familyopt = family inet6
+firewalld_familyopt = --option=family=inet6


 # DEV NOTES:
 #
-# Author: Edgar Hoch and Daniel Black
+# Author: Edgar Hoch, Daniel Black, Sergey Brester and Mihail Politaev
 # firewallcmd-new / iptables-ipset-proto6 combined for maximium goodness
@@ -13,16 +13,45 @@ actionstop =

 actioncheck = 

-actionban = [ -n "<application>" ] && app="app <application>"
-            ufw insert <insertpos> <blocktype> from <ip> to <destination> $app
+# ufw does "quickly process packets for which we already have a connection" in before.rules,
+# therefore all related sockets should be closed
+# actionban is using `ss` to do so, this only handles IPv4 and IPv6.

-actionunban = [ -n "<application>" ] && app="app <application>"
-              ufw delete <blocktype> from <ip> to <destination> $app
+actionban = if [ -n "<application>" ] && ufw app info "<application>"
+            then
+              ufw <add> <blocktype> from <ip> to <destination> app "<application>" comment "<comment>"
+            else
+              ufw <add> <blocktype> from <ip> to <destination> comment "<comment>"
+            fi
+            <kill>
+
+actionunban = if [ -n "<application>" ] && ufw app info "<application>"
+              then
+                ufw delete <blocktype> from <ip> to <destination> app "<application>"
+              else
+                ufw delete <blocktype> from <ip> to <destination>
+              fi
+
+# Option: kill-mode
+# Notes.: can be set to ss or conntrack (may be extended later with other modes) to immediately drop all connections from banned IP, default empty (no kill)
+# Example: banaction = ufw[kill-mode=ss]
+kill-mode =
+
+# intern conditional parameter used to provide killing mode after ban:
+_kill_ =
+_kill_ss = ss -K dst "[<ip>]"
+_kill_conntrack = conntrack -D -s "<ip>"
+
+# Option: kill
+# Notes.: can be used to specify custom killing feature, by default depending on option kill-mode
+# Examples: banaction = ufw[kill='ss -K "( sport = :http || sport = :https )" dst "[<ip>]"']
+#           banaction = ufw[kill='cutter "<ip>"']
+kill = <_kill_<kill-mode>>

 [Init]
-# Option: insertpos
-# Notes.:  The position number in the firewall list to insert the block rule
-insertpos = 1
+# Option: add
+# Notes.: can be set to "insert 1" to insert a rule at certain position (here 1):
+add = prepend

 # Option: blocktype
 # Notes.: reject or deny
@@ -36,6 +65,10 @@ destination = any
 # Notes.: application from sudo ufw app list
 application = 

+# Option: comment
+# Notes.: comment for rule added by fail2ban
+comment = by Fail2Ban after <failures> attempts against <name>
+
 # DEV NOTES:
 # 
 # Author: Guilhem Lettron
@@ -2,11 +2,11 @@

 [Definition]

-failregex = ^<HOST> .*Googlebot.*$
+failregex = ^\s*<HOST> \S+ \S+(?: \S+)?\s+\S+ "[A-Z]+ /\S* [^"]*" \d+ \d+ \"[^"]*\" "[^"]*\bGooglebot/[^"]*"

 ignoreregex =

-datepattern = ^[^\[]*\[({DATE})
+datepattern = ^[^\[]*(\[{DATE}\s*\])
              {^LN-BEG}

 # DEV Notes:
@@ -21,7 +21,7 @@ log_prefix= (?:NOTICE|SECURITY|WARNING)%(__pid_re)s:?(?:\[C-[\da-f]*\])?:? [^:]+
 prefregex = ^%(__prefix_line)s%(log_prefix)s <F-CONTENT>.+</F-CONTENT>$

 failregex = ^Registration from '[^']*' failed for '<HOST>(:\d+)?' - (?:Wrong password|Username/auth name mismatch|No matching peer found|Not a local domain|Device does not match ACL|Peer is not supposed to register|ACL error \(permit/deny\)|Not a local domain)$
-            ^Call from '[^']*' \(<HOST>:\d+\) to extension '[^']*' rejected because extension not found in context
+            ^Call from '[^']*' \((?:(?:TCP|UDP):)?<HOST>:\d+\) to extension '[^']*' rejected because extension not found in context
            ^(?:Host )?<HOST> (?:failed (?:to authenticate\b|MD5 authentication\b)|tried to authenticate with nonexistent user\b)
            ^No registration for peer '[^']*' \(from <HOST>\)$
            ^hacking attempt detected '<HOST>'$
@@ -14,7 +14,7 @@ before = common.conf

 [Definition]

-failregex = ^%(__prefix_line)s(https?:\/\/)([\da-z\.-]+)\.([a-z\.]{2,6})(\/[\w\.-]+)*\|\d{10}\|user\|<HOST>\|.+\|.+\|\d\|.*\|Login attempt failed for .+\.$
+failregex = ^%(__prefix_line)s(?:https?:\/\/)[^|]+\|[^|]+\|[^|]+\|<ADDR>\|(?:[^|]*\|)*Login attempt failed (?:for|from) <F-USER>[^|]+</F-USER>\.$

 ignoreregex =

@@ -0,0 +1,25 @@
+# Fail2Ban filter for Monitorix (HTTP built-in server)
+#
+
+[INCLUDES]
+
+before = common.conf
+
+[Definition]
+
+_daemon = monitorix-httpd
+
+# Option:  failregex
+# Notes.:  regex to match the password failures messages in the logfile. The
+#          host must be matched by a group named "host". The tag "<HOST>" can
+#          be used for standard IP/hostname matching and is only an alias for
+#          (?:::f{4,6}:)?(?P<host>\S+)
+# Values:  TEXT
+#
+failregex = ^(?:\s+-)?\s*(?:NOTEXIST|AUTHERR|NOTALLOWED) - <ADDR>\b
+
+# Option:  ignoreregex
+# Notes.:  regex to ignore. If this regex matches, the line is ignored.
+# Values:  TEXT
+#
+ignoreregex =
@@ -0,0 +1,15 @@
+# Fail2Ban filter for failed MSSQL Server authentication attempts
+
+[Definition]
+
+failregex = ^\s*Logon\s+Login failed for user '<F-USER>(?:[^']*|.*)</F-USER>'\. [^'\[]+\[CLIENT: <ADDR>\]$
+
+
+# DEV Notes:
+#	Tested with SQL Server 2019 on Ubuntu 18.04
+#
+# Example:
+#	2020-02-24 14:48:55.12 Logon       Login failed for user 'root'. Reason: Could not find a login matching the name provided. [CLIENT: 127.0.0.1]
+#
+# Author: Rüdiger Olschewsky
+#
@@ -0,0 +1,16 @@
+# Fail2Ban filter to match bad requests to nginx
+#
+
+[Definition]
+
+# The request often doesn't contain a method, only some encoded garbage
+# This will also match requests that are entirely empty
+failregex = ^<HOST> - \S+ \[\] "[^"]*" 400
+
+datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)?
+              ^[^\[]*\[({DATE})
+              {^LN-BEG}
+
+journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx
+
+# Author: Jan Przybylak
@@ -17,7 +17,9 @@ datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]
              ^[^\[]*\[({DATE})
              {^LN-BEG}

+journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx
+
 # DEV Notes:
 # Based on apache-botsearch filter
 # 
-# Author: Frantisek Sumsal
+# Author: Frantisek Sumsal
@@ -3,15 +3,32 @@

 [Definition]

+mode = normal

-failregex = ^ \[error\] \d+#\d+: \*\d+ user "(?:[^"]+|.*?)":? (?:password mismatch|was not found in "[^\"]*"), client: <HOST>, server: \S*, request: "\S+ \S+ HTTP/\d+\.\d+", host: "\S+"(?:, referrer: "\S+")?\s*$
+mdre-auth = ^\s*\[error\] \d+#\d+: \*\d+ user "(?:[^"]+|.*?)":? (?:password mismatch|was not found in "[^\"]*"), client: <HOST>, server: \S*, request: "\S+ \S+ HTTP/\d+\.\d+", host: "\S+"(?:, referrer: "\S+")?\s*$
+mdre-fallback = ^\s*\[crit\] \d+#\d+: \*\d+ SSL_do_handshake\(\) failed \(SSL: error:\S+(?: \S+){1,3} too (?:long|short)\)[^,]*, client: <HOST>
+
+mdre-normal = %(mdre-auth)s
+mdre-aggressive = %(mdre-auth)s
+                  %(mdre-fallback)s
+
+failregex = <mdre-<mode>>

 ignoreregex = 

 datepattern = {^LN-BEG}

+journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx
+
 # DEV NOTES:
+# mdre-auth:
 # Based on samples in https://github.com/fail2ban/fail2ban/pull/43/files
 # Extensive search of all nginx auth failures not done yet.
 # 
 # Author: Daniel Black
+
+# mdre-fallback:
+# Ban people checking for TLS_FALLBACK_SCSV repeatedly
+# https://stackoverflow.com/questions/28010492/nginx-critical-error-with-ssl-handshaking/28010608#28010608
+# Author: Stephan Orlowsky
+
@@ -44,3 +44,6 @@ failregex = ^\s*\[[a-z]+\] \d+#\d+: \*\d+ limiting requests, excess: [\d\.]+ by
 ignoreregex = 

 datepattern = {^LN-BEG}
+
+journalmatch = _SYSTEMD_UNIT=nginx.service + _COMM=nginx
+
@@ -22,10 +22,10 @@ _daemon = nsd
 #          (?:::f{4,6}:)?(?P<host>[\w\-.^_]+)
 # Values:  TEXT

-failregex =  ^%(__prefix_line)sinfo: ratelimit block .* query <HOST> TYPE255$
-             ^%(__prefix_line)sinfo: .* <HOST> refused, no acl matches\.$
+failregex =  ^%(__prefix_line)sinfo: ratelimit block .* query <ADDR> TYPE255$
+             ^%(__prefix_line)sinfo: .* from(?: client)? <ADDR> refused, no acl matches\.?$

 ignoreregex =

 datepattern = {^LN-BEG}Epoch
-              {^LN-BEG}
+              {^LN-BEG}
@@ -16,8 +16,11 @@ _pref = [A-Z]{4}

 prefregex = ^%(__prefix_line)s<mdpr-<mode>> <F-CONTENT>.+</F-CONTENT>$

+# Extended RE for normal mode to match reject by unknown users or undeliverable address, can be set to empty to avoid this:
+exre-user = |[Uu](?:ser unknown|ndeliverable address)
+
 mdpr-normal = (?:\w+: (?:milter-)?reject:|(?:improper command pipelining|too many errors) after \S+)
-mdre-normal=^%(_pref)s from [^[]*\[<HOST>\]%(_port)s: [45][50][04] [45]\.\d\.\d+ (?:(?:<[^>]*>)?: )?(?:(?:Helo command|(?:Sender|Recipient) address) rejected: )?(?:Service unavailable|User unknown|(?:Client host|Command|Data command) rejected|Relay access denied|(?:Host|Domain) not found|need fully-qualified hostname|match)\b
+mdre-normal=^%(_pref)s from [^[]*\[<HOST>\]%(_port)s: [45][50][04] [45]\.\d\.\d+ (?:(?:<[^>]*>)?: )?(?:(?:Helo command|(?:Sender|Recipient) address) rejected: )?(?:Service unavailable|(?:Client host|Command|Data command) rejected|Relay access denied|(?:Host|Domain) not found|need fully-qualified hostname|match%(exre-user)s)\b
            ^from [^[]*\[<HOST>\]%(_port)s:?

 mdpr-auth = warning:
@@ -33,7 +36,9 @@ mdre-rbl  = ^%(_pref)s from [^[]*\[<HOST>\]%(_port)s: [45]54 [45]\.7\.1 Service
 mdpr-more = %(mdpr-normal)s
 mdre-more = %(mdre-normal)s

-mdpr-ddos = (?:lost connection after(?! DATA) [A-Z]+|disconnect(?= from \S+(?: \S+=\d+)* auth=0/(?:[1-9]|\d\d+)))
+# Includes some of the log messages described in
+# <http://www.postfix.org/POSTSCREEN_README.html>.
+mdpr-ddos = (?:lost connection after(?! DATA) [A-Z]+|disconnect(?= from \S+(?: \S+=\d+)* auth=0/(?:[1-9]|\d\d+))|(?:PREGREET \d+|HANGUP) after \S+|COMMAND (?:TIME|COUNT|LENGTH) LIMIT)
 mdre-ddos = ^from [^[]*\[<HOST>\]%(_port)s:?

 mdpr-extra = (?:%(mdpr-auth)s|%(mdpr-normal)s)
@@ -0,0 +1,17 @@
+# Fail2Ban filter for port scans detected by scanlogd
+
+[INCLUDES]
+
+# Read common prefixes. If any customizations available -- read them from
+# common.local
+before = common.conf
+
+[Definition]
+
+_daemon = scanlogd
+
+failregex =  ^%(__prefix_line)s<ADDR>(?::<F-PORT/>)? to \S+ ports\b
+
+ignoreregex =
+
+# Author: Mike Gabriel <mike.gabriel@das-netzwerkteam.de>
@@ -5,17 +5,23 @@ before = apache-common.conf

 [Definition]

-# pattern: [Wed Apr 27 23:12:07.736196 2016] [:error] [pid 2460] [client 10.1.1.1:47296] WAR [Login denied for user "test"], referer: https://zoneminderurl/index.php
-#
+# patterns: [Mon Mar 28 16:50:49.522240 2016] [:error] [pid 1795] [client 10.1.1.1:50700] WAR [Login denied for user "username1"], referer: https://zoneminder/
+#           [Sun Mar 28 16:53:00.472693 2021] [php7:notice] [pid 11328] [client 10.1.1.1:39568] ERR [Could not retrieve user test details], referer: https://zm/
+#           [Sun Mar 28 16:59:14.150625 2021] [php7:notice] [pid 11336] [client 10.1.1.1:39654] ERR [Login denied for user "john"], referer: https://zm/
 #
 # Option:  failregex
-# Notes.:  regex to match the password failure messages in the logfile.
+# Notes.:  regex to match the login failure and non-existent user error messages in the logfile.

-failregex = ^%(_apache_error_client)s WAR \[Login denied for user "[^"]*"\]
+prefregex = ^%(_apache_error_client)s (?:ERR|WAR) <F-CONTENT>\[(?:Login denied|Could not retrieve).*</F-CONTENT>$
+
+failregex = ^\[Login denied for user "<F-USER>[^"]*</F-USER>"\]
+            ^\[Could not retrieve user <F-USER>\S*</F-USER>

 ignoreregex =

 # Notes:
-#	Tested on Zoneminder 1.29.0
+#	Tested on Zoneminder 1.29 and 1.35.21
+#
+#	Zoneminder versions > 1.3x use "ERR" and < 1.3x use "WAR" level logs, so i've kept both for compatibility reasons
 #
 # Author: John Marzella
@@ -67,7 +67,7 @@ before = paths-debian.conf
 # more aggressive example of formula has the same values only for factor "2.0 / 2.885385" :
 #bantime.formula = ban.Time * math.exp(float(ban.Count+1)*banFactor)/math.exp(1*banFactor)

-# "bantime.multipliers" used to calculate next value of ban time instead of formula, coresponding 
+# "bantime.multipliers" used to calculate next value of ban time instead of formula, corresponding
 # previously ban count and given "bantime.factor" (for multipliers default is 1);
 # following example grows ban time by 1, 2, 4, 8, 16 ... and if last ban count greater as multipliers count, 
 # always used last multiplier (64 in example), for factor '1' and original ban time 600 - 10.6 hours
@@ -77,7 +77,7 @@ before = paths-debian.conf
 #bantime.multipliers = 1 5 30 60 300 720 1440 2880

 # "bantime.overalljails" (if true) specifies the search of IP in the database will be executed 
-# cross over all jails, if false (dafault), only current jail of the ban IP will be searched
+# cross over all jails, if false (default), only current jail of the ban IP will be searched
 #bantime.overalljails = false

 # --------------------
@@ -370,8 +370,11 @@ banaction = %(banaction_allports)s
 logpath = /opt/openhab/logs/request.log


+# To use more aggressive http-auth modes set filter parameter "mode" in jail.local:
+# normal (default), aggressive (combines all), auth or fallback
+# See "tests/files/logs/nginx-http-auth" or "filter.d/nginx-http-auth.conf" for usage example and details.
 [nginx-http-auth]
-
+# mode = normal
 port    = http,https
 logpath = %(nginx_error_log)s

@@ -387,8 +390,10 @@ logpath = %(nginx_error_log)s

 port     = http,https
 logpath  = %(nginx_error_log)s
-maxretry = 2

+[nginx-bad-request]
+port    = http,https
+logpath = %(nginx_access_log)s

 # Ban attackers that try to use PHP's URL-fopen() functionality
 # through GET/POST variables. - Experimental, with more than a year
@@ -792,6 +797,14 @@ logpath  = %(mysql_log)s
 backend  = %(mysql_backend)s


+[mssql-auth]
+# Default configuration for Microsoft SQL Server for Linux
+# See the 'mssql-conf' manpage how to change logpath or port
+logpath = /var/opt/mssql/log/errorlog
+port = 1433
+filter = mssql-auth
+
+
 # Log wrong MongoDB auth (for details see filter 'filter.d/mongodb-auth.conf')
 [mongodb-auth]
 # change port when running with "--shardsvr" or "--configsvr" runtime operation
@@ -957,3 +970,11 @@ logpath = %(apache_error_log)s
 # see `filter.d/traefik-auth.conf` for details and service example.
 port    = http,https
 logpath = /var/log/traefik/access.log
+
+[scanlogd]
+logpath = %(syslog_local0)s
+banaction = %(banaction_allports)s
+
+[monitorix]
+port	= 8080
+logpath = /var/log/monitorix-httpd
@@ -230,7 +230,7 @@ class Fail2banClient(Fail2banCmdLine, Thread):
 			logSys.log(5, '  client phase %s', phase)
 		if not stream:
 			return False
-		# wait a litle bit for phase "start-ready" before enter active waiting:
+		# wait a little bit for phase "start-ready" before enter active waiting:
 		if phase is not None:
 			Utils.wait_for(lambda: phase.get('start-ready', None) is not None, 0.5, 0.001)
 			phase['configure'] = (True if stream else False)
@@ -289,9 +289,6 @@ class Fail2banRegex(object):
 	def output(self, line):
 		if not self._opts.out: output(line)

-	def decode_line(self, line):
-		return FileContainer.decode_line('<LOG>', self._encoding, line)
-
 	def encode_line(self, line):
 		return line.encode(self._encoding, 'ignore')

@@ -723,10 +720,6 @@ class Fail2banRegex(object):

 		return True

-	def file_lines_gen(self, hdlr):
-		for line in hdlr:
-			yield self.decode_line(line)
-
 	def start(self, args):

 		cmd_log, cmd_regex = args[:2]
@@ -745,10 +738,10 @@ class Fail2banRegex(object):

 		if os.path.isfile(cmd_log):
 			try:
-				hdlr = open(cmd_log, 'rb')
+				test_lines = FileContainer(cmd_log, self._encoding, doOpen=True)
+
 				self.output( "Use         log file : %s" % cmd_log )
 				self.output( "Use         encoding : %s" % self._encoding )
-				test_lines = self.file_lines_gen(hdlr)
 			except IOError as e: # pragma: no cover
 				output( e )
 				return False
@@ -134,7 +134,7 @@ protocol = [
 ["get <JAIL> ignoreregex", "gets the list of regular expressions which matches patterns to ignore for <JAIL>"],
 ["get <JAIL> findtime", "gets the time for which the filter will look back for failures for <JAIL>"],
 ["get <JAIL> bantime", "gets the time a host is banned for <JAIL>"],
-["get <JAIL> datepattern", "gets the patern used to match date/times for <JAIL>"],
+["get <JAIL> datepattern", "gets the pattern used to match date/times for <JAIL>"],
 ["get <JAIL> usedns", "gets the usedns setting for <JAIL>"],
 ["get <JAIL> banip [<SEP>|--with-time]", "gets the list of of banned IP addresses for <JAIL>. Optionally the separator character ('<SEP>', default is space) or the option '--with-time' (printing the times of ban) may be specified. The IPs are ordered by end of ban."],
 ["get <JAIL> maxretry", "gets the number of failures allowed for <JAIL>"],
@@ -410,7 +410,7 @@ class CommandAction(ActionBase):
 		cmd = self.replaceTag(tag, self._properties,
 			conditional=('family='+family if family else ''),
 			cache=self.__substCache)
-		if '<' not in cmd or not family: return cmd
+		if not family or '<' not in cmd: return cmd
 		# replace family as dynamic tags, important - don't cache, no recursion and auto-escape here:
 		cmd = self.replaceDynamicTags(cmd, {'family':family})
 		return cmd
@@ -977,31 +977,38 @@ class CommandAction(ActionBase):
 		except (KeyError, TypeError):
 			family = ''

-		# invariant check:
-		if self.actioncheck:
-			# don't repair/restore if unban (no matter):
-			def _beforeRepair():
-				if cmd == '<actionunban>' and not self._properties.get('actionrepair_on_unban'):
-					self._logSys.error("Invariant check failed. Unban is impossible.")
+		repcnt = 0
+		while True:
+
+			# got some error, do invariant check:
+			if repcnt and self.actioncheck:
+				# don't repair/restore if unban (no matter):
+				def _beforeRepair():
+					if cmd == '<actionunban>' and not self._properties.get('actionrepair_on_unban'):
+						self._logSys.error("Invariant check failed. Unban is impossible.")
+						return False
+					return True
+				# check and repair if broken:
+				ret = self._invariantCheck(family, _beforeRepair, forceStart=(cmd != '<actionunban>'))
+				# if not sane (and not restored) return:
+				if ret != 1:
 					return False
-				return True
-			# check and repair if broken:
-			ret = self._invariantCheck(family, _beforeRepair, forceStart=(cmd != '<actionunban>'))
-			# if not sane (and not restored) return:
-			if ret != 1:
-				return False

-		# Replace static fields
-		realCmd = self.replaceTag(cmd, self._properties, 
-			conditional=('family='+family if family else ''), cache=self.__substCache)
+			# Replace static fields
+			realCmd = self.replaceTag(cmd, self._properties, 
+				conditional=('family='+family if family else ''), cache=self.__substCache)

-		# Replace dynamical tags, important - don't cache, no recursion and auto-escape here
-		if aInfo is not None:
-			realCmd = self.replaceDynamicTags(realCmd, aInfo)
-		else:
-			realCmd = cmd
+			# Replace dynamical tags, important - don't cache, no recursion and auto-escape here
+			if aInfo is not None:
+				realCmd = self.replaceDynamicTags(realCmd, aInfo)
+			else:
+				realCmd = cmd

-		return self.executeCmd(realCmd, self.timeout)
+			# try execute command:
+			ret = self.executeCmd(realCmd, self.timeout)
+			repcnt += 1
+			if ret or repcnt > 1:
+				return ret

 	@staticmethod
 	def executeCmd(realCmd, timeout=60, **kwargs):
@@ -84,7 +84,7 @@ class Actions(JailThread, Mapping):
 		self._jail = jail
 		self._actions = OrderedDict()
 		## The ban manager.
-		self.__banManager = BanManager()
+		self.banManager = BanManager()
 		self.banEpoch = 0
 		self.__lastConsistencyCheckTM = 0
 		## Precedence of ban (over unban), so max number of tickets banned (to call an unban check):
@@ -203,7 +203,7 @@ class Actions(JailThread, Mapping):
 	
 	def setBanTime(self, value):
 		value = MyTime.str2seconds(value)
-		self.__banManager.setBanTime(value)
+		self.banManager.setBanTime(value)
 		logSys.info("  banTime: %s" % value)
 	
 	##
@@ -212,10 +212,10 @@ class Actions(JailThread, Mapping):
 	# @return the time
 	
 	def getBanTime(self):
-		return self.__banManager.getBanTime()
+		return self.banManager.getBanTime()

 	def getBanned(self, ids):
-		lst = self.__banManager.getBanList()
+		lst = self.banManager.getBanList()
 		if not ids:
 			return lst
 		if len(ids) == 1:
@@ -230,7 +230,7 @@ class Actions(JailThread, Mapping):
 		list
 			The list of banned IP addresses.
 		"""
-		return self.__banManager.getBanList(ordered=True, withTime=withTime)
+		return self.banManager.getBanList(ordered=True, withTime=withTime)

 	def addBannedIP(self, ip):
 		"""Ban an IP or list of IPs."""
@@ -282,7 +282,7 @@ class Actions(JailThread, Mapping):
 		if db and self._jail.database is not None:
 			self._jail.database.delBan(self._jail, ip)
 		# Find the ticket with the IP.
-		ticket = self.__banManager.getTicketByID(ip)
+		ticket = self.banManager.getTicketByID(ip)
 		if ticket is not None:
 			# Unban the IP.
 			self.__unBan(ticket)
@@ -291,7 +291,7 @@ class Actions(JailThread, Mapping):
 			if not isinstance(ip, IPAddr):
 				ipa = IPAddr(ip)
 				if not ipa.isSingle: # subnet (mask/cidr) or raw (may be dns/hostname):
-					ips = filter(ipa.contains, self.__banManager.getBanList())
+					ips = filter(ipa.contains, self.banManager.getBanList())
 					if ips:
 						return self.removeBannedIP(ips, db, ifexists)
 			# not found:
@@ -350,7 +350,7 @@ class Actions(JailThread, Mapping):
 					continue
 				# wait for ban (stop if gets inactive, pending ban or unban):
 				bancnt = 0
-				wt = min(self.sleeptime, self.__banManager._nextUnbanTime - MyTime.time())
+				wt = min(self.sleeptime, self.banManager._nextUnbanTime - MyTime.time())
 				logSys.log(5, "Actions: wait for pending tickets %s (default %s)", wt, self.sleeptime)
 				if Utils.wait_for(lambda: not self.active or self._jail.hasFailTickets, wt):
 					bancnt = self.__checkBan()
@@ -397,7 +397,12 @@ class Actions(JailThread, Mapping):
 			"ipfailures":			lambda self: self._mi4ip(True).getAttempt(),
 			"ipjailfailures":	lambda self: self._mi4ip().getAttempt(),
 			# raw ticket info:
-			"raw-ticket":			lambda self: repr(self.__ticket)
+			"raw-ticket":			lambda self: repr(self.__ticket),
+			# jail info:
+			"jail.banned":       lambda self: self.__jail.actions.banManager.size(),
+			"jail.banned_total": lambda self: self.__jail.actions.banManager.getBanTotal(),
+			"jail.found":        lambda self: self.__jail.filter.failManager.size(),
+			"jail.found_total":  lambda self: self.__jail.filter.failManager.getFailTotal()
 		}

 		__slots__ = CallingMap.__slots__ + ('__ticket', '__jail', '__mi4ip')
@@ -494,11 +499,11 @@ class Actions(JailThread, Mapping):
 		for ticket in tickets:

 			bTicket = BanTicket.wrap(ticket)
-			btime = ticket.getBanTime(self.__banManager.getBanTime())
+			btime = ticket.getBanTime(self.banManager.getBanTime())
 			ip = bTicket.getIP()
 			aInfo = self._getActionInfo(bTicket)
 			reason = {}
-			if self.__banManager.addBanTicket(bTicket, reason=reason):
+			if self.banManager.addBanTicket(bTicket, reason=reason):
 				cnt += 1
 				# report ticket to observer, to check time should be increased and hereafter observer writes ban to database (asynchronous)
 				if Observers.Main is not None and not bTicket.restored:
@@ -557,7 +562,7 @@ class Actions(JailThread, Mapping):
 			# and increase ticket time if "bantime.increment" set)
 		if cnt:
 			logSys.debug("Banned %s / %s, %s ticket(s) in %r", cnt, 
-				self.__banManager.getBanTotal(), self.__banManager.size(), self._jail.name)
+				self.banManager.getBanTotal(), self.banManager.size(), self._jail.name)
 		return cnt

 	def __reBan(self, ticket, actions=None, log=True):
@@ -597,7 +602,7 @@ class Actions(JailThread, Mapping):
 	def _prolongBan(self, ticket):
 		# prevent to prolong ticket that was removed in-between,
 		# if it in ban list - ban time already prolonged (and it stays there):
-		if not self.__banManager._inBanList(ticket): return
+		if not self.banManager._inBanList(ticket): return
 		# do actions :
 		aInfo = None
 		for name, action in self._actions.iteritems():
@@ -622,13 +627,13 @@ class Actions(JailThread, Mapping):

 		Unban IP addresses which are outdated.
 		"""
-		lst = self.__banManager.unBanList(MyTime.time(), maxCount)
+		lst = self.banManager.unBanList(MyTime.time(), maxCount)
 		for ticket in lst:
 			self.__unBan(ticket)
 		cnt = len(lst)
 		if cnt:
 			logSys.debug("Unbanned %s, %s ticket(s) in %r", 
-				cnt, self.__banManager.size(), self._jail.name)
+				cnt, self.banManager.size(), self._jail.name)
 		return cnt

 	def __flushBan(self, db=False, actions=None, stop=False):
@@ -642,10 +647,10 @@ class Actions(JailThread, Mapping):
 		log = True
 		if actions is None:
 			logSys.debug("  Flush ban list")
-			lst = self.__banManager.flushBanList()
+			lst = self.banManager.flushBanList()
 		else:
 			log = False # don't log "[jail] Unban ..." if removing actions only.
-			lst = iter(self.__banManager)
+			lst = iter(self.banManager)
 		cnt = 0
 		# first we'll execute flush for actions supporting this operation:
 		unbactions = {}
@@ -682,7 +687,7 @@ class Actions(JailThread, Mapping):
 			self.__unBan(ticket, actions=actions, log=log)
 			cnt += 1
 		logSys.debug("  Unbanned %s, %s ticket(s) in %r", 
-			cnt, self.__banManager.size(), self._jail.name)
+			cnt, self.banManager.size(), self._jail.name)
 		return cnt

 	def __unBan(self, ticket, actions=None, log=True):
@@ -725,18 +730,18 @@ class Actions(JailThread, Mapping):
 			logSys.warning("Unsupported extended jail status flavor %r. Supported: %s" % (flavor, supported_flavors))
 		# Always print this information (basic)
 		if flavor != "short":
-			banned = self.__banManager.getBanList()
+			banned = self.banManager.getBanList()
 			cnt = len(banned)
 		else:
-			cnt = self.__banManager.size()
+			cnt = self.banManager.size()
 		ret = [("Currently banned", cnt),
-			   ("Total banned", self.__banManager.getBanTotal())]
+			   ("Total banned", self.banManager.getBanTotal())]
 		if flavor != "short":
 			ret += [("Banned IP list", banned)]
 		if flavor == "cymru":
-			cymru_info = self.__banManager.getBanListExtendedCymruInfo()
+			cymru_info = self.banManager.getBanListExtendedCymruInfo()
 			ret += \
-				[("Banned ASN list", self.__banManager.geBanListExtendedASN(cymru_info)),
-				 ("Banned Country list", self.__banManager.geBanListExtendedCountry(cymru_info)),
-				 ("Banned RIR list", self.__banManager.geBanListExtendedRIR(cymru_info))]
+				[("Banned ASN list", self.banManager.geBanListExtendedASN(cymru_info)),
+				 ("Banned Country list", self.banManager.geBanListExtendedCountry(cymru_info)),
+				 ("Banned RIR list", self.banManager.geBanListExtendedRIR(cymru_info))]
 		return ret
@@ -502,7 +502,7 @@ class Fail2BanDb(object):
 		except TypeError:
 			firstLineMD5 = None

-		if not firstLineMD5 and (pos or md5):
+		if firstLineMD5 is None and (pos or md5 is not None):
 			cur.execute(
 					"INSERT OR REPLACE INTO logs(jail, path, firstlinemd5, lastfilepos) "
 						"VALUES(?, ?, ?, ?)", (jail.name, name, md5, pos))
@@ -1140,14 +1140,14 @@ class FileFilter(Filter):
 				while not self.idle:
 					line = log.readline()
 					if not self.active: break; # jail has been stopped
-					if not line:
+					if line is None:
 						# The jail reached the bottom, simply set in operation for this log
 						# (since we are first time at end of file, growing is only possible after modifications):
 						log.inOperation = True
 						break
 					# acquire in operation from log and process:
 					self.inOperation = inOperation if inOperation is not None else log.inOperation
-					self.processLineAndAdd(line.rstrip('\r\n'))
+					self.processLineAndAdd(line)
 		finally:
 			log.close()
 		if self.jail.database is not None:
@@ -1170,6 +1170,8 @@ class FileFilter(Filter):
 		if logSys.getEffectiveLevel() <= logging.DEBUG:
 			logSys.debug("Seek to find time %s (%s), file size %s", date, 
 				MyTime.time2str(date), fs)
+		if not fs:
+			return
 		minp = container.getPos()
 		maxp = fs
 		tryPos = minp
@@ -1193,8 +1195,8 @@ class FileFilter(Filter):
 			dateTimeMatch = None
 			nextp = None
 			while True:
-				line = container.readline()
-				if not line:
+				line = container.readline(False)
+				if line is None:
 					break
 				(timeMatch, template) = self.dateDetector.matchTime(line)
 				if timeMatch:
@@ -1312,25 +1314,34 @@ except ImportError: # pragma: no cover

 class FileContainer:

-	def __init__(self, filename, encoding, tail=False):
+	def __init__(self, filename, encoding, tail=False, doOpen=False):
 		self.__filename = filename
+		self.waitForLineEnd = True
 		self.setEncoding(encoding)
 		self.__tail = tail
 		self.__handler = None
+		self.__pos = 0
+		self.__pos4hash = 0
+		self.__hash = ''
+		self.__hashNextTime = time.time() + 30
 		# Try to open the file. Raises an exception if an error occurred.
 		handler = open(filename, 'rb')
-		stats = os.fstat(handler.fileno())
-		self.__ino = stats.st_ino
+		if doOpen: # fail2ban-regex only (don't need to reopen it and check for rotation)
+			self.__handler = handler
+			return
 		try:
-			firstLine = handler.readline()
-			# Computes the MD5 of the first line.
-			self.__hash = md5sum(firstLine).hexdigest()
-			# Start at the beginning of file if tail mode is off.
-			if tail:
-				handler.seek(0, 2)
-				self.__pos = handler.tell()
-			else:
-				self.__pos = 0
+			stats = os.fstat(handler.fileno())
+			self.__ino = stats.st_ino
+			if stats.st_size:
+				firstLine = handler.readline()
+				# first line available and contains new-line:
+				if firstLine != firstLine.rstrip(b'\r\n'):
+					# Computes the MD5 of the first line.
+					self.__hash = md5sum(firstLine).hexdigest()
+				# if tail mode scroll to the end of file
+				if tail:
+					handler.seek(0, 2)
+					self.__pos = handler.tell()
 		finally:
 			handler.close()
 		## shows that log is in operation mode (expecting new messages only from here):
@@ -1349,6 +1360,10 @@ class FileContainer:
 		return self.__filename

 	def getFileSize(self):
+		h = self.__handler
+		if h is not None:
+			stats = os.fstat(h.fileno())
+			return stats.st_size
 		return os.path.getsize(self.__filename);

 	def setEncoding(self, encoding):
@@ -1367,38 +1382,54 @@ class FileContainer:
 	def setPos(self, value):
 		self.__pos = value

-	def open(self):
-		self.__handler = open(self.__filename, 'rb')
-		# Set the file descriptor to be FD_CLOEXEC
-		fd = self.__handler.fileno()
-		flags = fcntl.fcntl(fd, fcntl.F_GETFD)
-		fcntl.fcntl(fd, fcntl.F_SETFD, flags | fcntl.FD_CLOEXEC)
-		# Stat the file before even attempting to read it
-		stats = os.fstat(self.__handler.fileno())
-		if not stats.st_size:
-			# yoh: so it is still an empty file -- nothing should be
-			#      read from it yet
-			# print "D: no content -- return"
-			return False
-		firstLine = self.__handler.readline()
-		# Computes the MD5 of the first line.
-		myHash = md5sum(firstLine).hexdigest()
-		## print "D: fn=%s hashes=%s/%s inos=%s/%s pos=%s rotate=%s" % (
-		## 	self.__filename, self.__hash, myHash, stats.st_ino, self.__ino, self.__pos,
-		## 	self.__hash != myHash or self.__ino != stats.st_ino)
-		## sys.stdout.flush()
-		# Compare hash and inode
-		if self.__hash != myHash or self.__ino != stats.st_ino:
-			logSys.log(logging.MSG, "Log rotation detected for %s", self.__filename)
-			self.__hash = myHash
-			self.__ino = stats.st_ino
-			self.__pos = 0
-		# Sets the file pointer to the last position.
-		self.__handler.seek(self.__pos)
+	def open(self, forcePos=None):
+		h = open(self.__filename, 'rb')
+		try:
+			# Set the file descriptor to be FD_CLOEXEC
+			fd = h.fileno()
+			flags = fcntl.fcntl(fd, fcntl.F_GETFD)
+			fcntl.fcntl(fd, fcntl.F_SETFD, flags | fcntl.FD_CLOEXEC)
+			myHash = self.__hash
+			# Stat the file before even attempting to read it
+			stats = os.fstat(h.fileno())
+			rotflg = stats.st_size < self.__pos or stats.st_ino != self.__ino
+			if rotflg or not len(myHash) or time.time() > self.__hashNextTime:
+				myHash = ''
+				firstLine = h.readline()
+				# Computes the MD5 of the first line (if it is complete)
+				if firstLine != firstLine.rstrip(b'\r\n'):
+					myHash = md5sum(firstLine).hexdigest()
+					self.__hashNextTime = time.time() + 30
+			elif stats.st_size == self.__pos:
+				myHash = self.__hash
+			# Compare size, hash and inode
+			if rotflg or myHash != self.__hash:
+				if self.__hash != '':
+					logSys.log(logging.MSG, "Log rotation detected for %s, reason: %r", self.__filename,
+						(stats.st_size, self.__pos, stats.st_ino, self.__ino, myHash, self.__hash))
+					self.__ino = stats.st_ino
+					self.__pos = 0
+				self.__hash = myHash
+			# if nothing to read from file yet (empty or no new data):
+			if forcePos is not None:
+				self.__pos = forcePos
+			elif stats.st_size <= self.__pos:
+				return False
+			# Sets the file pointer to the last position.
+			h.seek(self.__pos)
+			# leave file open (to read content):
+			self.__handler = h; h = None
+		finally:
+			# close (no content or error only)
+			if h:
+				h.close(); h = None
 		return True

 	def seek(self, offs, endLine=True):
 		h = self.__handler
+		if h is None:
+			self.open(offs)
+			h = self.__handler
 		# seek to given position
 		h.seek(offs, 0)
 		# goto end of next line
@@ -1416,6 +1447,9 @@ class FileContainer:
 		try:
 			return line.decode(enc, 'strict')
 		except (UnicodeDecodeError, UnicodeEncodeError) as e:
+			# avoid warning if got incomplete end of line (e. g. '\n' in "...[0A" followed by "00]..." for utf-16le:
+			if (e.end == len(line) and line[e.start] in b'\r\n'):
+				return line[0:e.start].decode(enc, 'replace')
 			global _decode_line_warn
 			lev = 7
 			if not _decode_line_warn.get(filename, 0):
@@ -1424,29 +1458,85 @@ class FileContainer:
 			logSys.log(lev,
 				"Error decoding line from '%s' with '%s'.", filename, enc)
 			if logSys.getEffectiveLevel() <= lev:
-				logSys.log(lev, "Consider setting logencoding=utf-8 (or another appropriate"
-					" encoding) for this jail. Continuing"
-					" to process line ignoring invalid characters: %r",
+				logSys.log(lev,
+					"Consider setting logencoding to appropriate encoding for this jail. "
+					"Continuing to process line ignoring invalid characters: %r",
 					line)
 			# decode with replacing error chars:
 			line = line.decode(enc, 'replace')
 		return line

-	def readline(self):
+	def readline(self, complete=True):
+		"""Read line from file
+
+		In opposite to pythons readline it doesn't return new-line, 
+		so returns either the line if line is complete (and complete=True) or None
+		if line is not complete (and complete=True) or there is no content to read.
+		If line is complete (and complete is True), it also shift current known 
+		position to begin of next line.
+
+		Also it is safe against interim new-line bytes (e. g. part of multi-byte char)
+		in given encoding.
+		"""
 		if self.__handler is None:
 			return ""
-		return FileContainer.decode_line(
-			self.getFileName(), self.getEncoding(), self.__handler.readline())
+		# read raw bytes up to \n char:
+		b = self.__handler.readline()
+		if not b:
+			return None
+		bl = len(b)
+		# convert to log-encoding (new-line char could disappear if it is part of multi-byte sequence):
+		r = FileContainer.decode_line(
+			self.getFileName(), self.getEncoding(), b)
+		# trim new-line at end and check the line was written complete (contains a new-line):
+		l = r.rstrip('\r\n')
+		if complete:
+			if l == r:
+				# try to fill buffer in order to find line-end in log encoding:
+				fnd = 0
+				while 1:
+					r = self.__handler.readline()
+					if not r:
+						break
+					b += r
+					bl += len(r)
+					# convert to log-encoding:
+					r = FileContainer.decode_line(
+						self.getFileName(), self.getEncoding(), b)
+					# ensure new-line is not in the middle (buffered 2 strings, e. g. in utf-16le it is "...[0A"+"00]..."):
+					e = r.find('\n')
+					if e >= 0 and e != len(r)-1:
+						l, r = r[0:e], r[0:e+1]
+						# back to bytes and get offset to seek after NL:
+						r = r.encode(self.getEncoding(), 'replace')
+						self.__handler.seek(-bl+len(r), 1)
+						return l
+					# trim new-line at end and check the line was written complete (contains a new-line):
+					l = r.rstrip('\r\n')
+					if l != r:
+						return l
+				if self.waitForLineEnd:
+					# not fulfilled - seek back and return:
+					self.__handler.seek(-bl, 1)
+					return None
+		return l

 	def close(self):
-		if not self.__handler is None:
-			# Saves the last position.
+		if self.__handler is not None:
+			# Saves the last real position.
 			self.__pos = self.__handler.tell()
 			# Closes the file.
 			self.__handler.close()
 			self.__handler = None
-		## print "D: Closed %s with pos %d" % (handler, self.__pos)
-		## sys.stdout.flush()
+
+	def __iter__(self):
+		return self
+	def next(self):
+		line = self.readline()
+		if line is None:
+			self.close()
+			raise StopIteration
+		return line

 _decode_line_warn = Utils.Cache(maxCount=1000, maxTime=24*60*60);

@@ -232,7 +232,7 @@ class ObserverThread(JailThread):
 					if self._paused:
 						continue
 				else:
-					## notify event deleted (shutdown) - just sleep a litle bit (waiting for shutdown events, prevent high cpu usage)
+					## notify event deleted (shutdown) - just sleep a little bit (waiting for shutdown events, prevent high cpu usage)
 					time.sleep(ObserverThread.DEFAULT_SLEEP_INTERVAL)
 					## stop by shutdown and empty queue :
 					if not self.is_full:
@@ -271,16 +271,12 @@ def reGroupDictStrptime(found_dict, msec=False, default_tz=None):
 			week_of_year = int(val)
 			# U starts week on Sunday, W - on Monday
 			week_of_year_start = 6 if key == 'U' else 0
-		elif key == 'z':
+		elif key in ('z', 'Z'):
 			z = val
 			if z in ("Z", "UTC", "GMT"):
 				tzoffset = 0
 			else:
 				tzoffset = zone2offset(z, 0); # currently offset-based only
-		elif key == 'Z':
-			z = val
-			if z in ("UTC", "GMT"):
-				tzoffset = 0

 	# Fail2Ban will assume it's this year
 	assume_year = False
@@ -332,11 +332,9 @@ class Utils():
 					timeout_expr = lambda: time.time() > time0
 				else:
 					timeout_expr = timeout
-				if not interval:
-					interval = Utils.DEFAULT_SLEEP_INTERVAL
 			if timeout_expr():
 				break
-			stm = min(stm + interval, Utils.DEFAULT_SLEEP_TIME)
+			stm = min(stm + (interval or Utils.DEFAULT_SLEEP_INTERVAL), Utils.DEFAULT_SLEEP_TIME)
 			time.sleep(stm)
 		return ret

@@ -217,6 +217,9 @@ class ExecuteActions(LogCaptureTestCase):
 		# flush for inet6 is intentionally "broken" here - test no unhandled except and invariant check:
 		act['actionflush?family=inet6'] = act.actionflush + '; exit 1'
 		act.actionstart_on_demand = True
+		# force errors via check in ban/unban:
+		act.actionban = "<actioncheck> ; " + act.actionban
+		act.actionunban = "<actioncheck> ; " + act.actionunban
 		self.__actions.start()
 		self.assertNotLogged("stdout: %r" % 'ip start')

@@ -294,6 +297,9 @@ class ExecuteActions(LogCaptureTestCase):
 		act['actionflush?family=inet6'] = act.actionflush + '; exit 1'
 		act.actionstart_on_demand = True
 		act.actionrepair_on_unban = True
+		# force errors via check in ban/unban:
+		act.actionban = "<actioncheck> ; " + act.actionban
+		act.actionunban = "<actioncheck> ; " + act.actionunban
 		self.__actions.start()
 		self.assertNotLogged("stdout: %r" % 'ip start')

@@ -305,8 +305,8 @@ class CommandActionTest(LogCaptureTestCase):
 		self.assertEqual(self.__action.actionstart, "touch '%s'" % tmp)
 		self.__action.actionstop = "rm -f '%s'" % tmp
 		self.assertEqual(self.__action.actionstop, "rm -f '%s'" % tmp)
-		self.__action.actionban = "echo -n"
-		self.assertEqual(self.__action.actionban, 'echo -n')
+		self.__action.actionban = "<actioncheck> && echo -n"
+		self.assertEqual(self.__action.actionban, "<actioncheck> && echo -n")
 		self.__action.actioncheck = "[ -e '%s' ]" % tmp
 		self.assertEqual(self.__action.actioncheck, "[ -e '%s' ]" % tmp)
 		self.__action.actionunban = "true"
@@ -316,6 +316,7 @@ class CommandActionTest(LogCaptureTestCase):
 		self.assertNotLogged('returned')
 		# no action was actually executed yet

+		# start on demand is false, so it should cause failure on first attempt of ban:
 		self.__action.ban({'ip': None})
 		self.assertLogged('Invariant check failed')
 		self.assertLogged('returned successfully')
@@ -365,12 +366,51 @@ class CommandActionTest(LogCaptureTestCase):
 		self.pruneLog('[phase 2]')
 		self.__action.actionstart = "touch '%s'" % tmp
 		self.__action.actionstop = "rm '%s'" % tmp
-		self.__action.actionban = """printf "%%%%b\n" <ip> >> '%s'""" % tmp
+		self.__action.actionban = """<actioncheck> && printf "%%%%b\n" <ip> >> '%s'""" % tmp
 		self.__action.actioncheck = "[ -e '%s' ]" % tmp
 		self.__action.ban({'ip': None})
 		self.assertLogged('Invariant check failed')
 		self.assertNotLogged('Unable to restore environment')

+	@with_tmpdir
+	def testExecuteActionCheckOnBanFailure(self, tmp):
+		tmp += '/fail2ban.test'
+		self.__action.actionstart = "touch '%s'; echo 'started ...'" % tmp
+		self.__action.actionstop = "rm -f '%s'" % tmp
+		self.__action.actionban = "[ -e '%s' ] && echo 'banned '<ip>" % tmp
+		self.__action.actioncheck = "[ -e '%s' ] && echo 'check ok' || { echo 'check failed'; exit 1; }" % tmp
+		self.__action.actionrepair = "echo 'repair ...'; touch '%s'" % tmp
+		self.__action.actionstart_on_demand = False
+		self.__action.start()
+		# phase 1: with repair;
+		# phase 2: without repair (start/stop), not on demand;
+		# phase 3: without repair (start/stop), start on demand.
+		for i in (1, 2, 3):
+			self.pruneLog('[phase %s]' % i)
+			# 1st time with success ban:
+			self.__action.ban({'ip': '192.0.2.1'})
+			self.assertLogged(
+				"stdout: %r" % 'banned 192.0.2.1', all=True)
+			self.assertNotLogged("Invariant check failed. Trying", 
+				"stdout: %r" % 'check failed',
+				"stdout: %r" % ('repair ...' if self.__action.actionrepair else 'started ...'),
+				"stdout: %r" % 'check ok', all=True)
+			# force error in ban:
+			os.remove(tmp)
+			self.pruneLog()
+			# 2nd time with fail recognition, success repair, check and ban:
+			self.__action.ban({'ip': '192.0.2.2'})
+			self.assertLogged("Invariant check failed. Trying",
+				"stdout: %r" % 'check failed',
+				"stdout: %r" % ('repair ...' if self.__action.actionrepair else 'started ...'),
+				"stdout: %r" % 'check ok', 
+				"stdout: %r" % 'banned 192.0.2.2', all=True)
+			# repeat without repair (stop/start), herafter enable on demand:
+			if self.__action.actionrepair:
+				self.__action.actionrepair = ""
+			elif not self.__action.actionstart_on_demand:
+				self.__action.actionstart_on_demand = True
+
 	@with_tmpdir
 	def testExecuteActionCheckRepairEnvironment(self, tmp):
 		tmp += '/fail2ban.test'
@@ -29,7 +29,7 @@ import tempfile
 import sqlite3
 import shutil

-from ..server.filter import FileContainer
+from ..server.filter import FileContainer, Filter
 from ..server.mytime import MyTime
 from ..server.ticket import FailTicket
 from ..server.actions import Actions, Utils
@@ -212,19 +212,20 @@ class DatabaseTest(LogCaptureTestCase):
 			self.jail.name in self.db.getJailNames(True),
 			"Jail not added to database")

-	def testAddLog(self):
+	def _testAddLog(self):
 		self.testAddJail() # Jail required

 		_, filename = tempfile.mkstemp(".log", "Fail2BanDb_")
 		self.fileContainer = FileContainer(filename, "utf-8")

-		self.db.addLog(self.jail, self.fileContainer)
+		pos = self.db.addLog(self.jail, self.fileContainer)
+		self.assertTrue(pos is None); # unknown previously

 		self.assertIn(filename, self.db.getLogPaths(self.jail))
 		os.remove(filename)

 	def testUpdateLog(self):
-		self.testAddLog() # Add log file
+		self._testAddLog() # Add log file

 		# Write some text
 		filename = self.fileContainer.getFileName()
@@ -544,17 +545,21 @@ class DatabaseTest(LogCaptureTestCase):
 		self.testAddJail() # Jail required
 		self.jail.database = self.db
 		self.db.addJail(self.jail)
-		actions = Actions(self.jail)
+		actions = self.jail.actions
 		actions.add(
 			"action_checkainfo",
 			os.path.join(TEST_FILES_DIR, "action.d/action_checkainfo.py"),
 			{})
+		actions.banManager.setBanTotal(20)
+		self.jail._Jail__filter = flt = Filter(self.jail)
+		flt.failManager.setFailTotal(50)
 		ticket = FailTicket("1.2.3.4")
 		ticket.setAttempt(5)
 		ticket.setMatches(['test', 'test'])
 		self.jail.putFailTicket(ticket)
 		actions._Actions__checkBan()
 		self.assertLogged("ban ainfo %s, %s, %s, %s" % (True, True, True, True))
+		self.assertLogged("jail info %d, %d, %d, %d" % (1, 21, 0, 50))

 	def testDelAndAddJail(self):
 		self.testAddJail() # Add jail
@@ -516,6 +516,9 @@ class CustomDateFormatsTest(unittest.TestCase):
 			(1072746123.0 - 3600, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %Z)?", "[2003-12-30 01:02:03] server ..."),
 			(1072746123.0,        "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %z)?", "[2003-12-30 01:02:03 UTC] server ..."),
 			(1072746123.0,        "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %Z)?", "[2003-12-30 01:02:03 UTC] server ..."),
+			(1072746123.0,        "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %z)?", "[2003-12-30 01:02:03 Z] server ..."),
+			(1072746123.0,        "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %z)?", "[2003-12-30 01:02:03 +0000] server ..."),
+			(1072746123.0,        "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %Z)?", "[2003-12-30 01:02:03 Z] server ..."),
 		):
 			logSys.debug('== test: %r', (matched, dp, line))
 			if dp is None:
@@ -230,7 +230,7 @@ def _start_params(tmp, use_stock=False, use_stock_cfg=None,
 			os.symlink(os.path.abspath(pjoin(STOCK_CONF_DIR, n)), pjoin(cfg, n))
 	if create_before_start:
 		for n in create_before_start:
-			_write_file(n % {'tmp': tmp}, 'w', '')
+			_write_file(n % {'tmp': tmp}, 'w')
 	# parameters (sock/pid and config, increase verbosity, set log, etc.):
 	vvv, llev = (), "INFO"
 	if unittest.F2B.log_level < logging.INFO: # pragma: no cover
@@ -937,10 +937,8 @@ class Fail2banServerTest(Fail2banClientServerBase):
 			"Jail 'broken-jail' skipped, because of wrong configuration", all=True)
 		
 		# enable both jails, 3 logs for jail1, etc...
-		# truncate test-log - we should not find unban/ban again by reload:
 		self.pruneLog("[test-phase 1b]")
 		_write_jail_cfg(actions=[1,2])
-		_write_file(test1log, "w+")
 		if unittest.F2B.log_level < logging.DEBUG: # pragma: no cover
 			_out_file(test1log)
 		self.execCmd(SUCCESS, startparams, "reload")
@@ -1003,7 +1001,7 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		
 		self.pruneLog("[test-phase 2b]")
 		# write new failures:
-		_write_file(test2log, "w+", *(
+		_write_file(test2log, "a+", *(
 			(str(int(MyTime.time())) + "   error 403 from 192.0.2.2: test 2",) * 3 +
 		  (str(int(MyTime.time())) + "   error 403 from 192.0.2.3: test 2",) * 3 +
 		  (str(int(MyTime.time())) + " failure 401 from 192.0.2.4: test 2",) * 3 +
@@ -1062,10 +1060,6 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		self.assertEqual(self.execCmdDirect(startparams,
 			'get', 'test-jail1', 'banned', '192.0.2.3', '192.0.2.9')[1],  [1, 0])

-		# rotate logs:
-		_write_file(test1log, "w+")
-		_write_file(test2log, "w+")
-
 		# restart jail without unban all:
 		self.pruneLog("[test-phase 2c]")
 		self.execCmd(SUCCESS, startparams,
@@ -1183,7 +1177,7 @@ class Fail2banServerTest(Fail2banClientServerBase):

 		# now write failures again and check already banned (jail1 was alive the whole time) and new bans occurred (jail1 was alive the whole time):
 		self.pruneLog("[test-phase 5]")
-		_write_file(test1log, "w+", *(
+		_write_file(test1log, "a+", *(
 			(str(int(MyTime.time())) + " failure 401 from 192.0.2.1: test 5",) * 3 + 
 			(str(int(MyTime.time())) + "   error 403 from 192.0.2.5: test 5",) * 3 +
 			(str(int(MyTime.time())) + " failure 401 from 192.0.2.6: test 5",) * 3
@@ -1410,8 +1404,9 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		'jails': (
 			# default:
 			'''test_action = dummy[actionstart_on_demand=1, init="start: %(__name__)s", target="%(tmp)s/test.txt",
-      actionban='<known/actionban>;
-        echo "<matches>"; printf "=====\\n%%b\\n=====\\n\\n" "<matches>" >> <target>']''',
+      actionban='<known/actionban>; echo "found: <jail.found> / <jail.found_total>, banned: <jail.banned> / <jail.banned_total>"
+        echo "<matches>"; printf "=====\\n%%b\\n=====\\n\\n" "<matches>" >> <target>',
+      actionstop='<known/actionstop>; echo "stats <name> - found: <jail.found_total>, banned: <jail.banned_total>"']''',
 			# jail sendmail-auth:
 			'[sendmail-auth]',
 			'backend = polling',
@@ -1456,7 +1451,8 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		_write_file(lgfn, "w+", *smaut_msg)
 		# wait and check it caused banned (and dump in the test-file):
 		self.assertLogged(
-			"[sendmail-auth] Ban 192.0.2.1", "1 ticket(s) in 'sendmail-auth'", all=True, wait=MID_WAITTIME)
+			"[sendmail-auth] Ban 192.0.2.1",  "stdout: 'found: 0 / 3, banned: 1 / 1'",
+			"1 ticket(s) in 'sendmail-auth'", all=True, wait=MID_WAITTIME)
 		_out_file(tofn)
 		td = _read_file(tofn)
 		# check matches (maxmatches = 2, so only 2 & 3 available):
@@ -1467,10 +1463,11 @@ class Fail2banServerTest(Fail2banClientServerBase):

 		self.pruneLog("[test-phase sendmail-reject]")
 		# write log:
-		_write_file(lgfn, "w+", *smrej_msg)
+		_write_file(lgfn, "a+", *smrej_msg)
 		# wait and check it caused banned (and dump in the test-file):
 		self.assertLogged(
-			"[sendmail-reject] Ban 192.0.2.2", "1 ticket(s) in 'sendmail-reject'", all=True, wait=MID_WAITTIME)
+			"[sendmail-reject] Ban 192.0.2.2", "stdout: 'found: 0 / 3, banned: 1 / 1'",
+			"1 ticket(s) in 'sendmail-reject'", all=True, wait=MID_WAITTIME)
 		_out_file(tofn)
 		td = _read_file(tofn)
 		# check matches (no maxmatches, so all matched messages are available):
@@ -1484,6 +1481,8 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		# wait a bit:
 		self.assertLogged(
 			"Reload finished.",
+			"stdout: 'stats sendmail-auth - found: 3, banned: 1'",
+			"stdout: 'stats sendmail-reject - found: 3, banned: 1'",
 			"[sendmail-auth] Restore Ban 192.0.2.1", "1 ticket(s) in 'sendmail-auth'", all=True, wait=MID_WAITTIME)
 		# check matches again - (dbmaxmatches = 1), so it should be only last match after restart:
 		td = _read_file(tofn)
@@ -1592,7 +1591,7 @@ class Fail2banServerTest(Fail2banClientServerBase):
 		wakeObs = False
 		_observer_wait_before_incrban(lambda: wakeObs)
 		# write again (IP already bad):
-		_write_file(test1log, "w+", *(
+		_write_file(test1log, "a+", *(
 		  (str(int(MyTime.time())) + " failure 401 from 192.0.2.11: I'm very bad \"hacker\" `` $(echo test)",) * 2
 		))
 		# wait for ban:
@@ -25,6 +25,7 @@ __license__ = "GPL"

 import os
 import sys
+import tempfile
 import unittest

 from ..client import fail2banregex
@@ -80,6 +81,11 @@ def _test_exec_command_line(*args):
 		sys.stderr = _org['stderr']
 	return _exit_code

+def _reset():
+	# reset global warn-counter:
+	from ..server.filter import _decode_line_warn
+	_decode_line_warn.clear()
+
 STR_00 = "Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 192.0.2.0"
 STR_00_NODT = "[sshd] error: PAM: Authentication failure for kevin from 192.0.2.0"

@@ -122,6 +128,7 @@ class Fail2banRegexTest(LogCaptureTestCase):
 		"""Call before every test case."""
 		LogCaptureTestCase.setUp(self)
 		setUpMyTime()
+		_reset()

 	def tearDown(self):
 		"""Call after every test case."""
@@ -464,14 +471,8 @@ class Fail2banRegexTest(LogCaptureTestCase):
 			FILENAME_ZZZ_GEN, FILENAME_ZZZ_GEN
 		))

-	def _reset(self):
-		# reset global warn-counter:
-		from ..server.filter import _decode_line_warn
-		_decode_line_warn.clear()
-
 	def testWronChar(self):
 		unittest.F2B.SkipIfCfgMissing(stock=True)
-		self._reset()
 		self.assertTrue(_test_exec(
 		"-l", "notice", # put down log-level, because of too many debug-messages
 			"--datepattern", r"^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?",
@@ -487,7 +488,6 @@ class Fail2banRegexTest(LogCaptureTestCase):

 	def testWronCharDebuggex(self):
 		unittest.F2B.SkipIfCfgMissing(stock=True)
-		self._reset()
 		self.assertTrue(_test_exec(
 		"-l", "notice", # put down log-level, because of too many debug-messages
 			"--datepattern", r"^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?",
@@ -500,6 +500,36 @@ class Fail2banRegexTest(LogCaptureTestCase):

 		self.assertLogged('https://')

+	def testNLCharAsPartOfUniChar(self):
+		fname = tempfile.mktemp(prefix='tmp_fail2ban', suffix='uni')
+		# test two multi-byte encodings (both contains `\x0A` in either \x02\x0A or \x0A\x02):
+		for enc in ('utf-16be', 'utf-16le'):
+			self.pruneLog("[test-phase encoding=%s]" % enc)
+			try:
+				fout = open(fname, 'wb')
+				# test on unicode string containing \x0A as part of uni-char,
+				# it must produce exactly 2 lines (both are failures):
+				for l in (
+					u'1490349000 \u20AC Failed auth: invalid user Test\u020A from 192.0.2.1\n',
+					u'1490349000 \u20AC Failed auth: invalid user TestI from 192.0.2.2\n'
+				):
+					fout.write(l.encode(enc))
+				fout.close()
+
+				self.assertTrue(_test_exec(
+				"-l", "notice", # put down log-level, because of too many debug-messages
+					"--encoding", enc,
+					"--datepattern", r"^EPOCH",
+					fname, r"Failed .* from <HOST>",
+				))
+
+				self.assertLogged(" encoding : %s" % enc,
+					"Lines: 2 lines, 0 ignored, 2 matched, 0 missed", all=True)
+				self.assertNotLogged("Missed line(s)")
+			finally:
+				fout.close()
+				os.unlink(fname)
+
 	def testExecCmdLine_Usage(self):
 		self.assertNotEqual(_test_exec_command_line(), 0)
 		self.pruneLog()
@@ -8,6 +8,9 @@ class TestAction(ActionBase):
        self._logSys.info("ban ainfo %s, %s, %s, %s",
          aInfo["ipmatches"] != '', aInfo["ipjailmatches"] != '', aInfo["ipfailures"] > 0, aInfo["ipjailfailures"] > 0
        )
+        self._logSys.info("jail info %d, %d, %d, %d",
+          aInfo["jail.banned"], aInfo["jail.banned_total"], aInfo["jail.found"], aInfo["jail.found_total"]
+        )

    def unban(self, aInfo):
        pass
@@ -19,6 +19,8 @@
 [2012-02-13 17:44:26] NOTICE[1638] chan_iax2.c: Host 1.2.3.4 failed MD5 authentication for 'Fail2ban' (e7df7cd2ca07f4f1ab415d457a6e1c13 != 53ac4bc41ee4ec77888ed4aa50677247)
 # failJSON: { "time": "2013-02-05T23:44:42", "match": true , "host": "1.2.3.4" }
 [2013-02-05 23:44:42] NOTICE[436][C-00000fa9] chan_sip.c: Call from '' (1.2.3.4:10836) to extension '0972598285108' rejected because extension not found in context 'default'.
+# failJSON: { "time": "2005-01-18T17:39:50", "match": true , "host": "1.2.3.4" }
+[Jan 18 17:39:50] NOTICE[12049]: res_pjsip_session.c:2337 new_invite: Call from 'anonymous' (TCP:[1.2.3.4]:61470) to extension '9011+442037690237' rejected because extension not found in context 'default'.
 # failJSON: { "time": "2013-03-26T15:47:54", "match": true , "host": "1.2.3.4" }
 [2013-03-26 15:47:54] NOTICE[1237] chan_sip.c: Registration from '"100"sip:100@1.2.3.4' failed for '1.2.3.4:23930' - No matching peer found
 # failJSON: { "time": "2013-05-13T07:10:53", "match": true , "host": "1.2.3.4" }
@@ -3,5 +3,15 @@ Apr 26 13:15:25 webserver example.com: https://example.com|1430068525|user|1.2.3
 # failJSON: { "time": "2005-04-26T13:15:25", "match": true , "host": "1.2.3.4" }
 Apr 26 13:15:25 webserver example.com: https://example.com/subdir|1430068525|user|1.2.3.4|https://example.com/subdir/user|https://example.com/subdir/user|0||Login attempt failed for drupaladmin.

-# failJSON: { "time": "2005-04-26T13:19:08", "match": false , "host": "1.2.3.4" }
+# failJSON: { "time": "2005-04-26T13:19:08", "match": false , "host": "1.2.3.4", "user": "drupaladmin" }
 Apr 26 13:19:08 webserver example.com: https://example.com|1430068748|user|1.2.3.4|https://example.com/user|https://example.com/user|1||Session opened for drupaladmin.
+
+# failJSON: { "time": "2005-04-26T13:20:00", "match": false, "desc": "attempt to inject on URI (pipe, login failed for), not a failure, gh-2742" }
+Apr 26 13:20:00 host drupal-site: https://example.com|1613063581|user|192.0.2.5|https://example.com/user/login?test=%7C&test2=%7C...|https://example.com/user/login?test=|&test2=|0||Login attempt failed for tester|2||Session revisited for drupaladmin.
+
+# failJSON: { "time": "2005-04-26T13:20:01", "match": true , "host": "192.0.2.7", "user": "Jack Sparrow", "desc": "log-format change - for -> from, user name with space, gh-2742" }
+Apr 26 13:20:01 mweb drupal_site[24864]: https://www.example.com|1613058599|user|192.0.2.7|https://www.example.com/en/user/login|https://www.example.com/en/user/login|0||Login attempt failed from Jack Sparrow.
+# failJSON: { "time": "2005-04-26T13:20:02", "match": true , "host": "192.0.2.4", "desc": "attempt to inject on URI (pipe), login failed, gh-2742" }
+Apr 26 13:20:02 host drupal-site: https://example.com|1613063581|user|192.0.2.4|https://example.com/user/login?test=%7C&test2=%7C|https://example.com/user/login?test=|&test2=||0||Login attempt failed from 192.0.2.4.
+# failJSON: { "time": "2005-04-26T13:20:03", "match": false, "desc": "attempt to inject on URI (pipe, login failed from), not a failure, gh-2742" }
+Apr 26 13:20:03 host drupal-site: https://example.com|1613063581|user|192.0.2.5|https://example.com/user/login?test=%7C&test2=%7C...|https://example.com/user/login?test=|&test2=|0||Login attempt failed from 1.2.3.4|2||Session revisited for drupaladmin.
@@ -1,7 +1,7 @@
 # Previous version --
-# failJSON: { "time": "2005-04-16T21:05:29", "match": true , "host": "69.93.127.111" }
+# failJSON: { "time": "2005-04-17T06:05:29", "match": true , "host": "69.93.127.111" }
 [PDT Apr 16 21:05:29] error    : Warning: Client '69.93.127.111' supplied unknown user 'foo' accessing monit httpd
-# failJSON: { "time": "2005-04-16T20:59:33", "match": true , "host": "97.113.189.111" }
+# failJSON: { "time": "2005-04-17T05:59:33", "match": true , "host": "97.113.189.111" }
 [PDT Apr 16 20:59:33] error    : Warning: Client '97.113.189.111' supplied wrong password for user 'admin' accessing monit httpd

 # Current version -- corresponding "https://bitbucket.org/tildeslash/monit/src/6905335aa903d425cae732cab766bd88ea5f2d1d/src/http/processor.c?at=master&fileviewer=file-view-default#processor.c-728"
@@ -0,0 +1,8 @@
+# failJSON: { "time": "2021-04-14T08:11:01", "match": false, "desc": "should be ignored: successful request" }
+Wed Apr 14 08:11:01 2021 - OK - [127.0.0.1] "GET /monitorix-cgi/monitorix.cgi - Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"
+# failJSON: { "time": "2021-04-14T08:54:22", "match": true, "host": "127.0.0.1", "desc": "file does not exist"  }
+Wed Apr 14 08:54:22 2021 - NOTEXIST - [127.0.0.1] File does not exist: /manager/html
+# failJSON: { "time": "2021-04-14T11:24:31", "match": true, "host": "127.0.0.1", "desc": "access not allowed"  }
+Wed Apr 14 11:24:31 2021 - NOTALLOWED - [127.0.0.1] Access not allowed: /monitorix/
+# failJSON: { "time": "2021-04-14T11:26:08", "match": true, "host": "127.0.0.1", "desc": "authentication error"  }
+Wed Apr 14 11:26:08 2021 - AUTHERR - [127.0.0.1] Authentication error: /monitorix/
@@ -0,0 +1,11 @@
+# failJSON: { "time": "2020-02-24T16:05:21", "match": true , "host": "192.0.2.1" }
+2020-02-24 16:05:21.00 Logon       Login failed for user 'Backend'. Reason: Could not find a login matching the name provided. [CLIENT: 192.0.2.1]
+# failJSON: { "time": "2020-02-24T16:30:25", "match": true , "host": "192.0.2.2" }
+2020-02-24 16:30:25.88 Logon       Login failed for user '===)jf02hüas9ä##22f'. Reason: Could not find a login matching the name provided. [CLIENT: 192.0.2.2]
+# failJSON: { "time": "2020-02-24T16:31:12", "match": true , "host": "192.0.2.3" }
+2020-02-24 16:31:12.20 Logon       Login failed for user ''. Reason: An attempt to login using SQL authentication failed. Server is configured for Integrated authentication only. [CLIENT: 192.0.2.3]
+
+# failJSON: { "time": "2020-02-24T16:31:26", "match": true , "host": "192.0.2.4", "user":"O'Leary" }
+2020-02-24 16:31:26.01 Logon       Login failed for user 'O'Leary'. Reason: Could not find a login matching the name provided. [CLIENT: 192.0.2.4]
+# failJSON: { "time": "2020-02-24T16:31:26", "match": false, "desc": "test injection in possibly unescaped foreign input" }
+2020-02-24 16:31:26.02 Wrong data received: Logon       Login failed for user 'test'. Reason: Could not find a login matching the name provided. [CLIENT: 192.0.2.5]
@@ -0,0 +1,23 @@
+# failJSON: { "time": "2015-01-20T19:53:28", "match": true , "host": "12.34.56.78" }
+12.34.56.78 - - [20/Jan/2015:19:53:28 +0100] "" 400 47 "-" "-" "-"
+
+# failJSON: { "time": "2015-01-20T19:53:28", "match": true , "host": "12.34.56.78" }
+12.34.56.78 - root [20/Jan/2015:19:53:28 +0100] "" 400 47 "-" "-" "-"
+
+# failJSON: { "time": "2015-01-20T19:53:28", "match": true , "host": "12.34.56.78" }
+12.34.56.78 - - [20/Jan/2015:19:53:28 +0100] "\x03\x00\x00/*\xE0\x00\x00\x00\x00\x00Cookie: mstshash=Administr" 400 47 "-" "-" "-"
+
+# failJSON: { "time": "2015-01-20T19:53:28", "match": true , "host": "12.34.56.78" }
+12.34.56.78 - - [20/Jan/2015:19:53:28 +0100] "GET //admin/pma/scripts/setup.php HTTP/1.1" 400 47 "-" "-" "-"
+
+# failJSON: { "time": "2015-01-20T19:54:28", "match": true , "host": "12.34.56.78" }
+12.34.56.78 - - [20/Jan/2015:19:54:28 +0100] "HELP" 400 47 "-" "-" "-"
+
+# failJSON: { "time": "2015-01-20T19:55:28", "match": true , "host": "12.34.56.78" }
+12.34.56.78 - - [20/Jan/2015:19:55:28 +0100] "batman" 400 47 "-" "-" "-"
+
+# failJSON: { "time": "2015-01-20T01:17:07", "match": true , "host": "7.8.9.10" }
+7.8.9.10 - root [20/Jan/2015:01:17:07 +0100] "CONNECT 123.123.123.123 HTTP/1.1" 400 162 "-" "-" "-"
+
+# failJSON: { "time": "2014-12-12T22:59:02", "match": true , "host": "2.5.2.5" }
+2.5.2.5 - tomcat [12/Dec/2014:22:59:02 +0100] "GET /cgi-bin/tools/tools.pl HTTP/1.1" 400 162 "-" "-" "-"
@@ -1,3 +1,4 @@
+# filterOptions: [{"mode": "normal"}, {"mode": "auth"}]

 # failJSON: { "time": "2012-04-09T11:53:29", "match": true , "host": "192.0.43.10" }
 2012/04/09 11:53:29 [error] 2865#0: *66647 user "xyz" was not found in "/var/www/.htpasswd", client: 192.0.43.10, server: www.myhost.com, request: "GET / HTTP/1.1", host: "www.myhost.com"
@@ -11,3 +12,20 @@
 2014/04/03 22:20:38 [error] 30708#0: *3 user "scriben dio": password mismatch, client: 192.0.2.1, server: , request: "GET / HTTP/1.1", host: "localhost:8443"
 # failJSON: { "time": "2014-04-03T22:20:40", "match": true, "host": "192.0.2.2", "desc": "trying injection on user name"}
 2014/04/03 22:20:40 [error] 30708#0: *3 user "test": password mismatch, client: 127.0.0.1, server: test, request: "GET / HTTP/1.1", host: "localhost:8443"": was not found in "/etc/nginx/.htpasswd", client: 192.0.2.2, server: , request: "GET / HTTP/1.1", host: "localhost:8443"
+
+# filterOptions: [{"mode": "fallback"}]
+
+# failJSON: { "time": "2020-11-25T14:42:16", "match": true , "host": "142.93.180.14" }
+2020/11/25 14:42:16 [crit] 76952#76952: *2454307 SSL_do_handshake() failed (SSL: error:1408F0C6:SSL routines:ssl3_get_record:packet length too long) while SSL handshaking, client: 142.93.180.14, server: 0.0.0.0:443
+# failJSON: { "time": "2020-11-25T15:47:47", "match": true , "host": "80.191.166.166" }
+2020/11/25 15:47:47 [crit] 76952#76952: *5062354 SSL_do_handshake() failed (SSL: error:1408F0A0:SSL routines:ssl3_get_record:length too short) while SSL handshaking, client: 80.191.166.166, server: 0.0.0.0:443
+# failJSON: { "time": "2020-11-25T16:48:08", "match": true , "host": "5.126.32.148" }
+2020/11/25 16:48:08 [crit] 76952#76952: *7976400 SSL_do_handshake() failed (SSL: error:1408F096:SSL routines:ssl3_get_record:encrypted length too long) while SSL handshaking, client: 5.126.32.148, server: 0.0.0.0:443
+# failJSON: { "time": "2020-11-25T16:02:45", "match": false }
+2020/11/25 16:02:45 [error] 76952#76952: *5645766 connect() failed (111: Connection refused) while connecting to upstream, client: 5.126.32.148, server: www.google.de, request: "GET /admin/config HTTP/2.0", upstream: "http://127.0.0.1:3000/admin/config", host: "www.google.de"
+
+# filterOptions: [{"mode": "aggressive"}]
+# failJSON: { "time": "2020-11-25T14:42:16", "match": true , "host": "142.93.180.14" }
+2020/11/25 14:42:16 [crit] 76952#76952: *2454307 SSL_do_handshake() failed (SSL: error:1408F0C6:SSL routines:ssl3_get_record:packet length too long) while SSL handshaking, client: 142.93.180.14, server: 0.0.0.0:443
+# failJSON: { "time": "2012-04-09T11:53:29", "match": true , "host": "192.0.43.10" }
+2012/04/09 11:53:29 [error] 2865#0: *66647 user "xyz" was not found in "/var/www/.htpasswd", client: 192.0.43.10, server: www.myhost.com, request: "GET / HTTP/1.1", host: "www.myhost.com"
@@ -2,3 +2,5 @@
 [1387288694] nsd[7745]: info: ratelimit block example.com. type any target 192.0.2.0/24 query 192.0.2.105 TYPE255
 # failJSON: { "time": "2013-12-18T07:42:15", "match": true , "host": "192.0.2.115" }
 [1387348935] nsd[23600]: info: axfr for zone domain.nl. from client 192.0.2.115 refused, no acl matches.
+# failJSON: { "time": "2021-03-05T05:25:14", "match": true , "host": "192.0.2.32", "desc": "new format, no client after from, no dot at end, gh-2965" }
+[2021-03-05 05:25:14.562] nsd[160800]: info: axfr for example.com. from 192.0.2.32 refused, no acl matches
@@ -15,6 +15,9 @@ Aug 10 10:55:38 f-vanier-bourgeois postfix/smtpd[2162]: NOQUEUE: reject: VRFY fr
 # failJSON: { "time": "2005-08-13T15:45:46", "match": true , "host": "192.0.2.1" }
 Aug 13 15:45:46 server postfix/smtpd[13844]: 00ADB3C0899: reject: RCPT from example.com[192.0.2.1]: 550 5.1.1 <sales@server.com>: Recipient address rejected: User unknown in local recipient table; from=<xxxxxx@example.com> to=<sales@server.com> proto=ESMTP helo=<mail.example.com>

+# failJSON: { "time": "2005-05-19T00:00:30", "match": true , "host": "192.0.2.2", "desc": "undeliverable address (sender/recipient verification, gh-3039)" }
+May 19 00:00:30 proxy2 postfix/smtpd[16123]: NOQUEUE: reject: RCPT from example.net[192.0.2.2]: 550 5.1.1 <user1@example.com>: Recipient address rejected: undeliverable address: verification failed; from=<user2@example.org> to=<user1@example.com> proto=ESMTP helo=<example.net>
+
 # failJSON: { "time": "2005-01-12T11:07:49", "match": true , "host": "181.21.131.88" }
 Jan 12 11:07:49 emf1pt2-2-35-70 postfix/smtpd[13767]: improper command pipelining after DATA from unknown[181.21.131.88]:

@@ -161,6 +164,17 @@ Feb 18 09:48:04 xxx postfix/smtpd[23]: lost connection after AUTH from unknown[1
 # failJSON: { "time": "2005-02-18T09:48:04", "match": true , "host": "192.0.2.23" }
 Feb 18 09:48:04 xxx postfix/smtpd[23]: lost connection after AUTH from unknown[192.0.2.23]

+# failJSON: { "time": "2004-12-23T19:39:13", "match": true , "host": "192.0.2.2" }
+Dec 23 19:39:13 xxx postfix/postscreen[21057]: PREGREET 14 after 0.08 from [192.0.2.2]:59415: EHLO ylmf-pc\r\n
+# failJSON: { "time": "2004-12-24T00:54:36", "match": true , "host": "192.0.2.3" }
+Dec 24 00:54:36 xxx postfix/postscreen[22515]: HANGUP after 16 from [192.0.2.3]:48119 in tests after SMTP handshake
+
+# failJSON: { "time": "2005-06-08T23:14:28", "match": true , "host": "192.0.2.77", "desc": "abusive clients hitting command limit, see see http://www.postfix.org/POSTSCREEN_README.html (gh-3040)" }
+Jun  8 23:14:28 proxy2 postfix/postscreen[473]: COMMAND TIME LIMIT from [192.0.2.77]:3608 after CONNECT
+# failJSON: { "time": "2005-06-08T23:14:54", "match": true , "host": "192.0.2.26", "desc": "abusive clients hitting command limit (gh-3040)" }
+Jun  8 23:14:54 proxy2 postfix/postscreen[473]: COMMAND COUNT LIMIT from [192.0.2.26]:15592 after RCPT
+
+
 # filterOptions: [{}, {"mode": "ddos"}, {"mode": "aggressive"}]
 # failJSON: { "match": false, "desc": "don't affect lawful data (sporadical connection aborts within DATA-phase, see gh-1813 for discussion)" }
 Feb 18 09:50:05 xxx postfix/smtpd[42]: lost connection after DATA from good-host.example.com[192.0.2.10]
@@ -0,0 +1,8 @@
+# failJSON: { "time": "2005-03-05T21:44:43", "match": true , "host": "192.0.2.123" } 
+Mar  5 21:44:43 srv scanlogd: 192.0.2.123 to 192.0.2.1 ports 80, 81, 83, 88, 99, 443, 1080, 3128, ..., f????uxy, TOS 00, TTL 49 @20:44:43
+# failJSON: { "time": "2005-03-05T21:44:44", "match": true , "host": "192.0.2.123" } 
+Mar  5 21:44:44 srv scanlogd: 192.0.2.123 to 192.0.2.1 ports 497, 515, 544, 543, 464, 513, ..., fSrpauxy, TOS 00 @09:04:25
+# failJSON: { "time": "2005-03-05T21:44:45", "match": true , "host": "192.0.2.123" } 
+Mar  5 21:44:45 srv scanlogd: 192.0.2.123 to 192.0.2.1 ports 593, 548, 636, 646, 625, 631, ..., fSrpauxy, TOS 00, TTL 239 @17:34:00
+# failJSON: { "time": "2005-03-05T21:44:46", "match": true , "host": "192.0.2.123" } 
+Mar  5 21:44:46 srv scanlogd: 192.0.2.123 to 192.0.2.1 ports 22, 26, 37, 80, 25, 79, ..., fSrpauxy, TOS 00 @22:38:37
@@ -1,2 +1,8 @@
 # failJSON: { "time": "2016-03-28T16:50:49", "match": true , "host": "10.1.1.1" }
 [Mon Mar 28 16:50:49.522240 2016] [:error] [pid 1795] [client 10.1.1.1:50700] WAR [Login denied for user "username1"], referer: https://zoneminder/
+
+# failJSON: { "time": "2021-03-28T16:53:00", "match": true , "host": "10.1.1.1" }
+[Sun Mar 28 16:53:00.472693 2021] [php7:notice] [pid 11328] [client 10.1.1.1:39568] ERR [Could not retrieve user username1 details], referer: https://zm/zm/?view=logout
+
+# failJSON: { "time": "2021-03-28T16:59:14", "match": true , "host": "10.1.1.1" }
+[Sun Mar 28 16:59:14.150625 2021] [php7:notice] [pid 11336] [client 10.1.1.1:39654] ERR [Login denied for user "username1"], referer: https://zm/zm/?
@@ -196,7 +196,7 @@ def _assert_correct_last_attempt(utest, filter_, output, count=None):
 			_assert_equal_entries(utest, f, o)


-def _copy_lines_between_files(in_, fout, n=None, skip=0, mode='a', terminal_line=""):
+def _copy_lines_between_files(in_, fout, n=None, skip=0, mode='a', terminal_line="", lines=None):
 	"""Copy lines from one file to another (which might be already open)

 	Returns open fout
@@ -213,9 +213,9 @@ def _copy_lines_between_files(in_, fout, n=None, skip=0, mode='a', terminal_line
 		fin.readline()
 	# Read
 	i = 0
-	lines = []
+	if not lines: lines = []
 	while n is None or i < n:
-		l = FileContainer.decode_line(in_, 'UTF-8', fin.readline()).rstrip('\r\n')
+		l = fin.readline().decode('UTF-8', 'replace').rstrip('\r\n')
 		if terminal_line is not None and l == terminal_line:
 			break
 		lines.append(l)
@@ -223,6 +223,7 @@ def _copy_lines_between_files(in_, fout, n=None, skip=0, mode='a', terminal_line
 	# Write: all at once and flush
 	if isinstance(fout, str):
 		fout = open(fout, mode)
+	DefLogSys.debug('  ++ write %d test lines', len(lines))
 	fout.write('\n'.join(lines)+'\n')
 	fout.flush()
 	if isinstance(in_, str): # pragma: no branch - only used with str in test cases
@@ -254,7 +255,7 @@ def _copy_lines_to_journal(in_, fields={},n=None, skip=0, terminal_line=""): # p
 	# Read/Write
 	i = 0
 	while n is None or i < n:
-		l = FileContainer.decode_line(in_, 'UTF-8', fin.readline()).rstrip('\r\n')
+		l = fin.readline().decode('UTF-8', 'replace').rstrip('\r\n')
 		if terminal_line is not None and l == terminal_line:
 			break
 		journal.send(MESSAGE=l.strip(), **fields)
@@ -670,6 +671,19 @@ class LogFile(LogCaptureTestCase):
 		self.filter = FilterPoll(None)
 		self.assertRaises(IOError, self.filter.addLogPath, LogFile.MISSING)

+	def testDecodeLineWarn(self):
+		# incomplete line (missing byte at end), warning is suppressed:
+		l = u"correct line\n"
+		r = l.encode('utf-16le')
+		self.assertEqual(FileContainer.decode_line('TESTFILE', 'utf-16le', r), l)
+		self.assertEqual(FileContainer.decode_line('TESTFILE', 'utf-16le', r[0:-1]), l[0:-1])
+		self.assertNotLogged('Error decoding line')
+		# complete line (incorrect surrogate in the middle), warning is there:
+		r = b"incorrect \xc8\x0a line\n"
+		l = r.decode('utf-8', 'replace')
+		self.assertEqual(FileContainer.decode_line('TESTFILE', 'utf-8', r), l)
+		self.assertLogged('Error decoding line')
+

 class LogFileFilterPoll(unittest.TestCase):

@@ -1179,13 +1193,15 @@ def get_monitor_failures_testcase(Filter_):

 			# move aside, but leaving the handle still open...
 			os.rename(self.name, self.name + '.bak')
-			_copy_lines_between_files(GetFailures.FILENAME_01, self.name, skip=14, n=1).close()
+			_copy_lines_between_files(GetFailures.FILENAME_01, self.name, skip=14, n=1,
+				lines=["Aug 14 11:59:59 [logrotate] rotation 1"]).close()
 			self.assert_correct_last_attempt(GetFailures.FAILURES_01)
 			self.assertEqual(self.filter.failManager.getFailTotal(), 3)

 			# now remove the moved file
 			_killfile(None, self.name + '.bak')
-			_copy_lines_between_files(GetFailures.FILENAME_01, self.name, skip=12, n=3).close()
+			_copy_lines_between_files(GetFailures.FILENAME_01, self.name, skip=12, n=3,
+				lines=["Aug 14 11:59:59 [logrotate] rotation 2"]).close()
 			self.assert_correct_last_attempt(GetFailures.FAILURES_01)
 			self.assertEqual(self.filter.failManager.getFailTotal(), 6)

@@ -1239,7 +1255,7 @@ def get_monitor_failures_testcase(Filter_):
 			os.rename(tmpsub1, tmpsub2 + 'a')
 			os.mkdir(tmpsub1)
 			self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name,
-												  skip=12, n=1, mode='w')
+												  skip=12, n=1, mode='w', lines=["Aug 14 11:59:59 [logrotate] rotation 1"])
 			self.file.close()
 			self._wait4failures(2)

@@ -1250,7 +1266,7 @@ def get_monitor_failures_testcase(Filter_):
 			os.mkdir(tmpsub1)
 			self.waitForTicks(2)
 			self.file = _copy_lines_between_files(GetFailures.FILENAME_01, self.name,
-												  skip=12, n=1, mode='w')
+												  skip=12, n=1, mode='w', lines=["Aug 14 11:59:59 [logrotate] rotation 2"])
 			self.file.close()
 			self._wait4failures(3)

@@ -1700,16 +1716,49 @@ class GetFailures(LogCaptureTestCase):
 	def testCRLFFailures01(self):
 		# We first adjust logfile/failures to end with CR+LF
 		fname = tempfile.mktemp(prefix='tmp_fail2ban', suffix='crlf')
-		# poor man unix2dos:
-		fin, fout = open(GetFailures.FILENAME_01, 'rb'), open(fname, 'wb')
-		for l in fin.read().splitlines():
-			fout.write(l + b'\r\n')
-		fin.close()
-		fout.close()
+		try:
+			# poor man unix2dos:
+			fin, fout = open(GetFailures.FILENAME_01, 'rb'), open(fname, 'wb')
+			for l in fin.read().splitlines():
+				fout.write(l + b'\r\n')
+			fin.close()
+			fout.close()

-		# now see if we should be getting the "same" failures
-		self.testGetFailures01(filename=fname)
-		_killfile(fout, fname)
+			# now see if we should be getting the "same" failures
+			self.testGetFailures01(filename=fname)
+		finally:
+			_killfile(fout, fname)
+
+	def testNLCharAsPartOfUniChar(self):
+		fname = tempfile.mktemp(prefix='tmp_fail2ban', suffix='uni')
+		# test two multi-byte encodings (both contains `\x0A` in either \x02\x0A or \x0A\x02):
+		for enc in ('utf-16be', 'utf-16le'):
+			self.pruneLog("[test-phase encoding=%s]" % enc)
+			try:
+				fout = open(fname, 'wb')
+				tm = int(time.time())
+				# test on unicode string containing \x0A as part of uni-char,
+				# it must produce exactly 2 lines (both are failures):
+				for l in (
+					u'%s \u20AC Failed auth: invalid user Test\u020A from 192.0.2.1\n' % tm,
+					u'%s \u20AC Failed auth: invalid user TestI from 192.0.2.2\n' % tm
+				):
+					fout.write(l.encode(enc))
+				fout.close()
+
+				self.filter.setLogEncoding(enc)
+				self.filter.addLogPath(fname, autoSeek=0)
+				self.filter.setDatePattern((r'^EPOCH',))
+				self.filter.addFailRegex(r"Failed .* from <HOST>")
+				self.filter.getFailures(fname)
+				self.assertLogged(
+					"[DummyJail] Found 192.0.2.1",
+					"[DummyJail] Found 192.0.2.2", all=True, wait=True)
+			finally:
+				_killfile(fout, fname)
+				self.filter.delLogPath(fname)
+		# must find 4 failures and generate 2 tickets (2 IPs with each 2 failures):
+		self.assertEqual(self.filter.failManager.getFailCount(), (2, 4))

 	def testGetFailures02(self):
 		output = ('141.3.81.106', 4, 1124013539.0,
@@ -2273,6 +2322,7 @@ class DNSUtilsNetworkTests(unittest.TestCase):
 		ip1 = IPAddr('2606:2800:220:1:248:1893:25c8:1946'); ip2 = IPAddr('2606:2800:220:1:248:1893:25c8:1946'); self.assertEqual(id(ip1), id(ip2))

 	def testFQDN(self):
+		unittest.F2B.SkipIfNoNetwork()
 		sname = DNSUtils.getHostname(fqdn=False)
 		lname = DNSUtils.getHostname(fqdn=True)
 		# FQDN is not localhost if short hostname is not localhost too (or vice versa):
@@ -23,7 +23,6 @@ __copyright__ = "Copyright (c) 2013 Steven Hiscocks"
 __license__ = "GPL"

 import datetime
-import fileinput
 import inspect
 import json
 import os
@@ -156,12 +155,15 @@ def testSampleRegexsFactory(name, basedir):
 		i = 0
 		while i < len(filenames):
 			filename = filenames[i]; i += 1;
-			logFile = fileinput.FileInput(os.path.join(TEST_FILES_DIR, "logs",
-				filename), mode='rb')
+			logFile = FileContainer(os.path.join(TEST_FILES_DIR, "logs",
+				filename), 'UTF-8', doOpen=True)
+			# avoid errors if no NL char at end of test log-file:
+			logFile.waitForLineEnd = False

 			ignoreBlock = False
+			lnnum = 0
 			for line in logFile:
-				line = FileContainer.decode_line(logFile.filename(), 'UTF-8', line)
+				lnnum += 1
 				jsonREMatch = re.match("^#+ ?(failJSON|(?:file|filter)Options|addFILE):(.+)$", line)
 				if jsonREMatch:
 					try:
@@ -201,9 +203,8 @@ def testSampleRegexsFactory(name, basedir):
 						# failJSON - faildata contains info of the failure to check it.
 					except ValueError as e: # pragma: no cover - we've valid json's
 						raise ValueError("%s: %s:%i" %
-							(e, logFile.filename(), logFile.filelineno()))
+							(e, logFile.getFileName(), lnnum))
 					line = next(logFile)
-					line = FileContainer.decode_line(logFile.filename(), 'UTF-8', line)
 				elif ignoreBlock or line.startswith("#") or not line.strip():
 					continue
 				else: # pragma: no cover - normally unreachable
@@ -298,7 +299,7 @@ def testSampleRegexsFactory(name, basedir):
 						import pprint
 						raise AssertionError("%s: %s on: %s:%i, line:\n  %s\nregex (%s):\n  %s\n"
 							"faildata: %s\nfail: %s" % (
-								fltName, e, logFile.filename(), logFile.filelineno(), 
+								fltName, e, logFile.getFileName(), lnnum, 
 								line, failregex, regexList[failregex] if failregex != -1 else None,
 								'\n'.join(pprint.pformat(faildata).splitlines()),
 								'\n'.join(pprint.pformat(fail).splitlines())))
@@ -2079,25 +2079,38 @@ class ServerConfigReaderTests(LogCaptureTestCase):
 				action.ban(aInfos['ipv4'])
 				if tests.get('ip4-start'): self.assertLogged(*tests.get('*-start', ())+tests['ip4-start'], all=True)
 				if tests.get('ip6-start'): self.assertNotLogged(*tests['ip6-start'], all=True)
-				self.assertLogged(*tests.get('ip4-check',())+tests['ip4-ban'], all=True)
+				self.assertLogged(*tests['ip4-ban'], all=True)
 				self.assertNotLogged(*tests['ip6'], all=True)
 				# test unban ip4 :
 				self.pruneLog('# === unban ipv4 ===')
 				action.unban(aInfos['ipv4'])
-				self.assertLogged(*tests.get('ip4-check',())+tests['ip4-unban'], all=True)
+				self.assertLogged(*tests['ip4-unban'], all=True)
 				self.assertNotLogged(*tests['ip6'], all=True)
 				# test ban ip6 :
 				self.pruneLog('# === ban ipv6 ===')
 				action.ban(aInfos['ipv6'])
 				if tests.get('ip6-start'): self.assertLogged(*tests.get('*-start', ())+tests['ip6-start'], all=True)
 				if tests.get('ip4-start'): self.assertNotLogged(*tests['ip4-start'], all=True)
-				self.assertLogged(*tests.get('ip6-check',())+tests['ip6-ban'], all=True)
+				self.assertLogged(*tests['ip6-ban'], all=True)
 				self.assertNotLogged(*tests['ip4'], all=True)
 				# test unban ip6 :
 				self.pruneLog('# === unban ipv6 ===')
 				action.unban(aInfos['ipv6'])
-				self.assertLogged(*tests.get('ip6-check',())+tests['ip6-unban'], all=True)
+				self.assertLogged(*tests['ip6-unban'], all=True)
 				self.assertNotLogged(*tests['ip4'], all=True)
+				# test invariant check (normally on demand in error case only):
+				if tests.get('ip4-check'):
+					self.pruneLog('# === check ipv4 ===')
+					action._invariantCheck(aInfos['ipv4']['family'])
+					self.assertLogged(*tests['ip4-check'], all=True)
+					if tests.get('ip6-check') and tests['ip6-check'] != tests['ip4-check']:
+						self.assertNotLogged(*tests['ip6-check'], all=True)
+				if tests.get('ip6-check'):
+					self.pruneLog('# === check ipv6 ===')
+					action._invariantCheck(aInfos['ipv6']['family'])
+					self.assertLogged(*tests['ip6-check'], all=True)
+					if tests.get('ip4-check') and tests['ip4-check'] != tests['ip6-check']:
+						self.assertNotLogged(*tests['ip4-check'], all=True)
 				# test flush for actions should supported this:
 				if tests.get('flush'):
 					self.pruneLog('# === flush ===')
@@ -24,7 +24,7 @@ __author__ = "Cyril Jaquier, Yaroslav Halchenko, Steven Hiscocks, Daniel Black"
 __copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2005-2016 Yaroslav Halchenko, 2013-2014 Steven Hiscocks, Daniel Black"
 __license__ = "GPL-v2+"

-version = "0.11.2"
+version = "1.0.1.dev1"

 def normVersion():
  """ Returns fail2ban version in normalized machine-readable format"""
@@ -1,12 +1,12 @@
 .\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.4.
-.TH FAIL2BAN-CLIENT "1" "November 2020" "fail2ban-client v0.11.2" "User Commands"
+.TH FAIL2BAN-CLIENT "1" "February 2020" "fail2ban-client v1.0.1.dev1" "User Commands"
 .SH NAME
 fail2ban-client \- configure and control the server
 .SH SYNOPSIS
 .B fail2ban-client
 [\fI\,OPTIONS\/\fR] \fI\,<COMMAND>\/\fR
 .SH DESCRIPTION
-Fail2Ban v0.11.2 reads log file that contains password failure report
+Fail2Ban v1.0.1.dev1 reads log file that contains password failure report
 and bans the corresponding IP addresses using firewall rules.
 .SH OPTIONS
 .TP
@@ -415,7 +415,7 @@ gets the time a host is banned for
 <JAIL>
 .TP
 \fBget <JAIL> datepattern\fR
-gets the patern used to match
+gets the pattern used to match
 date/times for <JAIL>
 .TP
 \fBget <JAIL> usedns\fR
@@ -1,5 +1,5 @@
 .\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.4.
-.TH FAIL2BAN-PYTHON "1" "November 2020" "fail2ban-python 0.11.2" "User Commands"
+.TH FAIL2BAN-PYTHON "1" "January 2020" "fail2ban-python 1.0.1.1" "User Commands"
 .SH NAME
 fail2ban-python \- a helper for Fail2Ban to assure that the same Python is used
 .SH DESCRIPTION
@@ -1,5 +1,5 @@
 .\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.4.
-.TH FAIL2BAN-REGEX "1" "November 2020" "fail2ban-regex 0.11.2" "User Commands"
+.TH FAIL2BAN-REGEX "1" "January 2020" "fail2ban-regex 1.0.1.dev1" "User Commands"
 .SH NAME
 fail2ban-regex \- test Fail2ban "failregex" option
 .SH SYNOPSIS
@@ -1,12 +1,12 @@
 .\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.4.
-.TH FAIL2BAN-SERVER "1" "November 2020" "fail2ban-server v0.11.2" "User Commands"
+.TH FAIL2BAN-SERVER "1" "February 2020" "fail2ban-server v1.0.1.dev1" "User Commands"
 .SH NAME
 fail2ban-server \- start the server
 .SH SYNOPSIS
 .B fail2ban-server
 [\fI\,OPTIONS\/\fR]
 .SH DESCRIPTION
-Fail2Ban v0.11.2 reads log file that contains password failure report
+Fail2Ban v1.0.1.dev1 reads log file that contains password failure report
 and bans the corresponding IP addresses using firewall rules.
 .SH OPTIONS
 .TP
@@ -1,5 +1,5 @@
 .\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.4.
-.TH FAIL2BAN-TESTCASES "1" "November 2020" "fail2ban-testcases 0.11.2" "User Commands"
+.TH FAIL2BAN-TESTCASES "1" "January 2020" "fail2ban-testcases 1.0.1.dev1" "User Commands"
 .SH NAME
 fail2ban-testcases \- run Fail2Ban unit-tests
 .SH SYNOPSIS
@@ -271,7 +271,7 @@ effective ban duration (in seconds or time abbreviation format).
 time interval (in seconds or time abbreviation format) before the current time where failures will count towards a ban.
 .TP
 .B maxretry
-number of failures that have to occur in the last \fBfindtime\fR seconds to ban then IP.
+number of failures that have to occur in the last \fBfindtime\fR seconds to ban the IP.
 .TP
 .B backend
 backend to be used to detect changes in the logpath.
@@ -487,7 +487,7 @@ is the regex (\fBreg\fRular \fBex\fRpression) that will match failed attempts. T
 .IP
 \fI<CIDR>\fR - helper regex to match CIDR (simple integer form of net-mask).
 .IP
-\fI<SUBNET>\fR - regex to match sub-net adresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional).
+\fI<SUBNET>\fR - regex to match sub-net addresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional).
 .IP
 \fI<F-ID>...</F-ID>\fR - free regex capturing group targeting identifier used for ban (instead of IP address or hostname).
 .IP
@@ -540,7 +540,7 @@ There are several prefixes and words with special meaning that could be specifie
 .IP
 \fI{UNB}\fR - prefix to disable automatic word boundaries in regex.
 .IP
-\fI{NONE}\fR - value would allow to find failures totally without date-time in log message. Filter will use now as a timestamp (or last known timestamp from previous line with timestamp).
+\fI{NONE}\fR - value would allow one to find failures totally without date-time in log message. Filter will use now as a timestamp (or last known timestamp from previous line with timestamp).
 .RE
 .TP
 \fBjournalmatch\fR
Author	SHA1	Message	Date
sebres	095aeda840	Merge branch '0.11'	2021-12-02 18:56:19 +01:00
sebres	80805cabfc	Merge branch '0.11'	2021-11-03 16:01:00 +01:00
Sergey G. Brester	cb042dda7d	Merge pull request #3133 from sylvestre/master Fix typo found by lintian (Debian static analyzer)	2021-10-25 12:10:28 +02:00
Sylvestre Ledru	1cd3118da3	Fix typo found by lintian (Debian static analyzer)	2021-10-23 17:06:13 +02:00
Viktor Szépe	1e8ce20b32	Display ChangeLog as a Markdown document	2021-10-17 19:53:10 +02:00
sebres	5093ff8956	Merge branch '0.11' into master	2021-09-19 18:54:23 +02:00
sebres	ad1f9dc4d0	Merge branch '0.11'	2021-09-08 20:12:48 +02:00
Sergey G. Brester	cb667edf17	Merge pull request #3087 from alexporto2200/patch-3 README: added dependencies to setuptools or distutils (if installing from source)	2021-08-17 16:49:40 +02:00
Sergey G. Brester	ade79635b2	distutils/setuptools only required if installing from source	2021-08-17 16:41:20 +02:00
Alex Porto dos Santos	39fe0bdce6	Update Readme Add python3-setuptools or python-setuptools for dependencies. On some distributions this doesn't come by default, it would be nice to let users know about this in the documentation.	2021-08-17 10:39:04 -03:00
sebres	10cd815525	merge 0.11 to 1.0 (GHSA-m985-3f3v-cwmm)	2021-07-07 12:06:06 +02:00
sebres	579c6a94af	filter.d/postfix.conf: mode `ddos` (and `aggressive`) extended to consider abusive handling of clients hitting command limit (gh-3040)	2021-06-10 15:23:24 +02:00
sebres	43f2923fbd	filter.d/postfix.conf: matches rejects with "undeliverable address" (sender/recipient verification, gh-3039) additionally to "Unknown user"; both are configurable now via extended parameter and can be disabled using `exre-user=` supplied in filter parameters	2021-06-10 15:06:54 +02:00
Sergey G. Brester	bbfff18280	action.d/ufw.conf: amend to #3018 : parameter `kill-mode` extended with conntrack	2021-06-03 12:02:08 +02:00
sebres	c7a86b4616	action.d/firewallcmd-ipset.conf: amend to #2620 : - combines actions `firewallcmd-ipset` and `firewallcmd-ipset-native` (parameter `ipsettype=firewalld`); - IPv6-capability for firewalld ipset; - no internal timeout handling by default; - no permanent rules yet	2021-05-29 22:59:55 +02:00
Sergey G. Brester	2a508da5a0	Merge pull request #2620 from mspolitaev/master Using native firewalld ipset implementation	2021-05-29 21:30:55 +02:00
sebres	38535b0cca	Merge branch '0.11' into master	2021-05-29 21:25:24 +02:00
Sergey G. Brester	3d52fe3e4e	Merge pull request #2679 from mikaku/updated-to-latest-jail.conf Add new jail (and filter) Monitorix	2021-05-27 12:17:16 +02:00
sebres	0a05dbdbfc	Merge branch '0.11' into master	2021-05-25 23:19:25 +02:00
Sergey G. Brester	f07e0f7ade	Merge pull request #2984 from j-marz/zoneminder_filter_update Zoneminder filter update	2021-05-21 13:03:33 +02:00
Sergey G. Brester	ec4e0dd65b	padding with space, prefregex, regex review (simplifying, capture user name, consider possible space char in user name)	2021-05-21 13:00:24 +02:00
j-marz	2367ad115c	fixed typo in comment	2021-05-20 09:15:45 +10:00
Ioannis Cherouvim	5e3de882af	docs: Typo. (#3025 )	2021-05-12 13:18:46 +02:00
Sergey G. Brester	3f9cf27853	filter.d/apache-fakegooglebot.conf: better, more precise regex and datepattern (closes possible weakness like #3013 )	2021-05-11 13:47:48 +02:00
usernamepi	4f8427178a	Missing comment "#" (#3022 ) Missed this ... but the logs showed it.	2021-05-07 18:23:40 +02:00
sebres	6ac2156907	Merge branch '0.11' into master	2021-05-07 01:48:36 +02:00
usernamepi	88f779ed24	ufw.conf, amend to #3018 - add missing option for comment (#3019 )	2021-05-06 23:23:39 +02:00
Sergey G. Brester	5a8f1bceb8	Merge pull request #3018 from usernamepi/master Update ufw.conf: several fixes and enhancements	2021-05-06 22:20:39 +02:00
Sergey G. Brester	2958ad8636	Update ChangeLog	2021-05-06 22:19:38 +02:00
Sergey G. Brester	8f6a8df3a4	added new options `kill-mode` and `kill`, which makes the drop of all connections optional	2021-05-06 21:47:06 +02:00
Sergey G. Brester	5debaa4cac	option "add", can be set to "insert <num>" instead of prepend (customization or backwards compat)	2021-05-06 20:23:58 +02:00
usernamepi	e4e7a83cff	Update ufw.conf Prerequisites: * The ss command is available, kernel is compiled with option CONFIG_INET_DIAG_DESTROY. * Ufw version is => 0.36 (released in 2018) * Now using "prepend" instead of "insert" to be able to handle IPv6 addresses correctly. The current action will fail for IPv6 addresses. * Now application names containing a space should handled correctly, solves https://github.com/fail2ban/fail2ban/pull/1532 * Now closing IPv4 and IPv6 connections (if any) from the ip that is being banned. The current action will leave them open. Using ss to accomplish this. For this to work the kernel needs to be compiled with the CONFIG_INET_DIAG_DESTROY option. My system apparently is compiled that way.	2021-05-06 13:44:36 +02:00
sebres	71ce548117	Merge branch '0.11'	2021-04-27 14:05:53 +02:00
Sergey G. Brester	ab0847e2d5	more precise anchored RE (also combining all 3 REs in a single regex)	2021-04-14 13:06:58 +02:00
Jordi Sanfeliu	b6fac90b5a	Update monitorix	2021-04-14 11:46:23 +02:00
Jordi Sanfeliu	63b3f39adc	Rename monitorix-httpd to monitorix	2021-04-14 11:30:48 +02:00
Jordi Sanfeliu	8b741129a5	Create monitorix-httpd	2021-04-14 11:26:52 +02:00
Jordi Sanfeliu	7d173b7ce0	Merge branch 'master' into updated-to-latest-jail.conf	2021-04-13 20:24:08 +02:00
sebres	6893d5a8b7	Merge remote-tracking branch 'remotes/gh-upstream/0.11' into master	2021-04-11 19:05:02 +02:00
Sergey G. Brester	ff09c52927	Merge pull request #2881 from stepodev/master `filter.d/nginx-http-auth.conf` - extended with parameter mode, so additionally to `auth` (or `normal`) mode `fallback` (or combined as `aggressive`) can find SSL errors while SSL handshaking	2021-04-04 00:07:03 +02:00
Sergey G. Brester	dda70d60c0	Merge branch 'master' into master	2021-04-04 00:04:08 +02:00
Sergey G. Brester	c5d43d7573	Update ChangeLog	2021-04-04 00:00:59 +02:00
Michele Mondelli	7579072e3b	docs: fix typos	2021-04-03 23:49:23 +02:00
Sergey G. Brester	4eba9f2a4b	Merge pull request #2950 from sunweaver/pr/scanlogd-filter Add support for filtering out detected port scans via scanlogd.	2021-04-03 23:36:14 +02:00
Sergey G. Brester	2d51240b3e	correction for default log interpolation and added allports banaction	2021-04-03 23:33:49 +02:00
Sergey G. Brester	977dfe4bd7	small amend: sport after saddr is optional format of message: saddr[:sport] to daddr [and others,] ports port[, port...], ..., flags[, TOS TOS][, TTL TTL] @HH:MM:SS	2021-04-03 23:29:16 +02:00
Sergey G. Brester	14edeed310	fixed regex (don't need to match whole line, e. g. every port etc)	2021-04-03 23:24:55 +02:00
Sergey G. Brester	0c4d356d11	added test log-file	2021-04-03 23:10:51 +02:00
Sergey G. Brester	080dd12288	Merge pull request #2965 from oukb/patch-1 nsd.conf: fix for the current log format	2021-04-03 21:02:03 +02:00
Sergey G. Brester	a838deba7f	restore anchor (e. g. catch all in the middle), dot is optional now, RE rewritten a bit more precise	2021-04-03 21:00:14 +02:00
Sergey G. Brester	1215cb28ac	Update nsd	2021-04-03 20:58:26 +02:00
sebres	d445b5671d	Merge pull request #2642 from rolschewsky/mssql (and amend from sebres/mssql)	2021-04-03 20:28:18 +02:00
sebres	7f38b80d35	precise regex (left anchor and fewer catch-all's); fixed tests (added failJSON and more tests for some corner-cases around new RE)	2021-04-03 20:16:47 +02:00
Rüdiger Olschewsky	9eaa2322b0	Filter and Defaults for Microsoft SQL Server	2021-04-03 19:30:29 +02:00
Markus Felten	5aa20c30d8	fix: add journalmatch to nginx filters	2021-04-03 19:20:50 +02:00
j-marz	5d8f500471	updated formatting to pass tests	2021-03-29 08:36:53 +11:00
j-marz	2686811593	Updated zoneminder filter Support new log format, ERR instead of WAR. Add detection of non-existent user login attempts	2021-03-28 21:19:10 +11:00
sebres	80a33b1dee	Merge branch '0.11'	2021-03-25 12:14:11 +01:00
sebres	b259e81911	test-suite: skip testFQDN if no network	2021-03-25 12:13:46 +01:00
sebres	d8e450cf12	Merge branch 'fix-readline-multibyte'	2021-03-25 12:13:18 +01:00
sebres	4b17dddc23	update ChangeLog	2021-03-25 12:07:34 +01:00
sebres	ccf4f3a07d	amend with common log-file iterator in fail2ban-regex and test-suite (in sample regex factory also)	2021-03-25 12:07:31 +01:00
sebres	9659033523	fail2ban-regex: reimplemented log-file iterator - uses FileContainer facilities now instead of direct read from file and decode; fail2banregextestcase.py extended to cover proper line-ending handling by interim NL char as part of multi-byte encodings (utf-16be, utf-16le)	2021-03-25 12:07:29 +01:00
sebres	cbac7c176a	readline fixed to consider interim new-line character as part of code point in multi-byte logs (e. g. unicode: utf-16be, utf-16le); suppress warning "Error decoding line" for incomplete line (produced by not fully read multi-byte new-line character at end of data); added test coverage for such logs	2021-03-25 12:07:26 +01:00
sebres	996920cdaa	in operation mode the filter reads only complete lines (ended with new-line) now, otherwise it would wait for end of line (for its completion)	2021-03-22 01:17:26 +01:00
sebres	343ccd7e8a	small optimization	2021-03-21 23:35:38 +01:00
sebres	9bdc4be6cc	stability: better recognition of rotation (e. g. on hash collision, consider current size and last known position now), no hash of empty file (or not fulfilled line), etc; performance: avoid unnecessary seek to start of file and hash calculation - now it occurs only if file really rotated (ino changing or size shrinking), otherwise not earlier than in 30 seconds; avoid unneeded log-rotation in tests	2021-03-21 23:35:09 +01:00
sebres	725354c793	action info extended with new members for jail info (usable as tags in command actions): `jail.found`, `jail.found_total` - current and total found failures `jail.banned`, `jail.banned_total` - current and total bans closes #10	2021-03-20 22:33:31 +01:00
oukb	529866b2bb	nsd.conf: fix for the current log format New nsd 4.3.5 log format: \| [2021-03-05 05:25:14.562] nsd[160800]: info: axfr for example.com. from 192.35.168.32 refused, no acl matches \| [2021-03-06 05:24:33.223] nsd[356033]: info: axfr for localhost. from 192.35.168.160 refused, no acl matches \| [2021-03-07 05:23:26.641] nsd[547893]: info: axfr for example.com. from 192.35.168.64 refused, no acl matches \| [2021-03-08 05:18:54.067] nsd[739606]: info: axfr for example.com. from 192.35.168.32 refused, no acl matches	2021-03-08 19:14:28 +03:00
Mike Gabriel	f15ed35619	config/: Add support for filtering out detected port scans via scanlogd.	2021-03-05 16:35:13 +01:00
Sergey G. Brester	08393f9d82	Update filter_request.md	2021-03-03 20:28:27 +01:00
sebres	df5e024fb8	new issue templates	2021-03-03 20:16:34 +01:00
sebres	fb08534ed7	Merge branch '0.11'	2021-03-03 18:17:35 +01:00
sebres	884cbbd6e1	Merge branch '0.11'	2021-02-17 19:04:23 +01:00
sebres	abc5a4e062	ChangeLog (#2742 )	2021-02-17 19:02:22 +01:00
Sergey G. Brester	a2f0dbad87	Merge pull request #2742 from aresxc/patch-1 Update drupal-auth.conf	2021-02-11 19:10:55 +01:00
Sergey G. Brester	f4f92aa72d	more tests covering different cases, injections attempt etc	2021-02-11 18:56:53 +01:00
Sergey G. Brester	d678440658	more precise RE (avoids weakness with catch-all's and is injection safe)	2021-02-11 18:32:32 +01:00
sebres	ea26509594	Merge branch '0.11'	2021-02-03 14:59:00 +01:00
sebres	409348394e	Merge pull request #2913 Add transport to asterisk RE	2021-02-03 12:41:29 +01:00
Brian J. Murrell	7f185a828e	Update date in failJSON The date format in failJSON is specific, so convert the date to use that format.	2021-01-31 15:22:16 +01:00
Brian J. Murrell	69c96c00c0	Log entries for updated failregex Add a sample failregex. Signed-off-by: Brian J. Murrell <brian@interlinx.bc.ca>	2021-01-31 15:22:16 +01:00
Brian J. Murrell	dc4ee5aa47	Add transport to asterisk RE Call rejection messages from Asterisk can have the transport prefixed to the IP address. Signed-off-by: Brian J. Murrell <brian@interlinx.bc.ca>	2021-01-31 15:22:16 +01:00
sebres	21dd317870	Merge branch '0.11'	2021-01-21 19:13:13 +01:00
Michael Haro	2781e7b9d1	Add blank line so markdown renders correctly	2021-01-09 13:38:33 +01:00
Sergey G. Brester	2d4c3edeb6	Merge pull request #2906 from fail2ban/ci-py-3.10 CI: add python 3.10 (alpha) to github actions	2021-01-06 14:45:11 +01:00
Sergey G. Brester	c9907bef0d	CI: 3.10.0-alpha.4	2021-01-06 14:37:34 +01:00
Sergey G. Brester	097d9ea7fe	try wildcard (3.10 is still alpha)	2021-01-06 14:36:09 +01:00
Sergey G. Brester	3c8aa0e6ba	amend with correct yaml notation (float vs string)	2021-01-06 14:34:08 +01:00
Sergey G. Brester	0a7f5b7b5c	CI: add python 3.10 to github actions	2021-01-06 14:29:18 +01:00
sebres	e60b10392d	Merge branch '0.11' into master	2021-01-04 02:50:33 +01:00
sebres	2c60d08b28	Merge '0.11' (fix gh-2899) into master	2020-12-29 21:27:02 +01:00
Sergey G. Brester	6b5cc2dee8	Merge pull request #2898 from defanor/postscreen-pregreet-hangup Handle postscreen's PREGREET and HANGUP messages in ddos and aggressive mode	2020-12-29 13:21:47 +01:00
defanor	ba7daef86c	Handle postscreen's PREGREET and HANGUP messages Provoking those seems to be a popular activity among spammers.	2020-12-24 17:29:09 +03:00
Tim Gates	3bb19ecc9d	docs: fix simple typo, litle -> little There is a small typo in fail2ban/client/fail2banclient.py, fail2ban/server/observer.py. Should read `little` rather than `litle`.	2020-12-23 12:59:03 +01:00
sebres	7d256681a4	Merge branch '0.11' into master	2020-12-22 20:41:41 +01:00
stepodev	cecc3d62ff	add mode explanation to nginx-http-auth in jail.conf	2020-11-30 12:26:32 +01:00
stepodev	d0ba27cf46	move nginx-tls-fallback rules to nginx-http-auth	2020-11-30 12:14:49 +01:00
Sergey G. Brester	d959f6d199	Update nginx-tls-fallback.conf more precise and conclusive regex without catch-all's	2020-11-26 12:25:32 +01:00
stepodev	c0256724a7	fix monitoring wrong error log. was access log, should be error.log	2020-11-25 21:30:21 +01:00
stepodev	27c40a77a3	add nginx-tls-downgrade	2020-11-25 20:59:43 +01:00
sebres	b8e8a87ee9	small amend to `0f27a8add8` - datedetector, strptime: token `%Z` recognizes all known zone abbreviation besides Z, GMT, UTC correctly, if it is matching (`%z` remains unchanged for backwards-compatibility, see comment in code); test cases fixed (PDT zone will be found now).	2020-11-24 20:23:39 +01:00
sebres	0f27a8add8	datedetector, strptime: token `%Z` must recognize zone abbreviation `Z` (GMT/UTC) also, similar to `%z`; more test cases added.	2020-11-24 19:28:46 +01:00
sebres	a03109d096	Merge branch '0.11' into master (0.11.2 released)	2020-11-24 12:41:10 +01:00
Sergey G. Brester	753fff9c15	amend to #2750 , add jail for new filter nginx-bad-request	2020-11-23 18:38:41 +01:00
Sergey G. Brester	071048b8f2	Merge pull request #2750 from janprzy/master Added filter nginx-bad-request	2020-11-23 18:28:07 +01:00
sebres	f5ea40c7da	Merge branch '0.11' into master	2020-11-11 11:19:05 +01:00
sebres	9f635c66fb	Merge branch '0.11' into master	2020-11-09 12:45:54 +01:00
sebres	690ad20958	Merge branch '0.11' into master	2020-11-05 16:16:17 +01:00
Sergey G. Brester	9d77fb2b4c	1st try of GH actions flow (CI only, no coverage atm)	2020-11-03 15:38:06 +01:00
sebres	960e30cfcd	Merge branch '0.11'	2020-09-23 19:41:04 +02:00
sebres	8bc7623388	Merge branch '0.11'	2020-08-26 13:49:41 +02:00
Sergey G. Brester	57caf8ec90	Merge pull request #2768 from TorontoMedia/patch Remove duplicate method and rename invalid parameter	2020-08-24 16:39:25 +02:00
sebres	7327fee2c8	Merge branch '0.11'	2020-08-24 16:33:30 +02:00
sebres	1ea36c3045	Merge branch '0.11'	2020-08-04 17:19:51 +02:00
TorontoMedia	fd25c4cbb8	Remove duplicate method and rename invalid parameter	2020-06-28 12:58:41 -04:00
Jan Przybylak	56fefe9240	Added test file "nginx-bad-request" I tested with `./fail2ban-testcases testSampleRegex`, which did not return any errors.	2020-06-21 18:25:27 +02:00
Jan Przybylak	a5ab4406d8	Removed unnecessary escape sequence This commit also contains changes to match requests that are 100% empty (by using "*" instead of "+" in the regex)	2020-06-21 18:24:09 +02:00
Jan Przybylak	d7ef5d166d	Removed vulnerable catchall & anchor	2020-06-11 16:44:48 +02:00
sebres	78fc99ec7e	Merge branch '0.11'	2020-06-11 12:52:34 +02:00
Jan Przybylak	3c83c19070	Added filter nginx-bad-request	2020-06-06 19:51:46 +02:00
aresdr	412120ac3c	Update drupal-auth.conf Small fix for Drupal 8. D8 uses "Login attempt failed from" while D7 uses "Login attempt failed for". The referer part is a must currently, but some requests did not have one and are not failing.	2020-05-30 15:25:31 -07:00
sebres	6cff2bb007	Merge branch '0.11'	2020-05-25 19:09:29 +02:00
Jordi Sanfeliu	ede2009708	added new jail (and filter) Monitorix	2020-04-03 12:52:19 +02:00
sebres	6e570b8644	Merge branch '0.11'	2020-03-13 23:23:32 +01:00
sebres	9905904bba	Merge branch '0.11'	2020-03-13 22:43:22 +01:00
sebres	4c22d4a801	Merge branch '0.11'	2020-03-13 17:47:03 +01:00
sebres	8b43d54878	Merge branch '0.11'	2020-03-05 14:32:42 +01:00
sebres	ef1eaf9b37	Merge branch '0.11'	2020-02-25 17:18:50 +01:00
sebres	35591db3e8	Merge branch '0.11'	2020-02-14 12:14:51 +01:00
sebres	65ce64ba6d	Merge branch '0.11'	2020-02-07 13:57:00 +01:00
Mihail Politaev	303861d7c7	Using native firewalld ipset implementation By creating additional action file firewallcmd-ipset-native.conf	2020-01-30 21:17:32 +02:00
sebres	ccbc0c7405	Merge branch '0.11'	2020-01-30 10:23:14 +01:00
sebres	7f1b578af4	Merge branch 'gh-488-check-on-error'	2020-01-16 16:52:44 +01:00
sebres	8cbc1e0ebb	ChangeLog (change actioncheck behavior)	2020-01-16 16:51:57 +01:00
sebres	546fff8dc0	Merge branch '0.11'	2020-01-16 16:25:20 +01:00
sebres	ed4b5b8bdc	change actioncheck behavior (on error only, gh-488)	2020-01-15 14:54:31 +01:00
sebres	94b64055e1	Merge branch '0.11'	2020-01-15 14:53:28 +01:00
Sergey G. Brester	3f5c382a98	readme: amend to bump version (v.1.0.1 is master now)	2020-01-14 20:55:24 +01:00
sebres	bb0f732ae6	version bump (master is 1.0.x-dev now)	2020-01-14 20:38:26 +01:00