merge-point for 0.11.1

don't error on clean-up of missing files
debian\rules: better clean-up of non-debian paths includes
2020-11-20 17:23:15 +01:00 · 2020-11-20 16:58:20 +01:00 · 2020-11-20 15:25:42 +01:00 · 2020-09-24 08:42:58 +02:00 · 2020-09-24 08:42:58 +02:00 · 2020-09-24 08:42:58 +02:00
158 changed files with 5436 additions and 2494 deletions
@@ -1,66 +0,0 @@
-name: CI
-
-# Controls when the action will run. Triggers the workflow on push or pull request
-# events but only for the master branch
-on:
-  push:
-    paths-ignore:
-    - 'doc/**'
-    - 'files/**'
-    - 'man/**'
-  pull_request:
-    paths-ignore:
-    - 'doc/**'
-    - 'files/**'
-    - 'man/**'
-
-# A workflow run is made up of one or more jobs that can run sequentially or in parallel
-jobs:
-  # This workflow contains a single job called "build"
-  build:
-    # The type of runner that the job will run on
-    runs-on: ubuntu-20.04
-    strategy:
-      matrix:
-        python-version: [2.7, 3.5, 3.6, 3.7, 3.8, 3.9, pypy2, pypy3]
-      fail-fast: false
-    # Steps represent a sequence of tasks that will be executed as part of the job
-    steps:
-      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
-      - uses: actions/checkout@v2
-
-      - name: Set up Python
-        uses: actions/setup-python@v2
-        with:
-          python-version: ${{ matrix.python-version }}
-      
-      - name: Python version
-        run: |
-          F2B_PY=$(python -c "import sys; print(sys.version)")
-          echo "Python: ${{ matrix.python-version }} -- $F2B_PY"
-          F2B_PY=${F2B_PY:0:1}
-          echo "Set F2B_PY=$F2B_PY"
-          echo "F2B_PY=$F2B_PY" >> $GITHUB_ENV
-
-      - name: Install dependencies
-        run: |
-          python -m pip install --upgrade pip
-          if [[ "$F2B_PY" = 3 ]] && ! command -v 2to3x -v 2to3 > /dev/null; then
-            pip install 2to3
-          fi
-          pip install systemd-python || echo 'systemd not available'
-          pip install pyinotify || echo 'inotify not available'
-
-      - name: Before scripts
-        run: |
-          cd "$GITHUB_WORKSPACE"
-          # Manually execute 2to3 for now
-          if [[ "$F2B_PY" = 3 ]]; then echo "2to3 ..." && ./fail2ban-2to3; fi
-          # (debug) output current preferred encoding:
-          python -c 'import locale, sys; from fail2ban.helpers import PREFER_ENC; print(PREFER_ENC, locale.getpreferredencoding(), (sys.stdout and sys.stdout.encoding))'
-      
-      - name: Test suite
-        run: if [[ "$F2B_PY" = 2 ]]; then python setup.py test; else python bin/fail2ban-testcases --verbosity=2; fi
-        
-      #- name: Test initd scripts
-      #  run: shellcheck -s bash -e SC1090,SC1091 files/debian-initd
@@ -18,14 +18,14 @@ matrix:
  - python: 2.7
    name: 2.7 (xenial)
  - python: pypy
+    dist: trusty
  - python: 3.3
    dist: trusty
  - python:  3.4
  - python:  3.5
  - python:  3.6
  - python:  3.7
-  - python:  3.8
-  - python:  3.9-dev
+  - python:  3.8-dev
  - python:  pypy3.5
 before_install:
  - echo "running under $TRAVIS_PYTHON_VERSION"
@@ -69,8 +69,8 @@ script:
  - if [[ "$F2B_PY" = 3 ]]; then coverage run bin/fail2ban-testcases --verbosity=2; fi
  # Use $VENV_BIN (not python) or else sudo will always run the system's python (2.7)
  - sudo $VENV_BIN/pip install .
-  # Doc files should get installed on Travis under Linux (some builds/python's seem to use another path segment)
-  - test -e /usr/share/doc/fail2ban/FILTERS && echo 'found' || echo 'not found'
+  # Doc files should get installed on Travis under Linux (python >= 3.8 seem to use another path segment)
+  - if [[ $TRAVIS_PYTHON_VERSION < 3.8 ]]; then test -e /usr/share/doc/fail2ban/FILTERS; fi
  # Test initd script
  - shellcheck -s bash -e SC1090,SC1091 files/debian-initd
 after_success:
@@ -6,91 +6,60 @@
 Fail2Ban: Changelog
 ===================

-ver. 0.10.6 (2020/11/23) - just-what-the-doctor-ordered
+ver. 0.11.1 (2020/01/11) - this-is-the-way
 -----------

-### Incompatibility list (v.0.10 compared to v.0.9):
-* Filter (or `failregex`) internal capture-groups:
+### Compatibility:
+* to v.0.10:
+  - 0.11 is totally compatible to 0.10 (configuration- and API-related stuff), but the database
+    got some new tables and fields (auto-converted during the first start), so once updated to 0.11, you
+    have to remove the database /var/lib/fail2ban/fail2ban.sqlite3 (or its different to 0.10 schema)
+    if you would need to downgrade to 0.10 for some reason.
+* to v.0.9:
+  - Filter (or `failregex`) internal capture-groups:

-  - If you've your own `failregex` or custom filters using conditional match `(?P=host)`, you should
-    rewrite the regex like in example below resp. using `(?:(?P=ip4)|(?P=ip6)` instead of `(?P=host)`
-    (or `(?:(?P=ip4)|(?P=ip6)|(?P=dns))` corresponding your `usedns` and `raw` settings).
+    * If you've your own `failregex` or custom filters using conditional match `(?P=host)`, you should
+      rewrite the regex like in example below resp. using `(?:(?P=ip4)|(?P=ip6)` instead of `(?P=host)`
+      (or `(?:(?P=ip4)|(?P=ip6)|(?P=dns))` corresponding your `usedns` and `raw` settings).

-    Of course you can always define your own capture-group (like below `_cond_ip_`) to do this.
-    ```
-    testln="1500000000 failure from 192.0.2.1: bad host 192.0.2.1"
-    fail2ban-regex "$testln" "^\s*failure from (?P<_cond_ip_><HOST>): bad host (?P=_cond_ip_)$"
-    ```
-  - New internal groups (currently reserved for internal usage):
-    `ip4`, `ip6`, `dns`, `fid`, `fport`, additionally `user` and another captures in lower case if
-    mapping from tag `<F-*>` used in failregex (e. g. `user` by `<F-USER>`).
+      Of course you can always define your own capture-group (like below `_cond_ip_`) to do this.
+      ```
+      testln="1500000000 failure from 192.0.2.1: bad host 192.0.2.1"
+      fail2ban-regex "$testln" "^\s*failure from (?P<_cond_ip_><HOST>): bad host (?P=_cond_ip_)$"
+      ```
+    * New internal groups (currently reserved for internal usage):
+      `ip4`, `ip6`, `dns`, `fid`, `fport`, additionally `user` and another captures in lower case if
+      mapping from tag `<F-*>` used in failregex (e. g. `user` by `<F-USER>`).

-* v.0.10 uses more precise date template handling, that can be theoretically incompatible to some
-  user configurations resp. `datepattern`.
+  - v.0.10 and 0.11 use more precise date template handling, that can be theoretically incompatible to some
+    user configurations resp. `datepattern`.

-* Since v0.10 fail2ban supports the matching of IPv6 addresses, but not all ban actions are
-  IPv6-capable now.
+  - Since v0.10 fail2ban supports the matching of IPv6 addresses, but not all ban actions are
+    IPv6-capable now.

 ### Fixes
-* [stability] prevent race condition - no ban if filter (backend) is continuously busy if
-  too many messages will be found in log, e. g. initial scan of large log-file or journal (gh-2660)
-* pyinotify-backend sporadically avoided initial scanning of log-file by start
-* python 3.9 compatibility (and Travis CI support)
-* restoring a large number (500+ depending on files ulimit) of current bans when using PyPy fixed
-* manual ban is written to database, so can be restored by restart (gh-2647)
-* `jail.conf`: don't specify `action` directly in jails (use `action_` or `banaction` instead)
-* no mails-action added per default anymore (e. g. to allow that `action = %(action_mw)s` should be specified
-  per jail or in default section in jail.local), closes gh-2357
-* ensure we've unique action name per jail (also if parameter `actname` is not set but name deviates from standard name, gh-2686)
-* don't use `%(banaction)s` interpolation because it can be complex value (containing `[...]` and/or quotes), 
-  so would bother the action interpolation
-* fixed type conversion in config readers (take place after all interpolations get ready), that allows to 
-  specify typed parameters variable (as substitutions) as well as to supply it in other sections or as init parameters.
-* `action.d/*-ipset*.conf`: several ipset actions fixed (no timeout per default anymore), so no discrepancy
-  between ipset and fail2ban (removal from ipset will be managed by fail2ban only, gh-2703)
-* `action.d/cloudflare.conf`: fixed `actionunban` (considering new-line chars and optionally real json-parsing
-   with `jq`, gh-2140, gh-2656)
-* `action.d/nftables.conf` (type=multiport only): fixed port range selector, replacing `:` with `-` (gh-2763)
-* `action.d/firewallcmd-*.conf` (multiport only): fixed port range selector, replacing `:` with `-` (gh-2821)
-* `action.d/bsd-ipfw.conf`: fixed selection of rule-no by large list or initial `lowest_rule_num` (gh-2836)
-* `filter.d/common.conf`: avoid substitute of default values in related `lt_*` section, `__prefix_line`
-  should be interpolated in definition section (inside the filter-config, gh-2650)
-* `filter.d/dovecot.conf`: 
-  - add managesieve and submission support (gh-2795);
-  - accept messages with more verbose logging (gh-2573);
-* `filter.d/courier-smtp.conf`: prefregex extended to consider port in log-message (gh-2697)
-* `filter.d/traefik-auth.conf`: filter extended with parameter mode (`normal`, `ddos`, `aggressive`) to handle 
-  the match of username differently (gh-2693):
-  - `normal`: matches 401 with supplied username only
-  - `ddos`: matches 401 without supplied username only
-  - `aggressive`: matches 401 and any variant (with and without username)
-* `filter.d/sshd.conf`: normalizing of user pattern in all RE's, allowing empty user (gh-2749)
+* purge database will be executed now (within observer).
+* restoring currently banned ip after service restart fixed 
+  (now < timeofban + bantime), ignore old log failures (already banned)
+* upgrade database: update new created table `bips` with entries from table `bans` (allows restore
+  current bans after upgrade from version <= 0.10)

-### New Features and Enhancements
-* fail2ban-regex:
-  - speedup formatted output (bypass unneeded stats creation)
-  - extended with prefregex statistic
-  - more informative output for `datepattern` (e. g. set from filter) - pattern : description
-* parsing of action in jail-configs considers space between action-names as separator also
-  (previously only new-line was allowed), for example `action = a b` would specify 2 actions `a` and `b`
-* new filter and jail for GitLab recognizing failed application logins (gh-2689)
-* new filter and jail for Grafana recognizing failed application logins (gh-2855)
-* new filter and jail for SoftEtherVPN recognizing failed application logins (gh-2723)
-* `filter.d/guacamole.conf` extended with `logging` parameter to follow webapp-logging if it's configured (gh-2631)
-* `filter.d/bitwarden.conf` enhanced to support syslog (gh-2778)
-* introduced new prefix `{UNB}` for `datepattern` to disable word boundaries in regex;
-* datetemplate: improved anchor detection for capturing groups `(^...)`;
-* datepattern: improved handling with wrong recognized timestamps (timezones, no datepattern, etc)
-  as well as some warnings signaling user about invalid pattern or zone (gh-2814):
-  - filter gets mode in-operation, which gets activated if filter starts processing of new messages;
-    in this mode a timestamp read from log-line that appeared recently (not an old line), deviating too much
-    from now (up too 24h), will be considered as now (assuming a timezone issue), so could avoid unexpected 
-    bypass of failure (previously exceeding `findtime`);
-  - better interaction with non-matching optional datepattern or invalid timestamps;
-  - implements special datepattern `{NONE}` - allow to find failures totally without date-time in log messages,
-    whereas filter will use now as timestamp (gh-2802)
-* performance optimization of `datepattern` (better search algorithm in datedetector, especially for single template);
-* fail2ban-client: extended to unban IP range(s) by subnet (CIDR/mask) or hostname (DNS), gh-2791;
+### New Features
+* Increment ban time (+ observer) functionality introduced.
+* Database functionality extended with bad ips.
+* New tags (usable in actions):
+  - `<bancount>` - ban count of this offender if known as bad (started by 1 for unknown)
+  - `<bantime>` - current ban-time of the ticket (prolongation can be retarded up to 10 sec.)
+* Introduced new action command `actionprolong` to prolong ban-time (e. g. set new timeout if expected);
+  Several actions (like ipset, etc.) rewritten using net logic with `actionprolong`.
+  Note: because ban-time is dynamic, it was removed from jail.conf as timeout argument (check jail.local).
+
+### Enhancements
+* algorithm of restore current bans after restart changed: update the restored ban-time (and therefore 
+  end of ban) of the ticket with ban-time of jail (as maximum), for all tickets with ban-time greater
+  (or persistent); not affected if ban-time of the jail is unchanged between stop/start.
+* added new setup-option `--without-tests` to skip building and installing of tests files (gh-2287).
+* added new command `fail2ban-client get <JAIL> banip ?sep-char|--with-time?` to get the banned ip addresses (gh-1916).


 ver. 0.10.5 (2020/01/10) - deserve-more-respect-a-jedis-weapon-must
@@ -471,9 +440,14 @@ TODO: implementing of options resp. other tasks from PR #1346
  - `<fid>` - failure identifier (if raw resp. failures without IP address)
  - `<ip-rev>` - PTR reversed representation of IP address
  - `<ip-host>` - host name of the IP address
+  - `<bancount>` - ban count of this offender if known as bad (started by 1 for unknown)
+  - `<bantime>` - current ban-time of the ticket (prolongation can be retarded up to 10 sec.)
  - `<F-...>` - interpolates to the corresponding filter group capture `...`
  - `<fq-hostname>` - fully-qualified name of host (the same as `$(hostname -f)`)
  - `<sh-hostname>` - short hostname (the same as `$(uname -n)`)
+* Introduced new action command `actionprolong` to prolong ban-time (e. g. set new timeout if expected);
+  Several actions (like ipset, etc.) rewritten using net logic with `actionprolong`.
+  Note: because ban-time is dynamic, it was removed from jail.conf as timeout argument (check jail.local).
 * Allow to use filter options by `fail2ban-regex`, example:
  fail2ban-regex text.log "sshd[mode=aggressive]"
 * Samples test case factory extended with filter options - dict in JSON to control 
@@ -547,6 +521,9 @@ ver. 0.10.0-alpha-1 (2016/07/14) - ipv6-support-etc
 * testSocket: sporadical bug repaired - wait for server thread starts a socket (listener)
 * testExecuteTimeoutWithNastyChildren: sporadical bug repaired - wait for pid file inside bash,
  kill tree in any case (gh-1155)
+* purge database will be executed now (within observer).
+* restoring currently banned ip after service restart fixed 
+  (now < timeofban + bantime), ignore old log failures (already banned)
 * Fixed high-load of pyinotify-backend,
  see https://github.com/fail2ban/fail2ban/issues/885#issuecomment-248964591
 * Database: stability fix - repack cursor iterator as long as locked
@@ -584,6 +561,9 @@ ver. 0.10.0-alpha-1 (2016/07/14) - ipv6-support-etc
    - new conditional section functionality used in config resp. includes:
      - [Init?family=inet4] - IPv4 qualified hosts only
      - [Init?family=inet6] - IPv6 qualified hosts only
+* Increment ban time (+ observer) functionality introduced.
+  Thanks Serg G. Brester (sebres)
+* Database functionality extended with bad ips.
 * New reload functionality (now totally without restart, unbanning/rebanning, etc.),
  see gh-1557
 * Several commands extended and new commands introduced:
@@ -100,8 +100,6 @@ config/filter.d/exim.conf
 config/filter.d/exim-spam.conf
 config/filter.d/freeswitch.conf
 config/filter.d/froxlor-auth.conf
-config/filter.d/gitlab.conf
-config/filter.d/grafana.conf
 config/filter.d/groupoffice.conf
 config/filter.d/gssftpd.conf
 config/filter.d/guacamole.conf
@@ -141,7 +139,6 @@ config/filter.d/sendmail-auth.conf
 config/filter.d/sendmail-reject.conf
 config/filter.d/sieve.conf
 config/filter.d/slapd.conf
-config/filter.d/softethervpn.conf
 config/filter.d/sogo-auth.conf
 config/filter.d/solid-pop3d.conf
 config/filter.d/squid.conf
@@ -210,6 +207,7 @@ fail2ban/server/jail.py
 fail2ban/server/jails.py
 fail2ban/server/jailthread.py
 fail2ban/server/mytime.py
+fail2ban/server/observer.py
 fail2ban/server/server.py
 fail2ban/server/strptime.py
 fail2ban/server/ticket.py
@@ -229,8 +227,6 @@ fail2ban/tests/clientreadertestcase.py
 fail2ban/tests/config/action.d/action.conf
 fail2ban/tests/config/action.d/brokenaction.conf
 fail2ban/tests/config/fail2ban.conf
-fail2ban/tests/config/filter.d/checklogtype.conf
-fail2ban/tests/config/filter.d/checklogtype_test.conf
 fail2ban/tests/config/filter.d/simple.conf
 fail2ban/tests/config/filter.d/test.conf
 fail2ban/tests/config/filter.d/test.local
@@ -266,10 +262,9 @@ fail2ban/tests/files/config/apache-auth/digest_wrongrelm/.htpasswd
 fail2ban/tests/files/config/apache-auth/noentry/.htaccess
 fail2ban/tests/files/config/apache-auth/README
 fail2ban/tests/files/database_v1.db
+fail2ban/tests/files/database_v2.db
 fail2ban/tests/files/filter.d/substition.conf
 fail2ban/tests/files/filter.d/testcase01.conf
-fail2ban/tests/files/filter.d/testcase02.conf
-fail2ban/tests/files/filter.d/testcase02.local
 fail2ban/tests/files/filter.d/testcase-common.conf
 fail2ban/tests/files/ignorecommand.py
 fail2ban/tests/files/logs/3proxy
@@ -304,8 +299,6 @@ fail2ban/tests/files/logs/exim
 fail2ban/tests/files/logs/exim-spam
 fail2ban/tests/files/logs/freeswitch
 fail2ban/tests/files/logs/froxlor-auth
-fail2ban/tests/files/logs/gitlab
-fail2ban/tests/files/logs/grafana
 fail2ban/tests/files/logs/groupoffice
 fail2ban/tests/files/logs/gssftpd
 fail2ban/tests/files/logs/guacamole
@@ -343,7 +336,6 @@ fail2ban/tests/files/logs/sendmail-auth
 fail2ban/tests/files/logs/sendmail-reject
 fail2ban/tests/files/logs/sieve
 fail2ban/tests/files/logs/slapd
-fail2ban/tests/files/logs/softethervpn
 fail2ban/tests/files/logs/sogo-auth
 fail2ban/tests/files/logs/solid-pop3d
 fail2ban/tests/files/logs/squid
@@ -376,6 +368,7 @@ fail2ban/tests/files/zzz-sshd-obsolete-multiline.log
 fail2ban/tests/filtertestcase.py
 fail2ban/tests/__init__.py
 fail2ban/tests/misctestcase.py
+fail2ban/tests/observertestcase.py
 fail2ban/tests/samplestestcase.py
 fail2ban/tests/servertestcase.py
 fail2ban/tests/sockettestcase.py
@@ -2,7 +2,7 @@
                        / _|__ _(_) |_  ) |__  __ _ _ _  
                       |  _/ _` | | |/ /| '_ \/ _` | ' \ 
                       |_| \__,_|_|_/___|_.__/\__,_|_||_|
-                       v0.10.3.dev1            20??/??/??
+                       v0.11.0.dev1            20??/??/??

 ## Fail2Ban: ban hosts that cause multiple authentication errors

@@ -18,11 +18,12 @@ attempts, it cannot eliminate the risk presented by weak authentication.
 Set up services to use only two factor, or public/private authentication
 mechanisms if you really want to protect services.
     
-<img src="http://www.worldipv6launch.org/wp-content/themes/ipv6/downloads/World_IPv6_launch_logo.svg" height="52pt"/> | Since v0.10 fail2ban supports the matching of the IPv6 addresses.
+<img src="http://www.worldipv6launch.org/wp-content/themes/ipv6/downloads/World_IPv6_launch_logo.svg" height="52pt"/> | Since v0.10 fail2ban supports the matching of IPv6 addresses.
 ------|------

 This README is a quick introduction to Fail2Ban. More documentation, FAQ, and HOWTOs
-to be found on fail2ban(1) manpage, [Wiki](https://github.com/fail2ban/fail2ban/wiki)
+to be found on fail2ban(1) manpage, [Wiki](https://github.com/fail2ban/fail2ban/wiki),
+[Developers documentation](https://fail2ban.readthedocs.io/)
 and the website: https://www.fail2ban.org

 Installation:
@@ -45,10 +46,16 @@ Optional:

 To install:

-    tar xvfj fail2ban-0.10.3.tar.bz2
-    cd fail2ban-0.10.3
-    python setup.py install
+    tar xvfj fail2ban-0.11.0.tar.bz2
+    cd fail2ban-0.11.0
+    sudo python setup.py install
+   
+Alternatively, you can clone the source from GitHub to a directory of Your choice, and do the install from there. Pick the correct branch, for example, 0.11

+    git clone https://github.com/fail2ban/fail2ban.git
+    cd fail2ban
+    sudo python setup.py install 
+    
 This will install Fail2Ban into the python library directory. The executable
 scripts are placed into `/usr/bin`, and configuration in `/etc/fail2ban`.

@@ -58,6 +65,9 @@ Fail2Ban should be correctly installed now. Just type:

 to see if everything is alright. You should always use fail2ban-client and
 never call fail2ban-server directly.
+You can verify that you have the correct version installed with 
+
+    fail2ban-client version

 Please note that the system init/service script is not automatically installed.
 To enable fail2ban as an automatic service, simply copy the script for your
@@ -79,11 +89,11 @@ fail2ban(1) and jail.conf(5)  manpages for further references.
 Code status:
 ------------

-* [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.png?branch=0.10)](https://travis-ci.org/fail2ban/fail2ban?branch=0.10) travis-ci.org (0.10 branch) / [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.png?branch=master)](https://travis-ci.org/fail2ban/fail2ban) travis-ci.org (master branch)
+* travis-ci.org: [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=0.11)](https://travis-ci.org/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [![tests status](https://secure.travis-ci.org/fail2ban/fail2ban.svg?branch=0.10)](https://travis-ci.org/fail2ban/fail2ban?branch=0.10) (0.10 branch) 

-* [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.png?branch=0.10)](https://coveralls.io/github/fail2ban/fail2ban?branch=0.10)
+* coveralls.io: [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=0.11)](https://coveralls.io/github/fail2ban/fail2ban?branch=0.11) (0.11 branch) / [![Coverage Status](https://coveralls.io/repos/fail2ban/fail2ban/badge.svg?branch=0.10)](https://coveralls.io/github/fail2ban/fail2ban?branch=0.10) / (0.10 branch)

-* [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=0.10)](https://codecov.io/gh/fail2ban/fail2ban/branch/0.10)
+* codecov.io: [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=0.11)](https://codecov.io/gh/fail2ban/fail2ban/branch/0.11) (0.11 branch) / [![codecov.io](https://codecov.io/gh/fail2ban/fail2ban/coverage.svg?branch=0.10)](https://codecov.io/gh/fail2ban/fail2ban/branch/0.10) (0.10 branch)

 Contact:
 --------
@@ -111,7 +111,7 @@ Russell Odom
 SATO Kentaro
 Sean DuBois
 Sebastian Arcus
-Serg G. Brester
+Serg G. Brester (sebres)
 Sergey Safarov
 Shaun C.
 Sireyessire
@@ -21,13 +21,14 @@
 #
 # Example, for ssh bruteforce (in section [sshd] of `jail.local`): 
 #   action = %(known/action)s
-#            abuseipdb[abuseipdb_apikey="my-api-key", abuseipdb_category="18,22"]
+#            %(action_abuseipdb)s[abuseipdb_apikey="my-api-key", abuseipdb_category="18,22"]
 #
-# See below for categories.
+# See below for catagories.
 #
+# Original Ref: https://wiki.shaunc.com/wikka.php?wakka=ReportingToAbuseIPDBWithFail2Ban
 # Added to fail2ban by Andrew James Collett (ajcollett)

-## abuseIPDB Categories, `the abuseipdb_category` MUST be set in the jail.conf action call.
+## abuseIPDB Catagories, `the abuseipdb_category` MUST be set in the jail.conf action call.
 # Example, for ssh bruteforce: action = %(action_abuseipdb)s[abuseipdb_category="18,22"]
 # ID	Title	Description
 # 3	  Fraud Orders
@@ -14,10 +14,7 @@
 # Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
 # Values:  CMD
 #
-actionstart = ipfw show | fgrep -c -m 1 -s 'table(<table>)' > /dev/null 2>&1 || (
-    num=$(ipfw show | awk 'BEGIN { b = <lowest_rule_num> } { if ($1 == b) { b = $1 + 1 } } END { print b }');
-    ipfw -q add "$num" <blocktype> <block> from table\(<table>\) to me <port>; echo "$num" > "<startstatefile>"
-  )
+actionstart = ipfw show | fgrep -c -m 1 -s 'table(<table>)' > /dev/null 2>&1 || ( ipfw show | awk 'BEGIN { b = <lowest_rule_num> } { if ($1 < b) {} else if ($1 == b) { b = $1 + 1 } else { e = b } } END { if (e) exit e <br> else exit b }'; num=$?; ipfw -q add $num <blocktype> <block> from table\(<table>\) to me <port>; echo $num > "<startstatefile>" )


 # Option:  actionstop
@@ -5,7 +5,7 @@
 #
 # Please set jail.local's permission to 640 because it contains your CF API key.
 #
-# This action depends on curl (and optionally jq).
+# This action depends on curl.
 # Referenced from http://www.normyee.net/blog/2012/02/02/adding-cloudflare-support-to-fail2ban by NORM YEE
 #
 # To get your CloudFlare API Key: https://www.cloudflare.com/a/account/my-account
@@ -43,9 +43,9 @@ actioncheck =
 # API v1
 #actionban = curl -s -o /dev/null https://www.cloudflare.com/api_json.html -d 'a=ban' -d 'tkn=<cftoken>' -d 'email=<cfuser>' -d 'key=<ip>'
 # API v4
-actionban = curl -s -o /dev/null -X POST <_cf_api_prms> \
-            -d '{"mode":"block","configuration":{"target":"ip","value":"<ip>"},"notes":"Fail2Ban <name>"}' \
-            <_cf_api_url>
+actionban = curl -s -o /dev/null -X POST -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' \
+            -H 'Content-Type: application/json' -d '{ "mode": "block", "configuration": { "target": "ip", "value": "<ip>" } }' \
+            https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules

 # Option:  actionunban
 # Notes.:  command executed when unbanning an IP. Take care that the
@@ -58,14 +58,9 @@ actionban = curl -s -o /dev/null -X POST <_cf_api_prms> \
 # API v1
 #actionunban = curl -s -o /dev/null https://www.cloudflare.com/api_json.html -d 'a=nul' -d 'tkn=<cftoken>' -d 'email=<cfuser>' -d 'key=<ip>'
 # API v4
-actionunban = id=$(curl -s -X GET <_cf_api_prms> \
-                   "<_cf_api_url>?mode=block&configuration_target=ip&configuration_value=<ip>&page=1&per_page=1&notes=Fail2Ban%%20<name>" \
-                   | { jq -r '.result[0].id' 2>/dev/null || tr -d '\n' | sed -nE 's/^.*"result"\s*:\s*\[\s*\{\s*"id"\s*:\s*"([^"]+)".*$/\1/p'; })
-              if [ -z "$id" ]; then echo "<name>: id for <ip> cannot be found"; exit 0; fi;
-              curl -s -o /dev/null -X DELETE <_cf_api_prms> "<_cf_api_url>/$id"
-
-_cf_api_url = https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules
-_cf_api_prms = -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' -H 'Content-Type: application/json'
+actionunban = curl -s -o /dev/null -X DELETE -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' \
+              https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules/$(curl -s -X GET -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' \
+              'https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules?mode=block&configuration_target=ip&configuration_value=<ip>&page=1&per_page=1' | cut -d'"' -f6)

 [Init]

@@ -18,7 +18,7 @@ before = firewallcmd-common.conf

 [Definition]

-actionstart = ipset create <ipmset> hash:ip timeout <default-ipsettime> <familyopt>
+actionstart = ipset create <ipmset> hash:ip timeout <default-timeout><familyopt>
              firewall-cmd --direct --add-rule <family> filter <chain> 0 <actiontype> -m set --match-set <ipmset> src -j <blocktype>

 actionflush = ipset flush <ipmset>
@@ -27,7 +27,9 @@ actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 <acti
             <actionflush>
             ipset destroy <ipmset>

-actionban = ipset add <ipmset> <ip> timeout <ipsettime> -exist
+actionban = ipset add <ipmset> <ip> timeout <bantime> -exist
+
+actionprolong = %(actionban)s

 actionunban = ipset del <ipmset> <ip> -exist

@@ -40,19 +42,11 @@ actionunban = ipset del <ipmset> <ip> -exist
 #
 chain = INPUT_direct

-# Option: default-ipsettime
+# Option: default-timeout
 # Notes:  specifies default timeout in seconds (handled default ipset timeout only)
-# Values:  [ NUM ]  Default: 0 (no timeout, managed by fail2ban by unban)
-default-ipsettime = 0
+# Values:  [ NUM ]  Default: 600

-# Option: ipsettime
-# Notes:  specifies ticket timeout (handled ipset timeout only)
-# Values:  [ NUM ]  Default: 0 (managed by fail2ban by unban)
-ipsettime = 0
-
-# expresion to caclulate timeout from bantime, example:
-# banaction = %(known/banaction)s[ipsettime='<timeout-bantime>']
-timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0)
+default-timeout = 600

 # Option: actiontype
 # Notes.: defines additions to the blocking rule
@@ -69,7 +63,7 @@ allports = -p <protocol>
 # Option: multiport
 # Notes.: addition to block access only to specific ports
 # Usage.: use in jail config:  banaction = firewallcmd-ipset[actiontype=<multiport>]
-multiport = -p <protocol> -m multiport --dports "$(echo '<port>' | sed s/:/-/g)"
+multiport = -p <protocol> -m multiport --dports <port>

 ipmset = f2b-<name>
 familyopt =
@@ -77,7 +71,7 @@ familyopt =
 [Init?family=inet6]

 ipmset = f2b-<name>6
-familyopt = family inet6
+familyopt = <sp>family inet6


 # DEV NOTES:
@@ -11,9 +11,9 @@ before = firewallcmd-common.conf

 actionstart = firewall-cmd --direct --add-chain <family> filter f2b-<name>
              firewall-cmd --direct --add-rule <family> filter f2b-<name> 1000 -j RETURN
-              firewall-cmd --direct --add-rule <family> filter <chain> 0 -m conntrack --ctstate NEW -p <protocol> -m multiport --dports "$(echo '<port>' | sed s/:/-/g)" -j f2b-<name>
+              firewall-cmd --direct --add-rule <family> filter <chain> 0 -m conntrack --ctstate NEW -p <protocol> -m multiport --dports <port> -j f2b-<name>

-actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 -m conntrack --ctstate NEW -p <protocol> -m multiport --dports "$(echo '<port>' | sed s/:/-/g)" -j f2b-<name>
+actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 -m conntrack --ctstate NEW -p <protocol> -m multiport --dports <port> -j f2b-<name>
             firewall-cmd --direct --remove-rules <family> filter f2b-<name>
             firewall-cmd --direct --remove-chain <family> filter f2b-<name>

@@ -10,9 +10,9 @@ before = firewallcmd-common.conf

 actionstart = firewall-cmd --direct --add-chain <family> filter f2b-<name>
              firewall-cmd --direct --add-rule <family> filter f2b-<name> 1000 -j RETURN
-              firewall-cmd --direct --add-rule <family> filter <chain> 0 -m state --state NEW -p <protocol> -m multiport --dports "$(echo '<port>' | sed s/:/-/g)" -j f2b-<name>
+              firewall-cmd --direct --add-rule <family> filter <chain> 0 -m state --state NEW -p <protocol> -m multiport --dports <port> -j f2b-<name>

-actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 -m state --state NEW -p <protocol> -m multiport --dports "$(echo '<port>' | sed s/:/-/g)" -j f2b-<name>
+actionstop = firewall-cmd --direct --remove-rule <family> filter <chain> 0 -m state --state NEW -p <protocol> -m multiport --dports <port> -j f2b-<name>
             firewall-cmd --direct --remove-rules <family> filter f2b-<name>
             firewall-cmd --direct --remove-chain <family> filter f2b-<name>

@@ -1,6 +1,6 @@
 # Fail2Ban configuration file
 #
-# Authors: Donald Yandt, Sergey G. Brester
+# Author: Donald Yandt 
 # 
 # Because of the rich rule commands requires firewalld-0.3.1+
 # This action uses firewalld rich-rules which gives you a cleaner iptables since it stores rules according to zones and not
@@ -10,15 +10,36 @@
 #
 # If you use the --permanent rule you get a xml file in /etc/firewalld/zones/<zone>.xml that can be shared and parsed easliy
 #
-# This is an derivative of firewallcmd-rich-rules.conf, see there for details and other parameters.
+# Example commands to view rules:
+# firewall-cmd [--zone=<zone>] --list-rich-rules
+# firewall-cmd [--zone=<zone>] --list-all
+# firewall-cmd [--zone=zone] --query-rich-rule='rule'

 [INCLUDES]

-before = firewallcmd-rich-rules.conf
+before = firewallcmd-common.conf

 [Definition]

-rich-suffix = log prefix='f2b-<name>' level='<level>' limit value='<rate>/m' <rich-blocktype>
+actionstart = 
+
+actionstop = 
+
+actioncheck = 
+
+# you can also use zones and/or service names. 
+#
+# zone example: 
+# firewall-cmd --zone=<zone> --add-rich-rule="rule family='<family>' source address='<ip>' port port='<port>' protocol='<protocol>' log prefix='f2b-<name>' level='<level>' limit value='<rate>/m' <rich-blocktype>"
+#
+# service name example:
+# firewall-cmd --zone=<zone> --add-rich-rule="rule family='<family>' source address='<ip>' service name='<service>' log prefix='f2b-<name>' level='<level>' limit value='<rate>/m' <rich-blocktype>"
+#
+# Because rich rules can only handle single or a range of ports we must split ports and execute the command for each port. Ports can be single and ranges separated by a comma or space for an example: http, https, 22-60, 18 smtp 
+
+actionban = ports="<port>"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='<family>' source address='<ip>' port port='$p' protocol='<protocol>' log prefix='f2b-<name>' level='<level>' limit value='<rate>/m' <rich-blocktype>"; done
+	   
+actionunban = ports="<port>"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="rule family='<family>' source address='<ip>' port port='$p' protocol='<protocol>' log prefix='f2b-<name>' level='<level>' limit value='<rate>/m' <rich-blocktype>"; done

 [Init]

@@ -27,3 +48,4 @@ level = info

 # log rate per minute
 rate = 1
+
@@ -35,10 +35,8 @@ actioncheck =
 #
 # Because rich rules can only handle single or a range of ports we must split ports and execute the command for each port. Ports can be single and ranges separated by a comma or space for an example: http, https, 22-60, 18 smtp 

-fwcmd_rich_rule = rule family='<family>' source address='<ip>' port port='$p' protocol='<protocol>' %(rich-suffix)s
-
-actionban = ports="$(echo '<port>' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="%(fwcmd_rich_rule)s"; done
+actionban = ports="<port>"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='<family>' source address='<ip>' port port='$p' protocol='<protocol>' <rich-blocktype>"; done
 	   
-actionunban = ports="$(echo '<port>' | sed s/:/-/g)"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="%(fwcmd_rich_rule)s"; done
+actionunban = ports="<port>"; for p in $(echo $ports | tr ", " " "); do firewall-cmd --remove-rich-rule="rule family='<family>' source address='<ip>' port port='$p' protocol='<protocol>' <rich-blocktype>"; done
+

-rich-suffix = <rich-blocktype>
@@ -26,7 +26,7 @@ before = iptables-common.conf
 # Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
 # Values:  CMD
 #
-actionstart = ipset create <ipmset> hash:ip timeout <default-ipsettime> <familyopt>
+actionstart = ipset create <ipmset> hash:ip timeout <default-timeout><familyopt>
              <iptables> -I <chain> -m set --match-set <ipmset> src -j <blocktype>

 # Option:  actionflush
@@ -49,7 +49,9 @@ actionstop = <iptables> -D <chain> -m set --match-set <ipmset> src -j <blocktype
 # Tags:    See jail.conf(5) man page
 # Values:  CMD
 #
-actionban = ipset add <ipmset> <ip> timeout <ipsettime> -exist
+actionban = ipset add <ipmset> <ip> timeout <bantime> -exist
+
+actionprolong = %(actionban)s

 # Option:  actionunban
 # Notes.:  command executed when unbanning an IP. Take care that the
@@ -61,19 +63,11 @@ actionunban = ipset del <ipmset> <ip> -exist

 [Init]

-# Option: default-ipsettime
+# Option: default-timeout
 # Notes:  specifies default timeout in seconds (handled default ipset timeout only)
-# Values:  [ NUM ]  Default: 0 (no timeout, managed by fail2ban by unban)
-default-ipsettime = 0
+# Values:  [ NUM ]  Default: 600

-# Option: ipsettime
-# Notes:  specifies ticket timeout (handled ipset timeout only)
-# Values:  [ NUM ]  Default: 0 (managed by fail2ban by unban)
-ipsettime = 0
-
-# expresion to caclulate timeout from bantime, example:
-# banaction = %(known/banaction)s[ipsettime='<timeout-bantime>']
-timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0)
+default-timeout = 600

 ipmset = f2b-<name>
 familyopt =
@@ -82,4 +76,4 @@ familyopt =
 [Init?family=inet6]

 ipmset = f2b-<name>6
-familyopt = family inet6
+familyopt = <sp>family inet6
@@ -26,7 +26,7 @@ before = iptables-common.conf
 # Notes.:  command executed on demand at the first ban (or at the start of Fail2Ban if actionstart_on_demand is set to false).
 # Values:  CMD
 #
-actionstart = ipset create <ipmset> hash:ip timeout <default-ipsettime> <familyopt>
+actionstart = ipset create <ipmset> hash:ip timeout <default-timeout><familyopt>
              <iptables> -I <chain> -p <protocol> -m multiport --dports <port> -m set --match-set <ipmset> src -j <blocktype>

 # Option:  actionflush
@@ -49,7 +49,9 @@ actionstop = <iptables> -D <chain> -p <protocol> -m multiport --dports <port> -m
 # Tags:    See jail.conf(5) man page
 # Values:  CMD
 #
-actionban = ipset add <ipmset> <ip> timeout <ipsettime> -exist
+actionban = ipset add <ipmset> <ip> timeout <bantime> -exist
+
+actionprolong = %(actionban)s

 # Option:  actionunban
 # Notes.:  command executed when unbanning an IP. Take care that the
@@ -61,19 +63,11 @@ actionunban = ipset del <ipmset> <ip> -exist

 [Init]

-# Option: default-ipsettime
+# Option: default-timeout
 # Notes:  specifies default timeout in seconds (handled default ipset timeout only)
-# Values:  [ NUM ]  Default: 0 (no timeout, managed by fail2ban by unban)
-default-ipsettime = 0
+# Values:  [ NUM ]  Default: 600

-# Option: ipsettime
-# Notes:  specifies ticket timeout (handled ipset timeout only)
-# Values:  [ NUM ]  Default: 0 (managed by fail2ban by unban)
-ipsettime = 0
-
-# expresion to caclulate timeout from bantime, example:
-# banaction = %(known/banaction)s[ipsettime='<timeout-bantime>']
-timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0)
+default-timeout = 600

 ipmset = f2b-<name>
 familyopt =
@@ -82,4 +76,4 @@ familyopt =
 [Init?family=inet6]

 ipmset = f2b-<name>6
-familyopt = family inet6
+familyopt = <sp>family inet6
@@ -34,7 +34,7 @@ type = multiport

 rule_match-custom =
 rule_match-allports = meta l4proto \{ <protocol> \}
-rule_match-multiport = $proto dport \{ $(echo '<port>' | sed s/:/-/g) \}
+rule_match-multiport = $proto dport \{ <port> \}
 match = <rule_match-<type>>

 # Option:  rule_stat
@@ -103,8 +103,6 @@ actionstop = %(actionflush)s

 actioncheck = 

-_echo_blck_row = printf '\%%s 1;\n' "<fid>"
+actionban = echo "\\\\<fid> 1;" >> '%(blck_lst_file)s'; %(blck_lst_reload)s

-actionban = %(_echo_blck_row)s >> '%(blck_lst_file)s'; %(blck_lst_reload)s
-
-actionunban = id=$(%(_echo_blck_row)s | sed -e 's/[]\/$*.^|[]/\\&/g'); sed -i "/^$id$/d" %(blck_lst_file)s; %(blck_lst_reload)s
+actionunban = id=$(echo "<fid>" | sed -e 's/[]\/$*.^|[]/\\&/g'); sed -i "/^\\\\$id 1;$/d" %(blck_lst_file)s; %(blck_lst_reload)s
@@ -12,5 +12,5 @@ actioncheck =
 actionban = /usr/libexec/afctl -a <ip> -t <bantime>
 actionunban = /usr/libexec/afctl -r <ip>

-[Init]
-bantime = 2880
+actionprolong = %(actionunban)s && %(actionban)s
+
@@ -51,7 +51,7 @@
 # Values:  CMD
 #
 actionstart = if ! ipset -quiet -name list f2b-<name> >/dev/null;
-              then ipset -quiet -exist create f2b-<name> hash:ip timeout <default-ipsettime>;
+              then ipset -quiet -exist create f2b-<name> hash:ip timeout <default-timeout>;
              fi

 # Option:  actionstop
@@ -66,7 +66,9 @@ actionstop = ipset flush f2b-<name>
 # Tags:    See jail.conf(5) man page
 # Values:  CMD
 #
-actionban = ipset add f2b-<name> <ip> timeout <ipsettime> -exist
+actionban = ipset add f2b-<name> <ip> timeout <bantime> -exist
+
+actionprolong = %(actionban)s

 # Option:  actionunban
 # Notes.:  command executed when unbanning an IP. Take care that the
@@ -76,16 +78,8 @@ actionban = ipset add f2b-<name> <ip> timeout <ipsettime> -exist
 #
 actionunban = ipset del f2b-<name> <ip> -exist

-# Option: default-ipsettime
+# Option: default-timeout
 # Notes:  specifies default timeout in seconds (handled default ipset timeout only)
-# Values:  [ NUM ]  Default: 0 (no timeout, managed by fail2ban by unban)
-default-ipsettime = 0
+# Values:  [ NUM ]  Default: 600

-# Option: ipsettime
-# Notes:  specifies ticket timeout (handled ipset timeout only)
-# Values:  [ NUM ]  Default: 0 (managed by fail2ban by unban)
-ipsettime = 0
-
-# expresion to caclulate timeout from bantime, example:
-# banaction = %(known/banaction)s[ipsettime='<timeout-bantime>']
-timeout-bantime = $([ "<bantime>" -le 2147483 ] && echo "<bantime>" || echo 0)
+default-timeout = 600
@@ -19,7 +19,7 @@
 #         NOTICE
 #         INFO
 #         DEBUG
-# Values: [ LEVEL ]  Default: INFO
+# Values: [ LEVEL ]  Default: ERROR
 #
 loglevel = INFO

@@ -17,9 +17,9 @@ before = apache-common.conf

 [Definition]

-script = /\S*(?:php(?:[45]|[.-]cgi)?|\.asp|\.exe|\.pl|\bcgi-bin/)
+script = /\S*(?:php(?:[45]|[.-]cgi)?|\.asp|\.exe|\.pl)

-prefregex = ^%(_apache_error_client)s (?:AH0(?:01(?:28|30)|1(?:264|071)|2811): )?(?:(?:[Ff]ile|script|[Gg]ot) )<F-CONTENT>.+</F-CONTENT>$
+prefregex = ^%(_apache_error_client)s (?:AH0(?:01(?:28|30)|1(?:264|071)): )?(?:(?:[Ff]ile|script|[Gg]ot) )<F-CONTENT>.+</F-CONTENT>$

 failregex = ^(?:does not exist|not found or unable to stat): <script>\b
            ^'<script>\S*' not found or unable to stat
@@ -2,12 +2,5 @@
 # Detecting failed login attempts
 # Logged in bwdata/logs/identity/Identity/log.txt

-[INCLUDES]
-before = common.conf
-
 [Definition]
-_daemon = Bitwarden-Identity
-failregex = ^%(__prefix_line)s\s*\[(?:W(?:RN|arning)|Bit\.Core\.[^\]]+)\]\s+Failed login attempt(?:, 2FA invalid)?\. <ADDR>$
-
-# DEV Notes:
-# __prefix_line can result to an empty string, so it can support syslog and non-syslog at once.
+failregex = ^\s*\[WRN\]\s+Failed login attempt(?:, 2FA invalid)?\. <HOST>$
@@ -25,7 +25,7 @@ __pid_re = (?:\[\d+\])

 # Daemon name (with optional source_file:line or whatever)
 # EXAMPLES: pam_rhosts_auth, [sshd], pop(pam_unix)
-__daemon_re = [\[\(]?<_daemon>(?:\(\S+\))?[\]\)]?:?
+__daemon_re = [\[\(]?%(_daemon)s(?:\(\S+\))?[\]\)]?:?

 # extra daemon info
 # EXAMPLE: [ID 800047 auth.info]
@@ -33,7 +33,7 @@ __daemon_extra_re = \[ID \d+ \S+\]

 # Combinations of daemon name and PID
 # EXAMPLES: sshd[31607], pop(pam_unix)[4920]
-__daemon_combs_re = (?:<__pid_re>?:\s+<__daemon_re>|<__daemon_re><__pid_re>?:?)
+__daemon_combs_re = (?:%(__pid_re)s?:\s+%(__daemon_re)s|%(__daemon_re)s%(__pid_re)s?:?)

 # Some messages have a kernel prefix with a timestamp
 # EXAMPLES: kernel: [769570.846956]
@@ -69,12 +69,12 @@ datepattern = <lt_<logtype>/datepattern>

 [lt_file]
 # Common line prefixes for logtype "file":
-__prefix_line = <__date_ambit>?\s*(?:<__bsd_syslog_verbose>\s+)?(?:<__hostname>\s+)?(?:<__kernel_prefix>\s+)?(?:<__vserver>\s+)?(?:<__daemon_combs_re>\s+)?(?:<__daemon_extra_re>\s+)?
+__prefix_line = %(__date_ambit)s?\s*(?:%(__bsd_syslog_verbose)s\s+)?(?:%(__hostname)s\s+)?(?:%(__kernel_prefix)s\s+)?(?:%(__vserver)s\s+)?(?:%(__daemon_combs_re)s\s+)?(?:%(__daemon_extra_re)s\s+)?
 datepattern = {^LN-BEG}

 [lt_short]
 # Common (short) line prefix for logtype "journal" (corresponds output of formatJournalEntry):
-__prefix_line = \s*(?:<__hostname>\s+)?(?:<_daemon><__pid_re>?:?\s+)?(?:<__kernel_prefix>\s+)?
+__prefix_line = \s*(?:%(__hostname)s\s+)?(?:%(_daemon)s%(__pid_re)s?:?\s+)?(?:%(__kernel_prefix)s\s+)?
 datepattern = %(lt_file/datepattern)s
 [lt_journal]
 __prefix_line = %(lt_short/__prefix_line)s
@@ -12,7 +12,7 @@ before = common.conf

 _daemon = courieresmtpd

-prefregex = ^%(__prefix_line)serror,relay=<HOST>,(?:port=\d+,)?<F-CONTENT>.+</F-CONTENT>$
+prefregex = ^%(__prefix_line)serror,relay=<HOST>,<F-CONTENT>.+</F-CONTENT>$

 failregex = ^[^:]*: 550 User (<.*> )?unknown\.?$
            ^msg="535 Authentication failed\.",cmd:( AUTH \S+)?( [0-9a-zA-Z\+/=]+)?(?: \S+)$
@@ -10,15 +10,15 @@ before = common.conf
 _auth_worker = (?:dovecot: )?auth(?:-worker)?
 _daemon = (?:dovecot(?:-auth)?|auth)

-prefregex = ^%(__prefix_line)s(?:%(_auth_worker)s(?:\([^\)]+\))?: )?(?:%(__pam_auth)s(?:\(dovecot:auth\))?: |(?:pop3|imap|managesieve|submission)-login: )?(?:Info: )?<F-CONTENT>.+</F-CONTENT>$
+prefregex = ^%(__prefix_line)s(?:%(_auth_worker)s(?:\([^\)]+\))?: )?(?:%(__pam_auth)s(?:\(dovecot:auth\))?: |(?:pop3|imap)-login: )?(?:Info: )?<F-CONTENT>.+</F-CONTENT>$

 failregex = ^authentication failure; logname=<F-ALT_USER1>\S*</F-ALT_USER1> uid=\S* euid=\S* tty=dovecot ruser=<F-USER>\S*</F-USER> rhost=<HOST>(?:\s+user=<F-ALT_USER>\S*</F-ALT_USER>)?\s*$
-            ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?::(?: [^ \(]+)+)? \((?:auth failed, \d+ attempts(?: in \d+ secs)?|tried to use (?:disabled|disallowed) \S+ auth|proxy dest auth failed)\):(?: user=<<F-USER>[^>]*</F-USER>>,)?(?: method=\S+,)? rip=<HOST>(?:[^>]*(?:, session=<\S+>)?)\s*$
+            ^(?:Aborted login|Disconnected)(?::(?: [^ \(]+)+)? \((?:auth failed, \d+ attempts(?: in \d+ secs)?|tried to use (?:disabled|disallowed) \S+ auth|proxy dest auth failed)\):(?: user=<<F-USER>[^>]*</F-USER>>,)?(?: method=\S+,)? rip=<HOST>(?:[^>]*(?:, session=<\S+>)?)\s*$
            ^pam\(\S+,<HOST>(?:,\S*)?\): pam_authenticate\(\) failed: (?:User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \(password mismatch\?\)|Permission denied)\s*$
-            ^[a-z\-]{3,15}\(\S*,<HOST>(?:,\S*)?\): (?:unknown user|invalid credentials|Password mismatch)
+            ^[a-z\-]{3,15}\(\S*,<HOST>(?:,\S*)?\): (?:unknown user|invalid credentials|Password mismatch)\s*$
            <mdre-<mode>>

-mdre-aggressive = ^(?:Aborted login|Disconnected|Remote closed connection|Client has quit the connection)(?::(?: [^ \(]+)+)? \((?:no auth attempts|disconnected before auth was ready,|client didn't finish \S+ auth,)(?: (?:in|waited) \d+ secs)?\):(?: user=<[^>]*>,)?(?: method=\S+,)? rip=<HOST>(?:[^>]*(?:, session=<\S+>)?)\s*$
+mdre-aggressive = ^(?:Aborted login|Disconnected)(?::(?: [^ \(]+)+)? \((?:no auth attempts|disconnected before auth was ready,|client didn't finish \S+ auth,)(?: (?:in|waited) \d+ secs)?\):(?: user=<[^>]*>,)?(?: method=\S+,)? rip=<HOST>(?:[^>]*(?:, session=<\S+>)?)\s*$

 mdre-normal = 

@@ -1,6 +0,0 @@
-# Fail2Ban filter for Gitlab
-# Detecting unauthorized access to the Gitlab Web portal
-# typically logged in /var/log/gitlab/gitlab-rails/application.log
-
-[Definition]
-failregex = ^: Failed Login: username=<F-USER>.+</F-USER> ip=<HOST>$
@@ -1,9 +0,0 @@
-# Fail2Ban filter for Grafana
-# Detecting unauthorized access
-# Typically logged in /var/log/grafana/grafana.log
-
-[Init]
-datepattern = ^t=%%Y-%%m-%%dT%%H:%%M:%%S%%z
-
-[Definition]
-failregex = ^(?: lvl=err?or)? msg="Invalid username or password"(?: uname=(?:"<F-ALT_USER>[^"]+</F-ALT_USER>"|<F-USER>\S+</F-USER>)| error="<F-ERROR>[^"]+</F-ERROR>"| \S+=(?:\S*|"[^"]+"))* remote_addr=<ADDR>$
@@ -5,47 +5,21 @@

 [Definition]

-logging = catalina
-failregex = <L_<logging>/failregex>
-maxlines = <L_<logging>/maxlines>
-datepattern = <L_<logging>/datepattern>
-
-[L_catalina]
-
+# Option: failregex
+# Notes.: regex to match the password failures messages in the logfile.
+# Values: TEXT
+#
 failregex = ^.*\nWARNING: Authentication attempt from <HOST> for user "[^"]*" failed\.$

+# Option:  ignoreregex
+# Notes.:  regex to ignore. If this regex matches, the line is ignored.
+# Values:  TEXT
+#
+ignoreregex = 
+
+# "maxlines" is number of log lines to buffer for multi-line regex searches
 maxlines = 2

 datepattern = ^%%b %%d, %%ExY %%I:%%M:%%S %%p
              ^WARNING:()**
-              {^LN-BEG}
-
-[L_webapp]
-
-failregex = ^ \[\S+\] WARN  \S+ - Authentication attempt from <HOST> for user "<F-USER>[^"]+</F-USER>" failed.
-
-maxlines = 1
-
-datepattern = ^%%H:%%M:%%S.%%f
-
-# DEV Notes:
-#
-# failregex is based on the default pattern given in Guacamole documentation :
-# https://guacamole.apache.org/doc/gug/configuring-guacamole.html#webapp-logging
-#
-# The following logback.xml Guacamole configuration file can then be used accordingly :
-# <configuration>
-#   <appender name="FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
-#     <file>/var/log/guacamole.log</file>
-#     <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
-#       <fileNamePattern>/var/log/guacamole.%d.log.gz</fileNamePattern>
-#       <maxHistory>32</maxHistory>
-#     </rollingPolicy>
-#     <encoder>
-#       <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
-#     </encoder>
-#   </appender>
-#   <root level="info">
-#     <appender-ref ref="FILE" />
-#   </root>
-# </configuration>
+              {^LN-BEG}
@@ -8,17 +8,13 @@
 # common.local
 before = common.conf

-# [DEFAULT]
-# logtype = short
-
 [Definition]

 _daemon = monit

-_prefix = Warning|HttpRequest
-
 # Regexp for previous (accessing monit httpd) and new (access denied) versions
-failregex = ^%(__prefix_line)s(?:error\s*:\s+)?(?:%(_prefix)s):\s+(?:access denied\s+--\s+)?[Cc]lient '?<HOST>'?(?:\s+supplied|\s*:)\s+(?:unknown user '<F-ALT_USER>[^']+</F-ALT_USER>'|wrong password for user '<F-USER>[^']*</F-USER>'|empty password)
+failregex = ^\[\s*\]\s*error\s*:\s*Warning:\s+Client '<HOST>' supplied (?:unknown user '[^']+'|wrong password for user '[^']*') accessing monit httpd$
+            ^%(__prefix_line)s\w+: access denied -- client <HOST>: (?:unknown user '[^']+'|wrong password for user '[^']*'|empty password)$

 # Ignore login with empty user (first connect, no user specified)
 # ignoreregex = %(__prefix_line)s\w+: access denied -- client <HOST>: (?:unknown user '')
@@ -17,7 +17,7 @@ before = common.conf

 _daemon = mysqld

-failregex = ^%(__prefix_line)s(?:(?:\d{6}|\d{4}-\d{2}-\d{2})[ T]\s?\d{1,2}:\d{2}:\d{2} )?(?:\d+ )?\[\w+\] (?:\[[^\]]+\] )*Access denied for user '<F-USER>[^']+</F-USER>'@'<HOST>' (to database '[^']*'|\(using password: (YES|NO)\))*\s*$
+failregex = ^%(__prefix_line)s(?:(?:\d{6}|\d{4}-\d{2}-\d{2})[ T]\s?\d{1,2}:\d{2}:\d{2} )?(?:\d+ )?\[\w+\] (?:\[[^\]]+\] )*Access denied for user '[^']+'@'<HOST>' (to database '[^']*'|\(using password: (YES|NO)\))*\s*$

 ignoreregex = 

@@ -1,4 +1,4 @@
-# Fail2Ban filter for the phpMyAdmin-syslog
+# Fail2Ban fitler for the phpMyAdmin-syslog
 #

 [INCLUDES]
@@ -37,7 +37,7 @@ mdre-rbl  = ^RCPT from [^[]*\[<HOST>\]%(_port)s: [45]54 [45]\.7\.1 Service unava
 mdpr-more = %(mdpr-normal)s
 mdre-more = %(mdre-normal)s

-mdpr-ddos = (?:lost connection after(?! DATA) [A-Z]+|disconnect(?= from \S+(?: \S+=\d+)* auth=0/(?:[1-9]|\d\d+)))
+mdpr-ddos = lost connection after(?! DATA) [A-Z]+
 mdre-ddos = ^from [^[]*\[<HOST>\]%(_port)s:?

 mdpr-extra = (?:%(mdpr-auth)s|%(mdpr-normal)s)
@@ -1,4 +1,4 @@
-# Fail2Ban filter for the Proftpd FTP daemon
+# Fail2Ban fitler for the Proftpd FTP daemon
 #
 # Set "UseReverseDNS off" in proftpd.conf to avoid the need for DNS.
 # See: http://www.proftpd.org/docs/howto/DNS.html
@@ -14,15 +14,16 @@ before = common.conf

 _daemon = proftpd

-__suffix_failed_login = ([uU]ser not authorized for login|[nN]o such user found|[iI]ncorrect password|[pP]assword expired|[aA]ccount disabled|[iI]nvalid shell: '\S+'|[uU]ser in \S+|[lL]imit (access|configuration) denies login|[nN]ot a UserAlias|[mM]aximum login length exceeded)
+__suffix_failed_login = (User not authorized for login|No such user found|Incorrect password|Password expired|Account disabled|Invalid shell: '\S+'|User in \S+|Limit (access|configuration) denies login|Not a UserAlias|maximum login length exceeded).?


-prefregex = ^%(__prefix_line)s%(__hostname)s \(\S+\[<HOST>\]\)[: -]+ <F-CONTENT>(?:USER|SECURITY|Maximum) .+</F-CONTENT>$
+prefregex = ^%(__prefix_line)s%(__hostname)s \(\S+\[<HOST>\]\)[: -]+ <F-CONTENT>(?:USER|SECURITY|Maximum).+</F-CONTENT>$


-failregex = ^USER <F-USER>\S+|.*?</F-USER>(?: \(Login failed\))?: %(__suffix_failed_login)s
-            ^SECURITY VIOLATION: <F-USER>\S+|.*?</F-USER> login attempted
-            ^Maximum login attempts \(\d+\) exceeded
+failregex = ^USER .*: no such user found from \S+ \[\S+\] to \S+:\S+ *$
+            ^USER .* \(Login failed\): %(__suffix_failed_login)s\s*$
+            ^SECURITY VIOLATION: .* login attempted\. *$
+            ^Maximum login attempts \(\d+\) exceeded *$

 ignoreregex = 

@@ -18,7 +18,7 @@ prefregex = ^\s*(\[\])?(%(__hostname)s\s*(?:roundcube(?:\[(\d*)\])?:)?\s*(<[\w]+
 failregex = ^(?:FAILED login|Login failed) for <F-USER>.*</F-USER> from <HOST>(?:(?:\([^\)]*\))?\. (?:(?! from ).)*(?: user=(?P=user))? in \S+\.php on line \d+ \(\S+ \S+\))?$
            ^(?:<[\w]+> )?Failed login for <F-USER>.*</F-USER> from <HOST> in session \w+( \(error: \d\))?$

-ignoreregex = 
+ignoreregex = Could not connect to .* Connection refused

 journalmatch = SYSLOG_IDENTIFIER=roundcube

@@ -8,14 +8,11 @@ before = common.conf
 [Definition]

 _daemon = (?:sendmail|sm-(?:mta|acceptingconnections))
-# "\w{14,20}" will give support for IDs from 14 up to 20 characters long
 __prefix_line = %(known/__prefix_line)s(?:\w{14,20}: )?
-addr = (?:IPv6:<IP6>|<IP4>)

-prefregex = ^<F-MLFID>%(__prefix_line)s</F-MLFID><F-CONTENT>.+</F-CONTENT>$
+# "w{14,20}" will give support for IDs from 14 up to 20 characters long
+failregex = ^%(__prefix_line)s(\S+ )?\[(?:IPv6:<IP6>|<IP4>)\]( \(may be forged\))?: possible SMTP attack: command=AUTH, count=\d+$

-failregex = ^(\S+ )?\[%(addr)s\]( \(may be forged\))?: possible SMTP attack: command=AUTH, count=\d+$
-            ^AUTH failure \(LOGIN\):(?: [^:]+:)? authentication failure: checkpass failed, user=<F-USER>(?:\S+|.*?)</F-USER>, relay=(?:\S+ )?\[%(addr)s\](?: \(may be forged\))?$
 ignoreregex =

 journalmatch = _SYSTEMD_UNIT=sendmail.service
@@ -21,20 +21,19 @@ before = common.conf

 _daemon = (?:(sm-(mta|acceptingconnections)|sendmail))
 __prefix_line = %(known/__prefix_line)s(?:\w{14,20}: )?
-addr = (?:IPv6:<IP6>|<IP4>)

 prefregex = ^<F-MLFID>%(__prefix_line)s</F-MLFID><F-CONTENT>.+</F-CONTENT>$

-cmnfailre = ^ruleset=check_rcpt, arg1=(?P<email><\S+@\S+>), relay=(\S+ )?\[%(addr)s\](?: \(may be forged\))?, reject=(550 5\.7\.1 (?P=email)\.\.\. Relaying denied\. (IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\])|553 5\.1\.8 (?P=email)\.\.\. Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$
-            ^ruleset=check_relay, arg1=(?P<dom>\S+), arg2=%(addr)s, relay=((?P=dom) )?\[(\d+\.){3}\d+\](?: \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$
-            ^rejecting commands from (\S* )?\[%(addr)s\] due to pre-greeting traffic after \d+ seconds$
-            ^(?:\S+ )?\[%(addr)s\]: (?:(?i)expn|vrfy) \S+ \[rejected\]$
+cmnfailre = ^ruleset=check_rcpt, arg1=(?P<email><\S+@\S+>), relay=(\S+ )?\[(?:IPv6:<IP6>|<IP4>)\](?: \(may be forged\))?, reject=(550 5\.7\.1 (?P=email)\.\.\. Relaying denied\. (IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\])|553 5\.1\.8 (?P=email)\.\.\. Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$
+            ^ruleset=check_relay, arg1=(?P<dom>\S+), arg2=(?:IPv6:<IP6>|<IP4>), relay=((?P=dom) )?\[(\d+\.){3}\d+\](?: \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$
+            ^rejecting commands from (\S* )?\[(?:IPv6:<IP6>|<IP4>)\] due to pre-greeting traffic after \d+ seconds$
+            ^(?:\S+ )?\[(?:IPv6:<IP6>|<IP4>)\]: (?:(?i)expn|vrfy) \S+ \[rejected\]$
            ^<[^@]+@[^>]+>\.\.\. No such user here$
-            ^<F-NOFAIL>from=<[^@]+@[^>]+></F-NOFAIL>, size=\d+, class=\d+, nrcpts=\d+, bodytype=\w+, proto=E?SMTP, daemon=MTA, relay=\S+ \[%(addr)s\]$
+            ^<F-NOFAIL>from=<[^@]+@[^>]+></F-NOFAIL>, size=\d+, class=\d+, nrcpts=\d+, bodytype=\w+, proto=E?SMTP, daemon=MTA, relay=\S+ \[(?:IPv6:<IP6>|<IP4>)\]$

 mdre-normal =

-mdre-extra = ^(?:\S+ )?\[%(addr)s\](?: \(may be forged\))? did not issue \S+ during connection
+mdre-extra = ^(?:\S+ )?\[(?:IPv6:<IP6>|<IP4>)\](?: \(may be forged\))? did not issue (?:[A-Z]{4}[/ ]?)+during connection to (?:TLS)?M(?:TA|S[PA])(?:-\w+)?$

 mdre-aggressive = %(mdre-extra)s

@@ -1,9 +0,0 @@
-# Fail2Ban filter for SoftEtherVPN
-# Detecting unauthorized access to SoftEtherVPN
-# typically logged in /usr/local/vpnserver/security_log/*/sec.log, or in syslog, depending on configuration
-
-[INCLUDES]
-before = common.conf
-
-[Definition]
-failregex = ^%(__prefix_line)s(?:(?:\([\d\-]+ [\d:.]+\) )?<SECURITY_LOG>: )?Connection "[^"]+": User authentication failed. The user name that has been provided was "<F-USER>(?:[^"]+|.+)</F-USER>", from <ADDR>\.$
@@ -25,7 +25,7 @@ __pref = (?:(?:error|fatal): (?:PAM: )?)?
 __suff = (?: (?:port \d+|on \S+|\[preauth\])){0,3}\s*
 __on_port_opt = (?: (?:port \d+|on \S+)){0,2}
 # close by authenticating user:
-__authng_user = (?: (?:invalid|authenticating) user <F-USER>\S+|.*?</F-USER>)?
+__authng_user = (?: (?:invalid|authenticating) user <F-USER>\S+|.+?</F-USER>)?

 # for all possible (also future) forms of "no matching (cipher|mac|MAC|compression method|key exchange method|host key type) found",
 # see ssherr.c for all possible SSH_ERR_..._ALG_MATCH errors.
@@ -40,45 +40,39 @@ prefregex = ^<F-MLFID>%(__prefix_line)s</F-MLFID>%(__pref)s<F-CONTENT>.+</F-CONT

 cmnfailre = ^[aA]uthentication (?:failure|error|failed) for <F-USER>.*</F-USER> from <HOST>( via \S+)?%(__suff)s$
            ^User not known to the underlying authentication module for <F-USER>.*</F-USER> from <HOST>%(__suff)s$
-            <cmnfailre-failed-pub-<publickey>>
-            ^Failed <cmnfailed> for (?P<cond_inv>invalid user )?<F-USER>(?P<cond_user>\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)
+            ^Failed publickey for invalid user <F-USER>(?P<cond_user>\S+)|(?:(?! from ).)*?</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)
+            ^Failed \b(?!publickey)\S+ for (?P<cond_inv>invalid user )?<F-USER>(?P<cond_user>\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)
            ^<F-USER>ROOT</F-USER> LOGIN REFUSED FROM <HOST>
            ^[iI](?:llegal|nvalid) user <F-USER>.*?</F-USER> from <HOST>%(__suff)s$
-            ^User <F-USER>\S+|.*?</F-USER> from <HOST> not allowed because not listed in AllowUsers%(__suff)s$
-            ^User <F-USER>\S+|.*?</F-USER> from <HOST> not allowed because listed in DenyUsers%(__suff)s$
-            ^User <F-USER>\S+|.*?</F-USER> from <HOST> not allowed because not in any group%(__suff)s$
+            ^User <F-USER>.+</F-USER> from <HOST> not allowed because not listed in AllowUsers%(__suff)s$
+            ^User <F-USER>.+</F-USER> from <HOST> not allowed because listed in DenyUsers%(__suff)s$
+            ^User <F-USER>.+</F-USER> from <HOST> not allowed because not in any group%(__suff)s$
            ^refused connect from \S+ \(<HOST>\)
            ^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>%(__on_port_opt)s:\s*3: .*: Auth fail%(__suff)s$
-            ^User <F-USER>\S+|.*?</F-USER> from <HOST> not allowed because a group is listed in DenyGroups%(__suff)s$
-            ^User <F-USER>\S+|.*?</F-USER> from <HOST> not allowed because none of user's groups are listed in AllowGroups%(__suff)s$
+            ^User <F-USER>.+</F-USER> from <HOST> not allowed because a group is listed in DenyGroups%(__suff)s$
+            ^User <F-USER>.+</F-USER> from <HOST> not allowed because none of user's groups are listed in AllowGroups%(__suff)s$
            ^<F-NOFAIL>%(__pam_auth)s\(sshd:auth\):\s+authentication failure;</F-NOFAIL>(?:\s+(?:(?:logname|e?uid|tty)=\S*)){0,4}\s+ruser=<F-ALT_USER>\S*</F-ALT_USER>\s+rhost=<HOST>(?:\s+user=<F-USER>\S*</F-USER>)?%(__suff)s$
-            ^maximum authentication attempts exceeded for <F-USER>.*</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?%(__suff)s$
-            ^User <F-USER>\S+|.*?</F-USER> not allowed because account is locked%(__suff)s
+            ^(error: )?maximum authentication attempts exceeded for <F-USER>.*</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?%(__suff)s$
+            ^User <F-USER>.+</F-USER> not allowed because account is locked%(__suff)s
            ^<F-MLFFORGET>Disconnecting</F-MLFFORGET>(?: from)?(?: (?:invalid|authenticating)) user <F-USER>\S+</F-USER> <HOST>%(__on_port_opt)s:\s*Change of username or service not allowed:\s*.*\[preauth\]\s*$
-            ^Disconnecting: Too many authentication failures(?: for <F-USER>\S+|.*?</F-USER>)?%(__suff)s$
+            ^<F-MLFFORGET>Disconnecting</F-MLFFORGET>: Too many authentication failures(?: for <F-USER>.+?</F-USER>)?%(__suff)s$
            ^<F-NOFAIL>Received <F-MLFFORGET>disconnect</F-MLFFORGET></F-NOFAIL> from <HOST>%(__on_port_opt)s:\s*11:
            <mdre-<mode>-other>
            ^<F-MLFFORGET><F-MLFGAINED>Accepted \w+</F-MLFGAINED></F-MLFFORGET> for <F-USER>\S+</F-USER> from <HOST>(?:\s|$)

-cmnfailed-any = \S+
-cmnfailed-ignore = \b(?!publickey)\S+
-cmnfailed-invalid = <cmnfailed-ignore>
-cmnfailed-nofail = (?:<F-NOFAIL>publickey</F-NOFAIL>|\S+)
-cmnfailed = <cmnfailed-<publickey>>
-
 mdre-normal =
 # used to differentiate "connection closed" with and without `[preauth]` (fail/nofail cases in ddos mode)
 mdre-normal-other = ^<F-NOFAIL><F-MLFFORGET>(Connection closed|Disconnected)</F-MLFFORGET></F-NOFAIL> (?:by|from)%(__authng_user)s <HOST>(?:%(__suff)s|\s*)$

 mdre-ddos = ^Did not receive identification string from <HOST>
-            ^kex_exchange_identification: (?:[Cc]lient sent invalid protocol identifier|[Cc]onnection closed by remote host)
            ^Bad protocol version identification '.*' from <HOST>
+            ^Connection <F-MLFFORGET>reset</F-MLFFORGET> by <HOST>
            ^<F-NOFAIL>SSH: Server;Ltype:</F-NOFAIL> (?:Authname|Version|Kex);Remote: <HOST>-\d+;[A-Z]\w+:
            ^Read from socket failed: Connection <F-MLFFORGET>reset</F-MLFFORGET> by peer
 # same as mdre-normal-other, but as failure (without <F-NOFAIL>) and [preauth] only:
-mdre-ddos-other = ^<F-MLFFORGET>(Connection (?:closed|reset)|Disconnected)</F-MLFFORGET> (?:by|from)%(__authng_user)s <HOST>%(__on_port_opt)s\s+\[preauth\]\s*$
+mdre-ddos-other = ^<F-MLFFORGET>(Connection closed|Disconnected)</F-MLFFORGET> (?:by|from)%(__authng_user)s <HOST>%(__on_port_opt)s\s+\[preauth\]\s*$

-mdre-extra = ^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>%(__on_port_opt)s:\s*14: No(?: supported)? authentication methods available
+mdre-extra = ^Received <F-MLFFORGET>disconnect</F-MLFFORGET> from <HOST>%(__on_port_opt)s:\s*14: No supported authentication methods available
            ^Unable to negotiate with <HOST>%(__on_port_opt)s: no matching <__alg_match> found.
            ^Unable to negotiate a <__alg_match>
            ^no matching <__alg_match> found:
@@ -90,17 +84,6 @@ mdre-aggressive = %(mdre-ddos)s
 # mdre-extra-other is fully included within mdre-ddos-other:
 mdre-aggressive-other = %(mdre-ddos-other)s

-# Parameter "publickey": nofail (default), invalid, any, ignore
-publickey = nofail
-# consider failed publickey for invalid users only:
-cmnfailre-failed-pub-invalid = ^Failed publickey for invalid user <F-USER>(?P<cond_user>\S+)|(?:(?! from ).)*?</F-USER> from <HOST>%(__on_port_opt)s(?: ssh\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$)
-# consider failed publickey for valid users too (don't need RE, see cmnfailed):
-cmnfailre-failed-pub-any =
-# same as invalid, but consider failed publickey for valid users too, just as no failure (helper to get IP and user-name only, see cmnfailed):
-cmnfailre-failed-pub-nofail = <cmnfailre-failed-pub-invalid>
-# don't consider failed publickey as failures (don't need RE, see cmnfailed):
-cmnfailre-failed-pub-ignore =
-
 cfooterre = ^<F-NOFAIL>Connection from</F-NOFAIL> <HOST>

 failregex = %(cmnfailre)s
@@ -51,26 +51,6 @@

 [Definition]

-# Parameter "method" can be used to specifiy request method
-req-method = \S+
-# Usage example (for jail.local):
-#   filter = traefik-auth[req-method="GET|POST|HEAD"]
-
-failregex = ^<HOST> \- <usrre-<mode>> \[\] \"(?:<req-method>) [^\"]+\" 401\b
+failregex = ^<HOST> \- (?!- )\S+ \[\] \"(GET|POST|HEAD) [^\"]+\" 401\b

 ignoreregex =
-
-# Parameter "mode": normal (default), ddos or aggressive
-# Usage example (for jail.local):
-#   [traefik-auth]
-#   mode = aggressive
-#   # or another jail (rewrite filter parameters of jail):
-#   [traefik-auth-ddos]
-#   filter = traefik-auth[mode=ddos]
-#
-mode = normal
-
-# part of failregex matches user name (must be available in normal mode, must be empty in ddos mode, and both for aggressive mode):
-usrre-normal = (?!- )<F-USER>\S+</F-USER>
-usrre-ddos = -
-usrre-aggressive = <F-USER>\S+</F-USER>
@@ -44,6 +44,44 @@ before = paths-debian.conf
 # MISCELLANEOUS OPTIONS
 #

+# "bantime.increment" allows to use database for searching of previously banned ip's to increase a 
+# default ban time using special formula, default it is banTime * 1, 2, 4, 8, 16, 32...
+#bantime.increment = true
+
+# "bantime.rndtime" is the max number of seconds using for mixing with random time 
+# to prevent "clever" botnets calculate exact time IP can be unbanned again:
+#bantime.rndtime = 
+
+# "bantime.maxtime" is the max number of seconds using the ban time can reach (don't grows further)
+#bantime.maxtime = 
+
+# "bantime.factor" is a coefficient to calculate exponent growing of the formula or common multiplier,
+# default value of factor is 1 and with default value of formula, the ban time 
+# grows by 1, 2, 4, 8, 16 ...
+#bantime.factor = 1
+
+# "bantime.formula" used by default to calculate next value of ban time, default value bellow,
+# the same ban time growing will be reached by multipliers 1, 2, 4, 8, 16, 32...
+#bantime.formula = ban.Time * (1<<(ban.Count if ban.Count<20 else 20)) * banFactor
+#
+# more aggressive example of formula has the same values only for factor "2.0 / 2.885385" :
+#bantime.formula = ban.Time * math.exp(float(ban.Count+1)*banFactor)/math.exp(1*banFactor)
+
+# "bantime.multipliers" used to calculate next value of ban time instead of formula, coresponding 
+# previously ban count and given "bantime.factor" (for multipliers default is 1);
+# following example grows ban time by 1, 2, 4, 8, 16 ... and if last ban count greater as multipliers count, 
+# always used last multiplier (64 in example), for factor '1' and original ban time 600 - 10.6 hours
+#bantime.multipliers = 1 2 4 8 16 32 64
+# following example can be used for small initial ban time (bantime=60) - it grows more aggressive at begin,
+# for bantime=60 the multipliers are minutes and equal: 1 min, 5 min, 30 min, 1 hour, 5 hour, 12 hour, 1 day, 2 day
+#bantime.multipliers = 1 5 30 60 300 720 1440 2880
+
+# "bantime.overalljails" (if true) specifies the search of IP in the database will be executed 
+# cross over all jails, if false (dafault), only current jail of the ban IP will be searched
+#bantime.overalljails = false
+
+# --------------------
+
 # "ignoreself" specifies whether the local resp. own IP addresses should be ignored
 # (default is true). Fail2ban will not ban a host which matches such addresses.
 #ignoreself = true
@@ -171,28 +209,28 @@ banaction = iptables-multiport
 banaction_allports = iptables-allports

 # The simplest action to take: ban only
-action_ = %(banaction)s[bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
+action_ = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]

 # ban & send an e-mail with whois report to the destemail.
-action_mw = %(action_)s
-            %(mta)s-whois[sender="%(sender)s", dest="%(destemail)s", protocol="%(protocol)s", chain="%(chain)s"]
+action_mw = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
+            %(mta)s-whois[name=%(__name__)s, sender="%(sender)s", dest="%(destemail)s", protocol="%(protocol)s", chain="%(chain)s"]

 # ban & send an e-mail with whois report and relevant log lines
 # to the destemail.
-action_mwl = %(action_)s
-             %(mta)s-whois-lines[sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]
+action_mwl = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
+             %(mta)s-whois-lines[name=%(__name__)s, sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]

 # See the IMPORTANT note in action.d/xarf-login-attack for when to use this action
 #
 # ban & send a xarf e-mail to abuse contact of IP address and include relevant log lines
 # to the destemail.
-action_xarf = %(action_)s
+action_xarf = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
             xarf-login-attack[service=%(__name__)s, sender="%(sender)s", logpath="%(logpath)s", port="%(port)s"]

 # ban IP on CloudFlare & send an e-mail with whois report and relevant log lines
 # to the destemail.
 action_cf_mwl = cloudflare[cfuser="%(cfemail)s", cftoken="%(cfapikey)s"]
-                %(mta)s-whois-lines[sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]
+                %(mta)s-whois-lines[name=%(__name__)s, sender="%(sender)s", dest="%(destemail)s", logpath="%(logpath)s", chain="%(chain)s"]

 # Report block via blocklist.de fail2ban reporting service API
 # 
@@ -202,7 +240,7 @@ action_cf_mwl = cloudflare[cfuser="%(cfemail)s", cftoken="%(cfapikey)s"]
 # in your `jail.local` globally (section [DEFAULT]) or per specific jail section (resp. in 
 # corresponding jail.d/my-jail.local file).
 #
-action_blocklist_de  = blocklist_de[email="%(sender)s", service="%(__name__)s", apikey="%(blocklist_de_apikey)s", agent="%(fail2ban_agent)s"]
+action_blocklist_de  = blocklist_de[email="%(sender)s", service=%(filter)s, apikey="%(blocklist_de_apikey)s", agent="%(fail2ban_agent)s"]

 # Report ban via badips.com, and use as blacklist
 #
@@ -333,7 +371,7 @@ maxretry = 1
 [openhab-auth]

 filter = openhab
-banaction = %(banaction_allports)s
+action = iptables-allports[name=NoAuthFailures]
 logpath = /opt/openhab/logs/request.log


@@ -440,7 +478,6 @@ backend  = %(syslog_backend)s

 port     = http,https
 logpath  = /var/log/tomcat*/catalina.out
-#logpath  = /var/log/guacamole.log

 [monit]
 #Ban clients brute-forcing the monit gui login
@@ -707,8 +744,8 @@ logpath  = /var/log/named/security.log
 [nsd]

 port     = 53
-action_  = %(default/action_)s[name=%(__name__)s-tcp, protocol="tcp"]
-           %(default/action_)s[name=%(__name__)s-udp, protocol="udp"]
+action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
+           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
 logpath = /var/log/nsd.log


@@ -719,8 +756,9 @@ logpath = /var/log/nsd.log
 [asterisk]

 port     = 5060,5061
-action_  = %(default/action_)s[name=%(__name__)s-tcp, protocol="tcp"]
-           %(default/action_)s[name=%(__name__)s-udp, protocol="udp"]
+action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
+           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
+           %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s"]
 logpath  = /var/log/asterisk/messages
 maxretry = 10

@@ -728,8 +766,9 @@ maxretry = 10
 [freeswitch]

 port     = 5060,5061
-action_  = %(default/action_)s[name=%(__name__)s-tcp, protocol="tcp"]
-           %(default/action_)s[name=%(__name__)s-udp, protocol="udp"]
+action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
+           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]
+           %(mta)s-whois[name=%(__name__)s, dest="%(destemail)s"]
 logpath  = /var/log/freeswitch.log
 maxretry = 10

@@ -814,23 +853,11 @@ logpath = /var/log/ejabberd/ejabberd.log
 [counter-strike]

 logpath = /opt/cstrike/logs/L[0-9]*.log
+# Firewall: http://www.cstrike-planet.com/faq/6
 tcpport = 27030,27031,27032,27033,27034,27035,27036,27037,27038,27039
 udpport = 1200,27000,27001,27002,27003,27004,27005,27006,27007,27008,27009,27010,27011,27012,27013,27014,27015
-action_  = %(default/action_)s[name=%(__name__)s-tcp, port="%(tcpport)s", protocol="tcp"]
-           %(default/action_)s[name=%(__name__)s-udp, port="%(udpport)s", protocol="udp"]
-
-[softethervpn]
-port     = 500,4500
-protocol = udp
-logpath  = /usr/local/vpnserver/security_log/*/sec.log
-
-[gitlab]
-port    = http,https
-logpath = /var/log/gitlab/gitlab-rails/application.log
-
-[grafana]
-port    = http,https
-logpath = /var/log/grafana/grafana.log
+action  = %(banaction)s[name=%(__name__)s-tcp, port="%(tcpport)s", protocol="tcp", chain="%(chain)s", actname=%(banaction)s-tcp]
+           %(banaction)s[name=%(__name__)s-udp, port="%(udpport)s", protocol="udp", chain="%(chain)s", actname=%(banaction)s-udp]

 [bitwarden]
 port    = http,https
@@ -882,8 +909,8 @@ findtime     = 1
 [murmur]
 # AKA mumble-server
 port     = 64738
-action_  = %(default/action_)s[name=%(__name__)s-tcp, protocol="tcp"]
-           %(default/action_)s[name=%(__name__)s-udp, protocol="udp"]
+action   = %(banaction)s[name=%(__name__)s-tcp, port="%(port)s", protocol=tcp, chain="%(chain)s", actname=%(banaction)s-tcp]
+           %(banaction)s[name=%(__name__)s-udp, port="%(port)s", protocol=udp, chain="%(chain)s", actname=%(banaction)s-udp]
 logpath  = /var/log/mumble-server/mumble-server.log


@@ -0,0 +1,91 @@
+fail2ban (0.10.2-1) unstable; urgency=medium
+
+  This version is a major development leap forward to provide
+  IPv6 support, which also required extensions to the configuration
+  system.  That is why it is not unlikely that configuration left from the
+  previous version(s) would either not work or would not work as intended.
+
+  You are advised to accept new configuration and adjust it for your
+  customizations (if any).  See changelog.Debian.gz for more information.
+
+ -- Yaroslav Halchenko <debian@onerussian.com>  Sun, 21 Jan 2018 22:25:26 -0500
+
+fail2ban (0.9.0+git48-gabcab00-1) experimental; urgency=low
+
+  [ Yaroslav Halchenko ]
+  * This version went through big refactoring which allowed to gain new
+    features such as multiline matching (see upstream's changelog for more
+    information).
+  * Although .local files are still supported, customizations are advised
+    to be provided under corresponding .d/ directories.  E.g. see
+    /etc/fail2ban/jail.d/defaults-debian.conf which is where now sshd
+    jail is enabled by default to match previous behavior of Fail2Ban in
+    Debian.
+
+  [ Daniel Schaal ]
+  * All jails definitions were rewritten to become more concise and uniform.
+    From this version on log paths are defined in distro specific files,
+    for Debian this is in /etc/fail2ban/paths-debian.conf.
+
+ -- Yaroslav Halchenko <debian@onerussian.com>  Tue, 25 Mar 2014 08:38:31 -0400
+
+fail2ban (0.8.11-1) unstable; urgency=low
+
+  * retroactive for 0.8.9: by default iptables-* actions do not simply
+    DROP packets from offending IP but rather reject with
+    icmp-port-unreachable.  If DROP behaviour is preferable, provide
+    config/action.d/iptables-blocktype.local with [Init] section defining
+    blocktype = DROP or override action definition to provide
+    blocktype=DROP option in jail.local
+  * Many failregex's were tight-up in this release which could
+    theoretically effect operation in comparison to previous release(s).
+
+ -- Yaroslav Halchenko <debian@onerussian.com>  Sat, 16 Nov 2013 22:27:50 -0500
+
+fail2ban (0.8.4-3) unstable; urgency=low
+
+  * Jail named-refused-udp is unsafe and opens possibility for easy DoS,
+    thus discouraged to be used, and commented out (see #583364 for more
+    information).
+
+ -- Yaroslav Halchenko <debian@onerussian.com>  Mon, 28 Jun 2010 22:12:22 -0400
+
+fail2ban (0.7.1-0.2) unstable; urgency=low
+
+  fail2ban 0.7 is a complete rewrite of the 0.6 version, and if you
+  customized any of provided configuration or startup files
+  (/etc/default/fail2ban, /etc/fail2ban.conf, /etc/init.d/fail2ban),
+  please read further. The configuration scheme has changed upstream:
+  0.7 ignores /etc/fail2ban.conf and instead uses a split configuration
+  under /etc/fail2ban/. To retain your customizations, for example to
+  monitor anything other than sshd, you will need to set them under that
+  new directory; use *.local files for customizations. Please see
+  /usr/share/doc/fail2ban/README.Debian.gz and
+  http://fail2ban.sourceforge.net for further description of new
+  configuration scheme. Detailed documentation is under development (see
+  #400416).  When you are satisfied with the new settings, please delete
+  /etc/fail2ban.conf to avoid confusion.
+
+  Fail2ban 0.7 uses client/server architecture and fail2ban-client is to
+  substitute fail2ban command to provide an interface between the user and
+  fail2ban-server. That is why some command line parameters present in
+  fail2ban 0.6 are invalid in fail2ban-client. Such change affects
+  /etc/default/fail2ban; you should review that file if you customized it.
+  Please enable sections as directed in README.Debian.gz mentioned above.
+  You must use newly shipped init.d/fail2ban, or otherwise fail2ban will
+  not start.
+
+  This note was rewritten in release 0.7.5-2 to clarify its meaning.
+
+ -- Yaroslav Halchenko <debian@onerussian.com>  Sat,  9 Dec 2006 18:24:36 -0500
+
+fail2ban (0.6.0-4) unstable; urgency=low
+
+  In this version the new section ApacheAttacks was introduced to ban IPs
+  which are found to run some known attack on the host. For now it captures
+  just awstats and mambo related attacks. To make this feature work, the bug of
+  wrongly specified timeregexp for Apache's access.log file was fixed.
+  Besides that group of log files has changed to be adm, and now they are
+  readable by the group.
+
+ -- Yaroslav Halchenko <debian@onerussian.com>  Fri, 10 Feb 2006 13:05:07 -0500
@@ -0,0 +1,251 @@
+fail2ban (>=0.7.0) for Debian
+-----------------------------
+
+This package is ~99% identical to the upstream version. Few features
+could have been added but not yet propagated into upstream version and
+some modifications might be Debian-specific. Debian specific jail.conf
+file is shipped. Original upstream file is available from
+/usr/share/doc/fail2ban/examples/jail.conf
+
+Currently, the major difference with upstream: python libraries are
+placed under /usr/share/fail2ban instead of /usr/lib/fail2ban to
+comply with policy regarding architecture independent resources.
+
+fail2ban supports both nftables and iptables.
+
+Shorewall and startup sequence (#847728)
+----------------------------------------
+
+If you are using systemd, create a
+/etc/systemd/system/fail2ban.service.d/override.conf with contents:
+
+[Unit]
+Requires=shorewall.service
+After=shorewall.service
+
+go guarantee a proper sequence of startup/shutdown (shorewall should
+be started before fail2ban, and stopped after).  Similar settings
+could be adopted for other firewall solutions.
+
+
+Upgrade from 0.6 versions:
+-------------------------
+
+* New Config Files Format:
+
+If you had introduced your own sections in /etc/fail2ban.conf, you
+would need manually to convert them into a new format. At minimum you
+need to create /etc/fail2ban/filter.d/NAME.local (leave .conf files
+for me and upstream please to avoid any conflicts -- introduce your
+changes in .local) with failregex in [Definition] section. And provide
+appropriate jail definition in /etc/fail2ban/jail.local
+
+
+* Enabled Sections:
+
+Only handling of ssh files is enabled by default. If you want to use
+fail2ban with apache, please enable apache section manually in
+/etc/fail2ban/jail.local by including next lines:
+
+[apache]
+enabled = true
+
+NOTE: -e command line parameter is non existent in 0.7.x
+
+
+* Interpolations vs actions/filters parameters:
+
+For details see #398739 or wait for a closure of #400416
+
+Every pair of .conf and then .local (if exists) files is read
+separately from any other configuration file, so interpolations cannot
+penetrate from jail.* into actions.d/*. To overcome this, it is
+necessary to create a PARAMETER which can be substituted in actions
+[Definition] section, if it is also defined in the [Init] section of
+that file and is used in place of necessary allocation as <PARAMETER>
+tag. Parameters can be specified in the definitions within
+jail.{conf,local}. For instance, 1 lengthy example, where the same
+name "fwchain" is used both as interpolation (in jail.local) and as a
+parameter (in iptables-flex.local) (from #398739)
+
+==> /etc/fail2ban/jail.local <==
+[DEFAULT]
+action = iptables-flex[name=%(__name__)s, port=%(port)s, fwchain=%(fwchain)s, post_start_commands=%(post_start_commands)s, pre_end_commands=%(pre_end_commands)s]
+fwchain = INPUT
+[ssh]
+fwchain = ssh-tarpit
+==> /etc/fail2ban/action.d/iptables-flex.local <==
+[Definition]
+actionstart = iptables -N fail2ban-<name>
+               iptables -I <fwchain> -m state --state NEW -p <protocol> --dport <port> -j fail2ban-<name>
+               iptables -I <fwchain> -j <whitelist>
+actionstop  = iptables -D <fwchain> -j <whitelist>
+               iptables -D <fwchain> -m state --state NEW -p <protocol> --dport <port> -j fail2ban-<name>
+               iptables -F fail2ban-<name>
+               iptables -X fail2ban-<name>
+actioncheck = iptables -n -L <fwchain> | grep -q fail2ban-<name>
+actionban   = iptables -I fail2ban-<name> 1 -s <ip> -j DROP
+actionunban = iptables -D fail2ban-<name> -s <ip> -j DROP
+[Init]
+whitelist = ssh-whitelist
+fwchain = INPUT
+name = default
+port = ssh
+protocol = tcp
+
+
+* Multiport banning: Comment for #373592, #545971
+
+iptables-multiport action is now default banaction (file jail.conf, to
+be customized within jail.local). Therefore assure that you have built
+multiport module if you use custom kernel.
+
+If you would like to ban all ports for that host, just redefine
+fwban/fwunban commands to don't have --dport %(port)s statement at
+all, or use shorewall, where actionban bans whole IP.
+
+* Blocking of NEW connections only
+Comment for the wishlist #350746.
+
+It might be benefitial in some cases to ban only new connections. For
+that just use iptables-new action instead of default banaction
+
+/etc/fail2ban/jail.local:
+
+[DEFAULT]
+banaction=iptables-new
+
+(you can override banaction within interesting for you section).
+ Also you can redefine the whole action parameter if you like.
+
+
+* Interaction with ipmasq
+  Comment to #461417
+
+Although fail2ban should detect and recreate missing chains if the external
+command wipes out iptables, it is better to explicitly to force-reload
+fail2ban. For this reason there is examples/ipmasq-ZZZzzz|fail2ban.rul file is
+shipped along to be installed under name ZZZzzz|fail2ban.rul within
+/etc/ipmasq.
+
+* Interaction with logrotate with custom logtarget
+  Comment to #631917
+
+if you use an alternative logtarget (e.g. SYSLOG) thus not using
+/var/log/fail2ban.log you should divert logrotate configuration into
+a disabled state, e.g.
+
+sudo dpkg-divert --rename --divert \
+	 /etc/logrotate.d/fail2ban.disabled /etc/logrotate.d/fail2ban
+
+
+Troubleshooting:
+---------------
+
+* Updated failregex:
+
+To resolve the security bug #330827 [1] failregex expressions must
+provide a named group (?P<host>...) as a placeholder of the abuser's
+host. Alternative tag (since 0.7.5) can be "<HOST>". The naming of the
+group was introduced to capture possible future generalizations of
+failregex to provide even more information.
+
+[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=330827
+
+You might benefit from using fail2ban-regex command shipped along to
+construct and debug your failregex statements.
+
+* "Interpolations" in the config file:
+
+Since version 0.6.0-3 to reduce duplication, thus to improve
+readability of the config file, interpolations provided by the module
+ConfigParser are used. If you had custom sections defined before, you
+might benefit from updating config file and adding appropriate
+information for the new sections.
+
+N.B. If you have some nice additional sections defined, I would really
+appreciate if you share them with me or upstream author, so they could
+be eventually included in the fail2ban package for general use by the
+rest of the community.
+
+
+* Mailing:
+
+Since actions.d/mail*.conf commands rely on presence of "mail"
+command, mailx package (or another package providing mailx
+functionality such as mailutils) is required if those actions are
+activated in jail.{conf,local}.
+
+
+* Dirty exit:
+
+If firewall rules gets cleaned out before fail2ban exits (like was
+happening with firestarter), errors get reported during the exit of
+fail2ban, but they are "safe" and can be ignored.
+
+
+** SSHD Configuration Specific Problems
+
+* Ban "Not allowed" attempts:
+
+Make sure that you have
+ChallengeResponseAuthentication no
+PasswordAuthentication yes
+
+Details from the bug report #350980 [2]
+
+[2]  http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=350980
+
+
+* Not caught attempts to login as root
+
+On the boxes running older versions of openssh (e.g. sarge
+distribution) in the case when PermitRootLogin is set to something
+else than "yes" and iff AllowUsers is active, failed root logins do
+not confirm to the standard logging message -- they omit the source
+IP, thus allowing attack to persist since such messages are not caught
+by fail2ban.
+
+
+* Bantime:
+
+An IP is banned for "bantime" not since the last failed login attempt
+from the IP, but rather since the moment when failed login was
+detected by fail2ban. Thus, if fail2ban gets [re]started, any IP which
+had enough of failed logins with durations less than "findtime" between
+them prior to the [re]start moment, will be banned for
+"bantime" since [re]start moment, not since the last failed login
+time.
+
+* Findtime:
+
+"Findtime" option of a jail actually defines a duration to reset the
+counter of failed login attempts, if no new attempt was detected within
+that time frame (i.e.  within "findtime").
+
+See
+http://www.fail2ban.org/wiki/index.php/MANUAL_0_8#Jail_Options
+for more information on jail options.
+
+
+* Syslog entries can be 'forged' by a regular user
+
+From
+http://www.fail2ban.org/wiki/index.php/MANUAL_0_8#Possibility_of_DOS_attack_by_a_local_user
+
+Especially on systems which provide ssh/CGI/PHP services to unknown
+users it is possible to block other users from ssh and probably other
+access as a unprivileged user may issue:
+
+logger -p auth.warning -t 'sshd[123]' 'Illegal user user1 from 1.2.3.4'
+
+N.B. chmod o-x /usr/bin/logger should provide at least obfuscation
+solution
+
+Or the malicious user may write via PHP's openlog()/syslog() to syslog.
+
+P.S. Anyone is welcome to recommend proper security solution to this
+issue, such as an alternative to sysklogd which allows better control
+over users logging to specific facilities (such as AUTH)
+
+ -- Yaroslav Halchenko <debian@onerussian.com>, Mon, 22 Jan 2018 10:37:00 -0500
@@ -0,0 +1,10 @@
+* completions installation
+
+W: fail2ban: package-installs-into-obsolete-dir etc/bash_completion.d/ : ^etc/bash_completion.d/ -> usr/share/bash-completion/completions (see also https://bugs.debian.org/776954)
+W: fail2ban: package-installs-into-obsolete-dir etc/bash_completion.d/fail2ban : ^etc/bash_completion.d/ -> usr/share/bash-completion/completions (see also https://bugs.debian.org/776954)
+
+* Find proper answer to "Syslog entries can be 'forged' by a regular
+   user" mentioned in README.Debian
+
+ -- Yaroslav O. Halchenko <debian@onerussian.com>  Wed,  6 Dec 2006 22:14:26 -0500
+
@@ -0,0 +1 @@
+nopycentral.patch
@@ -0,0 +1,40 @@
+diff -x '*~' -x .svn -Naur trunk/debian/control trunk.backports/debian/control
+--- trunk/debian/control	2006-10-23 00:57:02.000000000 -0400
+++ trunk.backports/debian/control	2006-12-04 08:45:25.000000000 -0500
+@@ -4,13 +4,13 @@
+ Maintainer: Yaroslav Halchenko <debian@onerussian.com>
+ Uploaders: Barak Pearlmutter <bap@debian.org>
+ Build-Depends: debhelper (>= 5.0.37.2), dpatch
+-Build-Depends-Indep: python, python-dev, help2man, python-central (>= 0.5.6)
+Build-Depends-Indep: python, python2.4, python2.4-dev, help2man
+ XS-Python-Version: current, >= 2.4
+ Standards-Version: 3.7.2
+ 
+ Package: fail2ban
+ Architecture: all
+-Depends: ${python:Depends}, iptables, lsb-base (>=2.0-7)
+Depends: python2.4, iptables, lsb-base (>=2.0-7)
+ Suggests: python-gamin
+ XB-Python-Version: ${python:Versions}
+ Description: bans IPs that cause multiple authentication errors
+diff -x '*~' -x .svn -Naur trunk/debian/rules trunk.backports/debian/rules
+--- trunk/debian/rules	2006-11-11 21:19:14.000000000 -0500
+++ trunk.backports/debian/rules	2006-12-04 08:45:45.000000000 -0500
+@@ -39,7 +39,7 @@
+ 	dh_installdirs
+ 
+ 	# Add here commands to install the package into debian/fail2ban.
+-	python setup.py install --root=$(DESTDIR) --no-compile
+	python2.4 setup.py install --root=$(DESTDIR) --no-compile
+ 	#X Evil - must be removed after Debian switches over to 2.4, now
+ 	#  distutils.setup will override the enterpreter line to /usr/bin/python
+ 	install fail2ban-server fail2ban-client $(DESTDIR)/usr/bin
+@@ -62,7 +62,7 @@
+ 	dh_installlogrotate
+ 	dh_installinit -- defaults 99
+ 	dh_installman man/*.1
+-	dh_pycentral
+	dh_python
+ 	dh_link
+ 	dh_compress
+ 	dh_fixperms
@@ -0,0 +1,48 @@
+Source: fail2ban
+Section: net
+Priority: optional
+Maintainer: Debian Python Team <team+python@tracker.debian.org>
+Uploaders: Yaroslav Halchenko <debian@onerussian.com>,
+ Sylvestre Ledru <sylvestre@debian.org>
+Build-Depends:
+ debhelper-compat (= 9)
+ , debhelper (>= 9.20160709)
+ , dh-python
+ , python3
+ , python3-setuptools
+ , python3-pyinotify
+ , sqlite3
+ , 2to3
+Homepage: http://www.fail2ban.org
+Vcs-Git: https://salsa.debian.org/python-team/packages/fail2ban.git
+Vcs-Browser: https://salsa.debian.org/python-team/packages/fail2ban
+Standards-Version: 4.4.1
+
+
+Package: fail2ban
+Architecture: all
+Depends: ${python3:Depends}, ${misc:Depends}, lsb-base (>=2.0-7)
+Recommends: nftables | iptables, whois, python3-pyinotify, python3-systemd
+Suggests: mailx, system-log-daemon, monit, sqlite3
+Description: ban hosts that cause multiple authentication errors
+ Fail2ban monitors log files (e.g. /var/log/auth.log,
+ /var/log/apache/access.log) and temporarily or persistently bans
+ failure-prone addresses by updating existing firewall rules.  Fail2ban
+ allows easy specification of different actions to be taken such as to ban
+ an IP using iptables or hostsdeny rules, or simply to send a notification
+ email.
+ .
+ By default, it comes with filter expressions for various services
+ (sshd, apache, proftpd, sasl, etc.) but configuration can be
+ easily extended for monitoring any other text file.  All filters and
+ actions are given in the config files, thus fail2ban can be adopted
+ to be used with a variety of files and firewalls.  Following recommends
+ are listed:
+ .
+  - iptables/nftables -- default installation uses iptables for banning.
+    nftables is also supported. You most probably need it
+  - whois -- used by a number of *mail-whois* actions to send notification
+    emails with whois information about attacker hosts. Unless you will use
+    those you don't need whois
+  - python3-pyinotify -- unless you monitor services logs via systemd, you
+    need pyinotify for efficient monitoring for log files changes
@@ -0,0 +1,31 @@
+This package was originally debianized by Yaroslav Halchenko
+<debian@onerussian.com> on Mon Jul  4 14:41:34 HST 2005
+
+It was downloaded from http://www.sourceforge.net/projects/fail2ban
+
+Author: Cyril Jaquier: <cyril.jaquier@fail2ban.org>
+        http://fail2ban.sourceforge.net
+
+Copyright: 2004-2009 Cyril Jaquier
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 2 of the License, or
+(at your option) any later version.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the
+Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston,
+MA 02110-1301, USA.
+
+On Debian systems, the complete text of the GNU General Public
+License, version 2, can be found in /usr/share/common-licenses/GPL-2.
+
+The Debian packaging is (C) 2006-2011, Yaroslav Halchenko <debian@onerussian.com>
+and is licensed under the GPL, see above.
+
@@ -0,0 +1,2 @@
+[sshd]
+enabled = true
@@ -0,0 +1,3 @@
+README.md
+TODO
+doc/run-rootless.txt
@@ -0,0 +1,39 @@
+# This file is part of Fail2Ban.
+#
+# Fail2Ban is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# Fail2Ban is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Fail2Ban; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+#
+# Author: Cyril Jaquier
+# 
+# $Revision$
+
+# Command line options for Fail2Ban. Refer to "fail2ban-client -h" for
+# valid options.
+FAIL2BAN_OPTS=""
+
+# Run fail2ban as a different user. If not set, fail2ban
+# will run as root.
+#
+# The user is not created automatically.
+# The user can be created e.g. with
+#    useradd --system --no-create-home --home-dir / --groups adm fail2ban
+# Log files are readable by group adm by default. Adding the fail2ban
+# user to this group allows it to read the logfiles.
+#
+# Another manual step that needs to be taken is to allow write access
+# for fail2ban user to fail2ban log files. The /etc/init.d/fail2ban
+# script will change the ownership when starting fail2ban. Logrotate
+# needs to be configured separately, see /etc/logrotate.d/fail2ban.
+#
+# FAIL2BAN_USER="fail2ban"
@@ -0,0 +1,19 @@
+/var/log/fail2ban.log {
+
+    weekly
+    rotate 4
+    compress
+    # Do not rotate if empty
+    notifempty
+
+    delaycompress
+    missingok
+    postrotate
+	fail2ban-client flushlogs 1>/dev/null
+    endscript
+
+    # If fail2ban runs as non-root it still needs to have write access
+    # to logfiles.
+    # create 640 fail2ban adm
+    create 640 root adm
+}
@@ -0,0 +1,18 @@
+[DEFAULT]
+# the default branch for upstream sources:
+upstream-branch = upstream
+# the default branch for the debian patch:
+debian-branch = debian-releases/experimental
+# use pristine-tar
+# pristine-tar = True
+# the default tag formats used:
+upstream-tag = %(version)s
+debian-tag = debian/%(version)s
+
+
+# Options only affecting git-buildpackage
+[git-buildpackage]
+# use this for more svn-buildpackage like behaviour:
+export-dir = ../build-area/
+tarball-dir = ../tarballs/
+
@@ -0,0 +1,25 @@
+From d1afbb566f0304487b5d578b4aacef8e647ee74b Mon Sep 17 00:00:00 2001
+From: Yaroslav Halchenko <debian@onerussian.com>
+Date: Sun, 21 Jan 2018 20:16:43 -0500
+Subject: [PATCH 2/4] ENH: verify that use_stock_cfg was not provided while
+ overriding it
+
+Just found this possibly confusing to outside programmer aspect
+so decided to make it more explicit
+---
+ fail2ban/tests/fail2banclienttestcase.py | 2 ++
+ 1 file changed, 2 insertions(+)
+
+Index: fail2ban/fail2ban/tests/fail2banclienttestcase.py
+===================================================================
+--- fail2ban.orig/fail2ban/tests/fail2banclienttestcase.py
+++ fail2ban/fail2ban/tests/fail2banclienttestcase.py
+@@ -173,6 +173,8 @@ def _start_params(tmp, use_stock=False,
+ 			"""Filters list of 'files' to contain only directories (under dir)"""
+ 			return [f for f in files if isdir(pjoin(dir, f))]
+ 		shutil.copytree(STOCK_CONF_DIR, cfg, ignore=ig_dirs)
+		assert use_stock_cfg is None, \
+			"We are about to overload use_stock_cfg from the one provided %s" % repr(use_stock_cfg)
+ 		if use_stock_cfg is None: use_stock_cfg = ('action.d', 'filter.d')
+ 		# replace fail2ban params (database with memory):
+ 		r = re.compile(r'^dbfile\s*=')
@@ -0,0 +1,13 @@
+Index: fail2ban/files/debian-initd
+===================================================================
+--- fail2ban.orig/files/debian-initd
+++ fail2ban/files/debian-initd
+@@ -28,7 +28,7 @@ NAME="fail2ban"
+ 
+ # fail2ban-client is not a daemon itself but starts a daemon and
+ # loads its with configuration
+-DAEMON="/usr/local/bin/$NAME-client"
+DAEMON="/usr/bin/$NAME-client"
+ SCRIPTNAME="/etc/init.d/$NAME"
+ 
+ # Ad-hoc way to parse out socket file name
@@ -0,0 +1,30 @@
+From: Yaroslav Halchenko <debian@onerussian.com>
+Date: Fri, 8 Feb 2008 00:40:57 -0500
+Subject: tune ups in upstream manpages to direct users to use reportbug
+
+Index: fail2ban/man/fail2ban-client.1
+===================================================================
+--- fail2ban.orig/man/fail2ban-client.1
+++ fail2ban/man/fail2ban-client.1
+@@ -470,7 +470,7 @@ the action <ACT> for <JAIL>
+ .SH FILES
+ \fI/etc/fail2ban/*\fR
+ .SH "REPORTING BUGS"
+-Report bugs to https://github.com/fail2ban/fail2ban/issues
+Report bugs via Debian bug tracking system \fIhttp://www.debian.org/Bugs/\fR .
+ .SH "SEE ALSO"
+ .br 
+ fail2ban-server(1)
+Index: fail2ban/man/fail2ban-server.1
+===================================================================
+--- fail2ban.orig/man/fail2ban-server.1
+++ fail2ban/man/fail2ban-server.1
+@@ -69,7 +69,7 @@ display this help message
+ \fB\-V\fR, \fB\-\-version\fR
+ print the version (\fB\-V\fR returns machine\-readable short format)
+ .SH "REPORTING BUGS"
+-Report bugs to https://github.com/fail2ban/fail2ban/issues
+Report bugs via Debian bug tracking system \fIhttp://www.debian.org/Bugs/\fR .
+ .SH "SEE ALSO"
+ .br 
+ fail2ban-client(1)
@@ -0,0 +1,30 @@
+From: Yaroslav Halchenko <debian@onerussian.com>
+Subject: Remove all non-provided .service's within PartOf of fail2ban.service
+
+  As reported and corroborated in the bug report, this causes inability
+  of firewalld to restart.
+  Correct solution would involve making systemd smarter and tune up
+  of involved .service files.
+  Since Debian ATM doesn't provide any of those ({ip{,6}tables,ipset}.service)
+  files, it should be safe and generic enough to just prune them from PartOf
+
+  Thanks  Joe Cooper <swelljoe@gmail.com> and Sunil Mohan Adapa <sunil@medhas.org>
+  for the reports and nagging ;)
+
+Origin: Fedora, Debian
+Bug-Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=871993
+Last-Update: 2018-04-04
+
+Index: fail2ban/files/fail2ban.service.in
+===================================================================
+--- fail2ban.orig/files/fail2ban.service.in
+++ fail2ban/files/fail2ban.service.in
+@@ -2,7 +2,7 @@
+ Description=Fail2Ban Service
+ Documentation=man:fail2ban(1)
+ After=network.target iptables.service firewalld.service ip6tables.service ipset.service nftables.service
+-PartOf=iptables.service firewalld.service ip6tables.service ipset.service nftables.service
+PartOf=firewalld.service
+ 
+ [Service]
+ Type=simple
@@ -0,0 +1,22 @@
+--- a/fail2ban/tests/config/filter.d/zzz-generic-example.conf
+++ b/fail2ban/tests/config/filter.d/zzz-generic-example.conf
+@@ -8,7 +8,7 @@
+ # Read common prefixes. If any customizations available -- read them from
+ # common.local.  common.conf is a symlink to the original common.conf and
+ # should be copied (dereferenced) during installation
+-before = ../../../../config/filter.d/common.conf
+before = ../../../../../../../config/filter.d/common.conf
+ 
+ [Definition]
+ 
+--- a/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf
+++ b/fail2ban/tests/config/filter.d/zzz-sshd-obsolete-multiline.conf
+@@ -5,7 +5,7 @@
+ 
+ # Read common prefixes. If any customizations available -- read them from
+ # common.local
+-before = ../../../../config/filter.d/common.conf
+before = ../../../../../../../config/filter.d/common.conf
+ 
+ [DEFAULT]
+ 
@@ -0,0 +1,12 @@
+Index: fail2ban/files/fail2ban.service.in
+===================================================================
+--- fail2ban.orig/files/fail2ban.service.in
+++ fail2ban/files/fail2ban.service.in
+@@ -15,6 +15,7 @@ ExecReload=@BINDIR@/fail2ban-client relo
+ PIDFile=/run/fail2ban/fail2ban.pid
+ Restart=on-failure
+ RestartPreventExitStatus=0 255
+Environment="PYTHONNOUSERSITE=yes"
+ 
+ [Install]
+ WantedBy=multi-user.target
@@ -0,0 +1,10 @@
+Index: fail2ban/bin/fail2ban-testcases
+===================================================================
+--- fail2ban.orig/bin/fail2ban-testcases
+++ fail2ban/bin/fail2ban-testcases
+@@ -1,4 +1,4 @@
+-#!/usr/bin/env python
+#!/usr/bin/env python3
+ # emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: t -*-
+ # vi: set ft=python sts=4 ts=4 sw=4 noet :
+ """Script to run Fail2Ban tests battery
@@ -0,0 +1,11 @@
+--- fail2ban-0.11.1.orig/config/filter.d/roundcube-auth.conf
+++ fail2ban-0.11.1/config/filter.d/roundcube-auth.conf
+@@ -18,7 +18,7 @@ prefregex = ^\s*(\[\])?(%(__hostname)s\s
+ failregex = ^(?:FAILED login|Login failed) for <F-USER>.*</F-USER> from <HOST>(?:(?:\([^\)]*\))?\. (?:(?! from ).)*(?: user=(?P=user))? in \S+\.php on line \d+ \(\S+ \S+\))?$
+             ^(?:<[\w]+> )?Failed login for <F-USER>.*</F-USER> from <HOST> in session \w+( \(error: \d\))?$
+ 
+-ignoreregex = 
+ignoreregex = Could not connect to .* Connection refused
+ 
+ journalmatch = SYSLOG_IDENTIFIER=roundcube
+ 
@@ -0,0 +1,8 @@
+deb_path_to_common
+deb_init_paths
+deb_manpages_reportbug
+0002-ENH-verify-that-use_stock_cfg-was-not-provided-while.patch
+deb_no_iptables_service
+python3-test-suite.diff
+no-python-user.diff
+roundcude-update.diff
@@ -0,0 +1,98 @@
+#! /bin/sh
+# postinst script for fail2ban
+#
+# see: dh_installdeb(1)
+
+set -e
+
+# summary of how this script can be called:
+#        * <postinst> `configure' <most-recently-configured-version>
+#        * <old-postinst> `abort-upgrade' <new version>
+#        * <conflictor's-postinst> `abort-remove' `in-favour' <package>
+#          <new-version>
+#        * <deconfigured's-postinst> `abort-deconfigure' `in-favour'
+#          <failed-install-package> <version> `removing'
+#          <conflicting-package> <version>
+# for details, see http://www.debian.org/doc/debian-policy/ or
+# the debian-policy package
+#
+preversion=$2
+
+case "$1" in
+    configure)
+       # To fix the bug in generated by previous version files permissions
+       # also closes #352053
+
+       LOG=/var/log/fail2ban.log
+       touch $LOG
+       chown root:adm ${LOG}*
+       chmod 640 ${LOG}*
+        
+       # Note regarding changed configuration file
+       # Note regarding changed configuration file
+		if [ ! -z $preversion ]; then
+			if dpkg --compare-versions $preversion lt 0.7.1-1; then
+			cat <<EOF
+WARNING!
+  
+  Fail2ban 0.7 is a complete rewrite of the 0.6 version, and if you
+  customized any of provided configuration or startup files
+  (/etc/default/fail2ban, /etc/fail2ban.conf, /etc/init.d/fail2ban), please
+  read relevant entry in /usr/share/doc/fail2ban/NEWS.Debian.gz.
+  
+EOF
+			fi
+			if dpkg --compare-versions $preversion lt 0.5.4-5.14; then
+			cat <<EOF
+WARNING!
+
+ Configuration file /etc/fail2ban.conf, failregex configuration
+ parameter specifically, were changed in 0.5.4-5 to close reported
+ security breach, and in 0.5.4-5.14 to close few other bugs.
+
+updating from <0.5.4-5
+ Unless configuration file (or corresponding failregex'es) gets updated,
+ security breach is not closed and corresponding warning will be reported
+ by the fail2ban (in the log files).
+
+updating from <0.5.4-5.14
+ Bugs #329163, #331695 dealing with changed iptables rules
+ outside of fail2ban were fixed in 0.5.4-5.14, and require upgrade of the
+ configuration file (fwcheck option was introduced) to take full
+ advantage of the problem solution (otherwise some problems might
+ persist)
+
+ Please review the configuration file and make appropriate changes.
+ENJOY!
+
+EOF
+			fi
+		fi
+		;;
+
+    abort-upgrade|abort-remove|abort-deconfigure)
+
+    ;;
+
+    *)
+        echo "postinst called with unknown argument \`$1'" >&2
+        exit 1
+    ;;
+esac
+
+if dpkg-maintscript-helper supports mv_conffile 2>/dev/null; then
+    dpkg-maintscript-helper mv_conffile /etc/fail2ban/action.d/firewall-cmd-direct-new.conf /etc/fail2ban/action.d/firewallcmd-new.conf 0.8.13-1~ -- "$@"
+    dpkg-maintscript-helper mv_conffile /etc/fail2ban/filter.d/lighttpd-fastcgi.conf /etc/fail2ban/filter.d/suhosin.conf 0.8.13-1~ -- "$@"
+    dpkg-maintscript-helper mv_conffile /etc/fail2ban/filter.d/sasl.conf /etc/fail2ban/filter.d/postfix-sasl.conf 0.8.13-1~ -- "$@"
+    dpkg-maintscript-helper mv_conffile /etc/fail2ban/filter.d/couriersmtp.conf /etc/fail2ban/filter.d/courier-smtp.conf 0.9.0-1~ -- "$@"
+    dpkg-maintscript-helper mv_conffile /etc/fail2ban/filter.d/courierlogin.conf /etc/fail2ban/filter.d/courier-auth.conf 0.9.0-1~ -- "$@"
+fi
+
+# dh_installdeb will replace this with shell code automatically
+# generated by other debhelper scripts.
+
+#DEBHELPER#
+
+exit 0
+
+
@@ -0,0 +1,52 @@
+#! /bin/sh
+# postrm script for fail2ban
+#
+# see: dh_installdeb(1)
+
+set -e
+
+# summary of how this script can be called:
+#        * <postrm> `remove'
+#        * <postrm> `purge'
+#        * <old-postrm> `upgrade' <new-version>
+#        * <new-postrm> `failed-upgrade' <old-version>
+#        * <new-postrm> `abort-install'
+#        * <new-postrm> `abort-install' <old-version>
+#        * <new-postrm> `abort-upgrade' <old-version>
+#        * <disappearer's-postrm> `disappear' <r>overwrit>r> <new-version>
+# for details, see /usr/doc/packaging-manual/
+
+
+case "$1" in
+       purge|disappear)
+       				
+		# Remove configuration
+                rm -f /etc/fail2ban.conf
+		
+		# Remove logs
+		rm -f /var/log/fail2ban*
+		
+		# Remove sqlite db
+		rm -f /var/lib/fail2ban/fail2ban.sqlite3
+		;;
+       remove|upgrade|failed-upgrade|abort-install|abort-upgrade)
+		# nothing
+		# We may not delete the user fail2ban, as there may be
+		# files owned by it in /var/log/ and /etc/.
+        	;;
+esac
+
+if dpkg-maintscript-helper supports mv_conffile 2>/dev/null; then
+    dpkg-maintscript-helper mv_conffile /etc/fail2ban/action.d/firewall-cmd-direct-new.conf /etc/fail2ban/action.d/firewallcmd-new.conf 0.8.13-1~ -- "$@"
+    dpkg-maintscript-helper mv_conffile /etc/fail2ban/filter.d/lighttpd-fastcgi.conf /etc/fail2ban/filter.d/suhosin.conf 0.8.13-1~ -- "$@"
+    dpkg-maintscript-helper mv_conffile /etc/fail2ban/filter.d/sasl.conf /etc/fail2ban/filter.d/postfix-sasl.conf 0.8.13-1~ -- "$@"
+    dpkg-maintscript-helper mv_conffile /etc/fail2ban/filter.d/couriersmtp.conf /etc/fail2ban/filter.d/courier-smtp.conf 0.9.0-1~ -- "$@"
+    dpkg-maintscript-helper mv_conffile /etc/fail2ban/filter.d/courierlogin.conf /etc/fail2ban/filter.d/courier-auth.conf 0.9.0-1~ -- "$@"
+fi
+
+# dh_installdeb will replace this with shell code automatically
+# generated by other debhelper scripts.
+
+#DEBHELPER#
+
+
@@ -0,0 +1,15 @@
+#!/bin/sh
+
+set -e
+
+if dpkg-maintscript-helper supports mv_conffile 2>/dev/null; then
+    dpkg-maintscript-helper mv_conffile /etc/fail2ban/action.d/firewall-cmd-direct-new.conf /etc/fail2ban/action.d/firewallcmd-new.conf 0.8.13-1~ -- "$@"
+    dpkg-maintscript-helper mv_conffile /etc/fail2ban/filter.d/lighttpd-fastcgi.conf /etc/fail2ban/filter.d/suhosin.conf 0.8.13-1~ -- "$@"
+    dpkg-maintscript-helper mv_conffile /etc/fail2ban/filter.d/sasl.conf /etc/fail2ban/filter.d/postfix-sasl.conf 0.8.13-1~ -- "$@"
+    dpkg-maintscript-helper mv_conffile /etc/fail2ban/filter.d/couriersmtp.conf /etc/fail2ban/filter.d/courier-smtp.conf 0.9.0-1~ -- "$@"
+    dpkg-maintscript-helper mv_conffile /etc/fail2ban/filter.d/courierlogin.conf /etc/fail2ban/filter.d/courier-auth.conf 0.9.0-1~ -- "$@"
+fi
+
+#DEBHELPER#
+
+exit 0
@@ -0,0 +1,69 @@
+#!/usr/bin/make -f
+# -*- makefile -*-
+# Sample debian/rules that uses debhelper.
+# This file was originally written by Joey Hess and Craig Small.
+# As a special exception, when this file is copied by dh-make into a
+# dh-make output file, you may use that output file without restriction.
+# This special exception was added by Craig Small in version 0.37 of dh-make.
+
+# Uncomment this to turn on verbose mode.
+export DH_VERBOSE=1
+
+export PYBUILD_DISABLE_python2=1
+
+%:
+	dh $@ --with python3,systemd --buildsystem pybuild
+
+DESTDIR=$(CURDIR)/debian/fail2ban
+PYVERSION=$(shell py3versions -dv)
+
+override_dh_clean:
+	-rm -rf fail2ban.egg-info
+	-rm -f debian/fail2ban.init
+	dh_clean
+	: # auto generated
+	-rm -f bin/fail2ban-python
+
+override_dh_auto_configure:
+	LANG=C ./fail2ban-2to3
+	dh_auto_configure
+
+override_dh_install:
+	rm -f $(DESTDIR)/usr/share/doc/fail2ban/README.Solaris
+	find "$(DESTDIR)/etc/fail2ban/" -type f -name 'paths-*.conf' -not -name 'paths-common.conf' -not -name 'paths-debian.conf' -delete
+	: # Remove explicitly created /var/run/fail2ban
+	: # just to please lintian since init file will
+	: # take care about it anyways
+	rm -rf $(DESTDIR)/var/run/ $(DESTDIR)/run/
+	: # Install monit configuration
+	install -d $(DESTDIR)/etc/monit/conf-available
+	install -d $(DESTDIR)/etc/monit/monitrc.d
+	install -m 644 files/monit/fail2ban $(DESTDIR)/etc/monit/monitrc.d/fail2ban
+	: # Install bash completion
+	install -d $(DESTDIR)/usr/share/bash-completion/completions
+	install -m 644 files/bash-completion $(DESTDIR)/usr/share/bash-completion/completions/fail2ban
+	: # Install systemd files
+	install -d $(DESTDIR)/lib/systemd/system
+	install -d $(DESTDIR)/usr/lib/tmpfiles.d
+	install -m 644 build/fail2ban.service $(DESTDIR)/lib/systemd/system
+	install -m 644 files/fail2ban-tmpfiles.conf $(DESTDIR)/usr/lib/tmpfiles.d
+	install -d $(DESTDIR)/lib/systemd/system
+	: # Install default jail enabler
+	install -m 644 debian/debian-files/jail.d_defaults-debian.conf $(DESTDIR)/etc/fail2ban/jail.d/defaults-debian.conf
+	dh_install
+
+override_dh_auto_test:
+ifeq (,$(filter nocheck,$(DEB_BUILD_OPTIONS)))
+# fails for now (4 tests), accept it for now
+	FAIL2BAN_CONFIG_DIR="$(CURDIR)/config" LC_ALL=C.UTF-8 bin/fail2ban-testcases --verbosity=2 --no-network || true
+endif
+
+override_dh_installexamples:
+	dh_installexamples files/ipmasq-* files/nagios files/cacti
+
+override_dh_installinit:
+	cp -p files/debian-initd debian/fail2ban.init
+	dh_installinit -- defaults 99
+
+override_dh_installman:
+	dh_installman man/*.[15]
@@ -0,0 +1 @@
+3.0 (quilt)
@@ -0,0 +1,4 @@
+Bug-Database: https://github.com/fail2ban/fail2ban/issues
+Bug-Submit: https://github.com/fail2ban/fail2ban/issues/new
+Repository: https://github.com/fail2ban/fail2ban.git
+Repository-Browse: https://github.com/fail2ban/fail2ban
@@ -0,0 +1,6 @@
+# watch control file for uscan
+# Run the "uscan" command to check for upstream updates and more.
+# Site		Directory		Pattern			Version	Script
+version=3
+opts="filenamemangle=s/.*\/(.*)/fail2ban-$1\.tar\.gz/" \
+ https://github.com/fail2ban/fail2ban/tags .*archive/(\d[\d\.]+).tar.gz
@@ -38,31 +38,28 @@ class ActionReader(DefinitionInitConfigReader):

 	_configOpts = {
 		"actionstart": ["string", None],
-		"actionstart_on_demand": ["bool", None],
+		"actionstart_on_demand": ["string", None],
 		"actionstop": ["string", None],
 		"actionflush": ["string", None],
 		"actionreload": ["string", None],
 		"actioncheck": ["string", None],
 		"actionrepair": ["string", None],
-		"actionrepair_on_unban": ["bool", None],
+		"actionrepair_on_unban": ["string", None],
 		"actionban": ["string", None],
+		"actionprolong": ["string", None],
 		"actionreban": ["string", None],
 		"actionunban": ["string", None],
-		"norestored": ["bool", None],
+		"norestored": ["string", None],
 	}

 	def __init__(self, file_, jailName, initOpts, **kwargs):
-		# always supply jail name as name parameter if not specified in options:
-		n = initOpts.get("name")
-		if n is None:
-			initOpts["name"] = n = jailName
 		actname = initOpts.get("actname")
 		if actname is None:
 			actname = file_
-			# ensure we've unique action name per jail:
-			if n != jailName:
-				actname += n[len(jailName):] if n.startswith(jailName) else '-' + n
 			initOpts["actname"] = actname
+		# always supply jail name as name parameter if not specified in options:
+		if initOpts.get("name") is None:
+			initOpts["name"] = jailName
 		self._name = actname
 		DefinitionInitConfigReader.__init__(
 			self, file_, jailName, initOpts, **kwargs)
@@ -83,6 +80,11 @@ class ActionReader(DefinitionInitConfigReader):
 	def convert(self):
 		opts = self.getCombined(
 			ignore=CommandAction._escapedTags | set(('timeout', 'bantime')))
+		# type-convert only after combined (otherwise boolean converting prevents substitution):
+		for o in ('norestored', 'actionstart_on_demand', 'actionrepair_on_unban'):
+			if opts.get(o):
+				opts[o] = self._convert_to_boolean(opts[o])
+		
 		# stream-convert:
 		head = ["set", self._jailName]
 		stream = list()
@@ -180,6 +180,12 @@ class Beautifier:
 					msg = "The jail %s action %s has the following " \
 						"methods:\n" % (inC[1], inC[3])
 					msg += ", ".join(response)
+			elif inC[2] == "banip" and inC[0] == "get":
+				if isinstance(response, list):
+					sep = " " if len(inC) <= 3 else inC[3]
+					if sep == "--with-time":
+						sep = "\n"
+					msg = sep.join(response)
 		except Exception:
 			logSys.warning("Beautifier error. Please report the error")
 			logSys.error("Beautify %r with %r failed", response, self.__inputCmd,
@@ -29,7 +29,7 @@ import re
 import sys
 from ..helpers import getLogger

-if sys.version_info >= (3,): # pragma: 2.x no cover
+if sys.version_info >= (3,2):

 	# SafeConfigParser deprecated from Python 3.2 (renamed to ConfigParser)
 	from configparser import ConfigParser as SafeConfigParser, BasicInterpolation, \
@@ -61,7 +61,7 @@ if sys.version_info >= (3,): # pragma: 2.x no cover
 				return super(BasicInterpolationWithName, self)._interpolate_some(
 					parser, option, accum, rest, section, map, *args, **kwargs)

-else: # pragma: 3.x no cover
+else: # pragma: no cover
 	from ConfigParser import SafeConfigParser, \
 		InterpolationMissingOptionError, NoOptionError, NoSectionError

@@ -372,8 +372,7 @@ after = 1.conf
 						s2 = alls.get(n)
 						if isinstance(s2, dict):
 							# save previous known values, for possible using in local interpolations later:
-							self.merge_section('KNOWN/'+n, 
-								dict(filter(lambda i: i[0] in s, s2.iteritems())), '')
+							self.merge_section('KNOWN/'+n, s2, '')
 							# merge section
 							s2.update(s)
 						else:
@@ -34,30 +34,6 @@ from ..helpers import getLogger, _as_bool, _merge_dicts, substituteRecursiveTags
 # Gets the instance of the logger.
 logSys = getLogger(__name__)

-CONVERTER = {
-	"bool": _as_bool,
-	"int": int,
-}
-def _OptionsTemplateGen(options):
-	"""Iterator over the options template with default options.
-	
-	Each options entry is composed of an array or tuple with:
-		[[type, name, ?default?], ...]
-	Or it is a dict:
-		{name: [type, default], ...}
-	"""
-	if isinstance(options, (list,tuple)):
-		for optname in options:
-			if len(optname) > 2:
-				opttype, optname, optvalue = optname
-			else:
-				(opttype, optname), optvalue = optname, None
-			yield opttype, optname, optvalue
-	else:
-		for optname in options:
-			opttype, optvalue = options[optname]
-			yield opttype, optname, optvalue
-

 class ConfigReader():
 	"""Generic config reader class.
@@ -144,10 +120,6 @@ class ConfigReader():
 		except AttributeError:
 			return False

-	def has_option(self, sec, opt, withDefault=True):
-		return self._cfg.has_option(sec, opt) if withDefault \
-			else opt in self._cfg._sections.get(sec, {})
-
 	def merge_defaults(self, d):
 		self._cfg.get_defaults().update(d)

@@ -252,22 +224,31 @@ class ConfigReaderUnshared(SafeConfigParserWithIncludes):
 	# Or it is a dict:
 	#  {name: [type, default], ...}
 	
-	def getOptions(self, sec, options, pOptions=None, shouldExist=False, convert=True):
+	def getOptions(self, sec, options, pOptions=None, shouldExist=False):
 		values = dict()
 		if pOptions is None:
 			pOptions = {}
 		# Get only specified options:
-		for opttype, optname, optvalue in _OptionsTemplateGen(options):
+		for optname in options:
+			if isinstance(options, (list,tuple)):
+				if len(optname) > 2:
+					opttype, optname, optvalue = optname
+				else:
+					(opttype, optname), optvalue = optname, None
+			else:
+				opttype, optvalue = options[optname]
 			if optname in pOptions:
 				continue
 			try:
-				v = self.get(sec, optname, vars=pOptions)
+				if opttype == "bool":
+					v = self.getboolean(sec, optname)
+					if v is None: continue
+				elif opttype == "int":
+					v = self.getint(sec, optname)
+					if v is None: continue
+				else:
+					v = self.get(sec, optname, vars=pOptions)
 				values[optname] = v
-				if convert:
-					conv = CONVERTER.get(opttype)
-					if conv:
-						if v is None: continue
-						values[optname] = conv(v)
 			except NoSectionError as e:
 				if shouldExist:
 					raise
@@ -280,8 +261,8 @@ class ConfigReaderUnshared(SafeConfigParserWithIncludes):
 					logSys.warning("'%s' not defined in '%s'. Using default one: %r"
 								% (optname, sec, optvalue))
 					values[optname] = optvalue
-				# elif logSys.getEffectiveLevel() <= logLevel:
-				# 	logSys.log(logLevel, "Non essential option '%s' not defined in '%s'.", optname, sec)
+				elif logSys.getEffectiveLevel() <= logLevel:
+					logSys.log(logLevel, "Non essential option '%s' not defined in '%s'.", optname, sec)
 			except ValueError:
 				logSys.warning("Wrong value for '" + optname + "' in '" + sec +
 							"'. Using default one: '" + repr(optvalue) + "'")
@@ -339,9 +320,8 @@ class DefinitionInitConfigReader(ConfigReader):
 			pOpts = dict()
 		if self._initOpts:
 			pOpts = _merge_dicts(pOpts, self._initOpts)
-		# type-convert only in combined (otherwise int/bool converting prevents substitution):
 		self._opts = ConfigReader.getOptions(
-			self, "Definition", self._configOpts, pOpts, convert=False)
+			self, "Definition", self._configOpts, pOpts)
 		self._pOpts = pOpts
 		if self.has_section("Init"):
 			# get only own options (without options from default):
@@ -362,21 +342,10 @@ class DefinitionInitConfigReader(ConfigReader):
 				if opt == '__name__' or opt in self._opts: continue
 				self._opts[opt] = self.get("Definition", opt)

-	def convertOptions(self, opts, configOpts):
-		"""Convert interpolated combined options to expected type.
-		"""
-		for opttype, optname, optvalue in _OptionsTemplateGen(configOpts):
-			conv = CONVERTER.get(opttype)
-			if conv:
-				v = opts.get(optname)
-				if v is None: continue
-				try:
-					opts[optname] = conv(v)
-				except ValueError:
-					logSys.warning("Wrong %s value %r for %r. Using default one: %r",
-						opttype, v, optname, optvalue)
-					opts[optname] = optvalue

+	def _convert_to_boolean(self, value):
+		return _as_bool(value)
+	
 	def getCombOption(self, optname):
 		"""Get combined definition option (as string) using pre-set and init
 		options as preselection (values with higher precedence as specified in section).
@@ -411,8 +380,6 @@ class DefinitionInitConfigReader(ConfigReader):
 			ignore=ignore, addrepl=self.getCombOption)
 		if not opts:
 			raise ValueError('recursive tag definitions unable to be resolved')
-		# convert options after all interpolations:
-		self.convertOptions(opts, self._configOpts)
 		return opts
 	
 	def convert(self):
@@ -48,8 +48,7 @@ class CSocket:
 	def send(self, msg, nonblocking=False, timeout=None):
 		# Convert every list member to string
 		obj = dumps(map(CSocket.convert, msg), HIGHEST_PROTOCOL)
-		self.__csock.send(obj)
-		self.__csock.send(CSPROTO.END)
+		self.__csock.send(obj + CSPROTO.END)
 		return self.receive(self.__csock, nonblocking, timeout)

 	def settimeout(self, timeout):
@@ -82,12 +81,9 @@ class CSocket:
 		msg = CSPROTO.EMPTY
 		if nonblocking: sock.setblocking(0)
 		if timeout: sock.settimeout(timeout)
-		bufsize = 1024
-		while msg.rfind(CSPROTO.END, -32) == -1:
-			chunk = sock.recv(bufsize)
-			if not len(chunk):
-				raise socket.error(104, 'Connection reset by peer')
-			if chunk == CSPROTO.END: break
+		while msg.rfind(CSPROTO.END) == -1:
+			chunk = sock.recv(512)
+			if chunk in ('', b''): # python 3.x may return b'' instead of ''
+				raise RuntimeError("socket connection broken")
 			msg = msg + chunk
-			if bufsize < 32768: bufsize <<= 1
 		return loads(msg)
@@ -168,6 +168,19 @@ class Fail2banClient(Fail2banCmdLine, Thread):
 		if not ret:
 			return None

+		# verify that directory for the socket file exists
+		socket_dir = os.path.dirname(self._conf["socket"])
+		if not os.path.exists(socket_dir):
+			logSys.error(
+				"There is no directory %s to contain the socket file %s."
+				% (socket_dir, self._conf["socket"]))
+			return None
+		if not os.access(socket_dir, os.W_OK | os.X_OK): # pragma: no cover
+			logSys.error(
+				"Directory %s exists but not accessible for writing"
+				% (socket_dir,))
+			return None
+
 		# Check already running
 		if not self._conf["force"] and os.path.exists(self._conf["socket"]):
 			logSys.error("Fail2ban seems to be in unexpected state (not running but the socket exists)")
@@ -27,20 +27,15 @@ import sys

 from ..version import version, normVersion
 from ..protocol import printFormatted
-from ..helpers import getLogger, str2LogLevel, getVerbosityFormat, BrokenPipeError
+from ..helpers import getLogger, str2LogLevel, getVerbosityFormat

 # Gets the instance of the logger.
 logSys = getLogger("fail2ban")

 def output(s): # pragma: no cover
-	try:
-		print(s)
-	except (BrokenPipeError, IOError) as e: # pragma: no cover
-		if e.errno != 32: # closed / broken pipe
-			raise
+	print(s)

-# Config parameters required to start fail2ban which can be also set via command line (overwrite fail2ban.conf),
-CONFIG_PARAMS = ("socket", "pidfile", "logtarget", "loglevel", "syslogsocket")
+CONFIG_PARAMS = ("socket", "pidfile", "logtarget", "loglevel", "syslogsocket",)
 # Used to signal - we are in test cases (ex: prevents change logging params, log capturing, etc)
 PRODUCTION = True

@@ -99,10 +94,9 @@ class Fail2banCmdLine():
 		output("and bans the corresponding IP addresses using firewall rules.")
 		output("")
 		output("Options:")
-		output("    -c, --conf <DIR>        configuration directory")
-		output("    -s, --socket <FILE>     socket path")
-		output("    -p, --pidfile <FILE>    pidfile path")
-		output("    --pname <NAME>          name of the process (main thread) to identify instance (default fail2ban-server)")
+		output("    -c <DIR>                configuration directory")
+		output("    -s <FILE>               socket path")
+		output("    -p <FILE>               pidfile path")
 		output("    --loglevel <LEVEL>      logging level")
 		output("    --logtarget <TARGET>    logging target, use file-name or stdout, stderr, syslog or sysout.")
 		output("    --syslogsocket auto|<FILE>")
@@ -135,15 +129,17 @@ class Fail2banCmdLine():
 		"""
 		for opt in optList:
 			o = opt[0]
-			if o in ("-c", "--conf"):
+			if o == "-c":
 				self._conf["conf"] = opt[1]
-			elif o in ("-s", "--socket"):
+			elif o == "-s":
 				self._conf["socket"] = opt[1]
-			elif o in ("-p", "--pidfile"):
+			elif o == "-p":
 				self._conf["pidfile"] = opt[1]
-			elif o in ("-d", "--dp", "--dump-pretty"):
+			elif o.startswith("--log") or o.startswith("--sys"):
+				self._conf[ o[2:] ] = opt[1]
+			elif o in ["-d", "--dp", "--dump-pretty"]:
 				self._conf["dump"] = True if o == "-d" else 2
-			elif o in ("-t", "--test"):
+			elif o == "-t" or o == "--test":
 				self.cleanConfOnly = True
 				self._conf["test"] = True
 			elif o == "-v":
@@ -167,14 +163,12 @@ class Fail2banCmdLine():
 				from ..server.mytime import MyTime
 				output(MyTime.str2seconds(opt[1]))
 				return True
-			elif o in ("-h", "--help"):
+			elif o in ["-h", "--help"]:
 				self.dispUsage()
 				return True
-			elif o in ("-V", "--version"):
+			elif o in ["-V", "--version"]:
 				self.dispVersion(o == "-V")
 				return True
-			elif o.startswith("--"): # other long named params (see also resetConf)
-				self._conf[ o[2:] ] = opt[1]
 		return None

 	def initCmdLine(self, argv):
@@ -191,7 +185,6 @@ class Fail2banCmdLine():
 			try:
 				cmdOpts = 'hc:s:p:xfbdtviqV'
 				cmdLongOpts = ['loglevel=', 'logtarget=', 'syslogsocket=', 'test', 'async',
-					'conf=', 'pidfile=', 'pname=', 'socket=',
 					'timeout=', 'str2sec=', 'help', 'version', 'dp', '--dump-pretty']
 				optList, self._args = getopt.getopt(self._argv[1:], cmdOpts, cmdLongOpts)
 			except getopt.GetoptError:
@@ -234,8 +227,7 @@ class Fail2banCmdLine():
 					if not conf:
 						self.configurator.readEarly()
 						conf = self.configurator.getEarlyOptions()
-					if o in conf:
-						self._conf[o] = conf[o]
+					self._conf[o] = conf[o]

 			logSys.info("Using socket file %s", self._conf["socket"])

@@ -312,24 +304,18 @@ class Fail2banCmdLine():
 	# since method is also exposed in API via globally bound variable
 	@staticmethod
 	def _exit(code=0):
-		# implicit flush without to produce broken pipe error (32):
-		sys.stderr.close()
-		try:
-			sys.stdout.flush()
-			# exit:
-			if hasattr(sys, 'exit') and sys.exit:
-				sys.exit(code)
-			else:
-				os._exit(code)
-		except (BrokenPipeError, IOError) as e: # pragma: no cover
-			if e.errno != 32: # closed / broken pipe
-				raise
+		if hasattr(os, '_exit') and os._exit:
+			os._exit(code)
+		else:
+			sys.exit(code)

 	@staticmethod
 	def exit(code=0):
 		logSys.debug("Exit with code %s", code)
 		# because of possible buffered output in python, we should flush it before exit:
 		logging.shutdown()
+		sys.stdout.flush()
+		sys.stderr.flush()
 		# exit
 		Fail2banCmdLine._exit(code)

@@ -21,6 +21,7 @@ Fail2Ban  reads log file that contains password failure report
 and bans the corresponding IP addresses using firewall rules.

 This tools can test regular expressions for "fail2ban".
+
 """

 __author__ = "Fail2Ban Developers"
@@ -108,22 +109,19 @@ class _f2bOptParser(OptionParser):
 	def format_help(self, *args, **kwargs):
 		""" Overwritten format helper with full ussage."""
 		self.usage = ''
-		return "Usage: " + usage() + "\n" + __doc__ + """
+		return "Usage: " + usage() + __doc__ + """
 LOG:
-  string                a string representing a log line
-  filename              path to a log file (/var/log/auth.log)
-  systemd-journal       search systemd journal (systemd-python required),
-                        optionally with backend parameters, see `man jail.conf`
-                        for usage and examples (systemd-journal[journalflags=1]).
+    string                  a string representing a log line
+    filename                path to a log file (/var/log/auth.log)
+    "systemd-journal"       search systemd journal (systemd-python required)

 REGEX:
-  string                a string representing a 'failregex'
-  filter                name of filter, optionally with options (sshd[mode=aggressive])
-  filename              path to a filter file (filter.d/sshd.conf)
+    string                  a string representing a 'failregex'
+    filename                path to a filter file (filter.d/sshd.conf)

 IGNOREREGEX:
-  string                a string representing an 'ignoreregex'
-  filename              path to a filter file (filter.d/sshd.conf)
+    string                  a string representing an 'ignoreregex'
+    filename                path to a filter file (filter.d/sshd.conf)
 \n""" + OptionParser.format_help(self, *args, **kwargs) + """\n
 Report bugs to https://github.com/fail2ban/fail2ban/issues\n
 """ + __copyright__ + "\n"
@@ -254,8 +252,6 @@ class Fail2banRegex(object):

 		self.share_config=dict()
 		self._filter = Filter(None)
-		self._prefREMatched = 0
-		self._prefREGroups = list()
 		self._ignoreregex = list()
 		self._failregex = list()
 		self._time_elapsed = None
@@ -276,10 +272,6 @@ class Fail2banRegex(object):
 		self._filter.returnRawHost = opts.raw
 		self._filter.checkFindTime = False
 		self._filter.checkAllRegex = opts.checkAllRegex and not opts.out
-		# ignore pending (without ID/IP), added to matches if it hits later (if ID/IP can be retreved)
-		self._filter.ignorePending = opts.out
-		# callback to increment ignored RE's by index (during process):
-		self._filter.onIgnoreRegex = self._onIgnoreRegex
 		self._backend = 'auto'

 	def output(self, line):
@@ -296,8 +288,8 @@ class Fail2banRegex(object):
 			self._filter.setDatePattern(pattern)
 			self._datepattern_set = True
 			if pattern is not None:
-				self.output( "Use      datepattern : %s : %s" % (
-					pattern, self._filter.getDatePattern()[1], ) )
+				self.output( "Use      datepattern : %s" % (
+					self._filter.getDatePattern()[1], ) )

 	def setMaxLines(self, v):
 		if not self._maxlines_set:
@@ -380,8 +372,11 @@ class Fail2banRegex(object):
 			if not ret:
 				output( "ERROR: failed to load filter %s" % value )
 				return False
-			# set backend-related options (logtype):
-			reader.applyAutoOptions(self._backend)
+			# overwrite default logtype (considering that the filter could specify this too in Definition/Init sections):
+			if not fltOpt.get('logtype'):
+				reader.merge_defaults({
+					'logtype': ['file','journal'][int(self._backend.startswith("systemd"))]
+				})
 			# get, interpolate and convert options:
 			reader.getOptions(None)
 			# show real options if expected:
@@ -441,140 +436,71 @@ class Fail2banRegex(object):
 					'add%sRegex' % regextype.title())(regex.getFailRegex())
 		return True

-	def _onIgnoreRegex(self, idx, ignoreRegex):
-		self._lineIgnored = True
-		self._ignoreregex[idx].inc()
+	def testIgnoreRegex(self, line):
+		found = False
+		try:
+			ret = self._filter.ignoreLine([(line, "", "")])
+			if ret is not None:
+				found = True
+				regex = self._ignoreregex[ret].inc()
+		except RegexException as e: # pragma: no cover
+			output( 'ERROR: %s' % e )
+			return False
+		return found

 	def testRegex(self, line, date=None):
 		orgLineBuffer = self._filter._Filter__lineBuffer
-		# duplicate line buffer (list can be changed inplace during processLine):
-		if self._filter.getMaxLines() > 1:
-			orgLineBuffer = orgLineBuffer[:]
 		fullBuffer = len(orgLineBuffer) >= self._filter.getMaxLines()
-		is_ignored = self._lineIgnored = False
+		is_ignored = False
 		try:
 			found = self._filter.processLine(line, date)
 			lines = []
+			line = self._filter.processedLine()
 			ret = []
 			for match in found:
-				if not self._opts.out:
-					# Append True/False flag depending if line was matched by
-					# more than one regex
-					match.append(len(ret)>1)
-					regex = self._failregex[match[0]]
-					regex.inc()
-					regex.appendIP(match)
+				# Append True/False flag depending if line was matched by
+				# more than one regex
+				match.append(len(ret)>1)
+				regex = self._failregex[match[0]]
+				regex.inc()
+				regex.appendIP(match)
 				if not match[3].get('nofail'):
 					ret.append(match)
 				else:
 					is_ignored = True
-			if self._opts.out: # (formated) output - don't need stats:
-				return None, ret, None
-			# prefregex stats:
-			if self._filter.prefRegex:
-				pre = self._filter.prefRegex
-				if pre.hasMatched():
-					self._prefREMatched += 1
-					if self._verbose:
-						if len(self._prefREGroups) < self._maxlines:
-							self._prefREGroups.append(pre.getGroups())
-						else:
-							if len(self._prefREGroups) == self._maxlines:
-								self._prefREGroups.append('...')
 		except RegexException as e: # pragma: no cover
 			output( 'ERROR: %s' % e )
-			return None, 0, None
-		if self._filter.getMaxLines() > 1:
-			for bufLine in orgLineBuffer[int(fullBuffer):]:
-				if bufLine not in self._filter._Filter__lineBuffer:
-					try:
-						self._line_stats.missed_lines.pop(
-							self._line_stats.missed_lines.index("".join(bufLine)))
-						if self._debuggex:
-							self._line_stats.missed_lines_timeextracted.pop(
-								self._line_stats.missed_lines_timeextracted.index(
-									"".join(bufLine[::2])))
-					except ValueError:
-						pass
-					# if buffering - add also another lines from match:
-					if self._print_all_matched:
-						if not self._debuggex:
-							self._line_stats.matched_lines.append("".join(bufLine))
-						else:
-							lines.append(bufLine[0] + bufLine[2])
-					self._line_stats.matched += 1
-					self._line_stats.missed -= 1
+			return False
+		for bufLine in orgLineBuffer[int(fullBuffer):]:
+			if bufLine not in self._filter._Filter__lineBuffer:
+				try:
+					self._line_stats.missed_lines.pop(
+						self._line_stats.missed_lines.index("".join(bufLine)))
+					if self._debuggex:
+						self._line_stats.missed_lines_timeextracted.pop(
+							self._line_stats.missed_lines_timeextracted.index(
+								"".join(bufLine[::2])))
+				except ValueError:
+					pass
+				# if buffering - add also another lines from match:
+				if self._print_all_matched:
+					if not self._debuggex:
+						self._line_stats.matched_lines.append("".join(bufLine))
+					else:
+						lines.append(bufLine[0] + bufLine[2])
+				self._line_stats.matched += 1
+				self._line_stats.missed -= 1
 		if lines: # pre-lines parsed in multiline mode (buffering)
-			lines.append(self._filter.processedLine())
+			lines.append(line)
 			line = "\n".join(lines)
-		return line, ret, (is_ignored or self._lineIgnored)
-
-	def _prepaireOutput(self):
-		"""Prepares output- and fetch-function corresponding given '--out' option (format)"""
-		ofmt = self._opts.out
-		if ofmt in ('id', 'ip'):
-			def _out(ret):
-				for r in ret:
-					output(r[1])
-		elif ofmt == 'msg':
-			def _out(ret):
-				for r in ret:
-					for r in r[3].get('matches'):
-						if not isinstance(r, basestring):
-							r = ''.join(r for r in r)
-						output(r)
-		elif ofmt == 'row':
-			def _out(ret):
-				for r in ret:
-					output('[%r,\t%r,\t%r],' % (r[1],r[2],dict((k,v) for k, v in r[3].iteritems() if k != 'matches')))
-		elif '<' not in ofmt:
-			def _out(ret):
-				for r in ret:
-					output(r[3].get(ofmt))
-		else: # extended format with tags substitution:
-			from ..server.actions import Actions, CommandAction, BanTicket
-			def _escOut(t, v):
-				# use safe escape (avoid inject on pseudo tag "\x00msg\x00"):
-				if t not in ('msg',):
-					return v.replace('\x00', '\\x00')
-				return v
-			def _out(ret):
-				rows = []
-				wrap = {'NL':0}
-				for r in ret:
-					ticket = BanTicket(r[1], time=r[2], data=r[3])
-					aInfo = Actions.ActionInfo(ticket)
-					# if msg tag is used - output if single line (otherwise let it as is to wrap multilines later):
-					def _get_msg(self):
-						if not wrap['NL'] and len(r[3].get('matches', [])) <= 1:
-							return self['matches']
-						else: # pseudo tag for future replacement:
-							wrap['NL'] = 1
-							return "\x00msg\x00"
-					aInfo['msg'] = _get_msg
-					# not recursive interpolation (use safe escape):
-					v = CommandAction.replaceDynamicTags(ofmt, aInfo, escapeVal=_escOut)
-					if wrap['NL']: # contains multiline tags (msg):
-						rows.append((r, v))
-						continue
-					output(v)
-				# wrap multiline tag (msg) interpolations to single line:
-				for r, v in rows:
-					for r in r[3].get('matches'):
-						if not isinstance(r, basestring):
-							r = ''.join(r for r in r)
-						r = v.replace("\x00msg\x00", r)
-						output(r)
-		return _out
-
+		return line, ret, is_ignored

 	def process(self, test_lines):
 		t0 = time.time()
-		if self._opts.out: # get out function
-			out = self._prepaireOutput()
 		for line in test_lines:
 			if isinstance(line, tuple):
-				line_datetimestripped, ret, is_ignored = self.testRegex(line[0], line[1])
+				line_datetimestripped, ret, is_ignored = self.testRegex(
+					line[0], line[1])
 				line = "".join(line[0])
 			else:
 				line = line.rstrip('\r\n')
@@ -582,10 +508,8 @@ class Fail2banRegex(object):
 					# skip comment and empty lines
 					continue
 				line_datetimestripped, ret, is_ignored = self.testRegex(line)
-
-			if self._opts.out: # (formated) output:
-				if len(ret) > 0 and not is_ignored: out(ret)
-				continue
+			if not is_ignored:
+				is_ignored = self.testIgnoreRegex(line_datetimestripped)

 			if is_ignored:
 				self._line_stats.ignored += 1
@@ -593,25 +517,42 @@ class Fail2banRegex(object):
 					self._line_stats.ignored_lines.append(line)
 					if self._debuggex:
 						self._line_stats.ignored_lines_timeextracted.append(line_datetimestripped)
-			elif len(ret) > 0:
+
+			if len(ret) > 0:
+				assert(not is_ignored)
+				if self._opts.out:
+					if self._opts.out in ('id', 'ip'):
+						for ret in ret:
+							output(ret[1])
+					elif self._opts.out == 'msg':
+						for ret in ret:
+							output('\n'.join(map(lambda v:''.join(v for v in v), ret[3].get('matches'))))
+					elif self._opts.out == 'row':
+						for ret in ret:
+							output('[%r,\t%r,\t%r],' % (ret[1],ret[2],dict((k,v) for k, v in ret[3].iteritems() if k != 'matches')))
+					else:
+						for ret in ret:
+							output(ret[3].get(self._opts.out))
+					continue
 				self._line_stats.matched += 1
 				if self._print_all_matched:
 					self._line_stats.matched_lines.append(line)
 					if self._debuggex:
 						self._line_stats.matched_lines_timeextracted.append(line_datetimestripped)
 			else:
-				self._line_stats.missed += 1
-				if not self._print_no_missed and (self._print_all_missed or self._line_stats.missed <= self._maxlines + 1):
-					self._line_stats.missed_lines.append(line)
-					if self._debuggex:
-						self._line_stats.missed_lines_timeextracted.append(line_datetimestripped)
+				if not is_ignored:
+					self._line_stats.missed += 1
+					if not self._print_no_missed and (self._print_all_missed or self._line_stats.missed <= self._maxlines + 1):
+						self._line_stats.missed_lines.append(line)
+						if self._debuggex:
+							self._line_stats.missed_lines_timeextracted.append(line_datetimestripped)
 			self._line_stats.tested += 1

 		self._time_elapsed = time.time() - t0

 	def printLines(self, ltype):
 		lstats = self._line_stats
-		assert(lstats.missed == lstats.tested - (lstats.matched + lstats.ignored))
+		assert(self._line_stats.missed == lstats.tested - (lstats.matched + lstats.ignored))
 		lines = lstats[ltype]
 		l = lstats[ltype + '_lines']
 		multiline = self._filter.getMaxLines() > 1
@@ -669,18 +610,7 @@ class Fail2banRegex(object):
 			pprint_list(out, " #) [# of hits] regular expression")
 			return total

-		# Print prefregex:
-		if self._filter.prefRegex:
-			#self._filter.prefRegex.hasMatched()
-			pre = self._filter.prefRegex 
-			out = [pre.getRegex()]
-			if self._verbose:
-				for grp in self._prefREGroups:
-					out.append("    %s" % (grp,))
-			output( "\n%s: %d total" % ("Prefregex", self._prefREMatched) )
-			pprint_list(out)
-
-		# Print regex's:
+		# Print title
 		total = print_failregexes("Failregex", self._failregex)
 		_ = print_failregexes("Ignoreregex", self._ignoreregex)

@@ -759,10 +689,10 @@ class Fail2banRegex(object):
 			test_lines = journal_lines_gen(flt, myjournal)
 		else:
 			# if single line parsing (without buffering)
-			if self._filter.getMaxLines() <= 1 and '\n' not in cmd_log:
+			if self._filter.getMaxLines() <= 1:
 				self.output( "Use      single line : %s" % shortstr(cmd_log.replace("\n", r"\n")) )
 				test_lines = [ cmd_log ]
-			else: # multi line parsing (with and without buffering)
+			else: # multi line parsing (with buffering)
 				test_lines = cmd_log.split("\n")
 				self.output( "Use      multi line : %s line(s)" % len(test_lines) )
 				for i, l in enumerate(test_lines):
@@ -782,7 +712,6 @@ class Fail2banRegex(object):


 def exec_command_line(*args):
-	logging.exitOnIOError = True
 	parser = get_opt_parser()
 	(opts, args) = parser.parse_args(*args)
 	errors = []
@@ -53,14 +53,6 @@ class FilterReader(DefinitionInitConfigReader):
 	def getFile(self):
 		return self.__file

-	def applyAutoOptions(self, backend):
-		# set init option to backend-related logtype, considering
-		# that the filter settings may be overwritten in its local:
-		if (not self._initOpts.get('logtype') and 
-		    not self.has_option('Definition', 'logtype', False)
-		  ):
-			self._initOpts['logtype'] = ['file','journal'][int(backend.startswith("systemd"))]
-
 	def convert(self):
 		stream = list()
 		opts = self.getCombined()
@@ -98,6 +98,13 @@ class JailReader(ConfigReader):
 		"maxmatches": ["int", None],
 		"findtime": ["string", None],
 		"bantime": ["string", None],
+		"bantime.increment": ["bool", None],
+		"bantime.factor": ["string", None],
+		"bantime.formula": ["string", None],
+		"bantime.multipliers": ["string", None],
+		"bantime.maxtime": ["string", None],
+		"bantime.rndtime": ["string", None],
+		"bantime.overalljails": ["bool", None],
 		"ignorecommand": ["string", None],
 		"ignoreself": ["bool", None],
 		"ignoreip": ["string", None],
@@ -142,8 +149,11 @@ class JailReader(ConfigReader):
 				ret = self.__filter.read()
 				if not ret:
 					raise JailDefError("Unable to read the filter %r" % filterName)
-				# set backend-related options (logtype):
-				self.__filter.applyAutoOptions(self.__opts.get('backend', ''))
+				if not filterOpt.get('logtype'):
+					# overwrite default logtype backend-related (considering that the filter settings may be overwritten):
+					self.__filter.merge_defaults({
+						'logtype': ['file','journal'][int(self.__opts.get('backend', '').startswith("systemd"))]
+					})
 				# merge options from filter as 'known/...' (all options unfiltered):
 				self.__filter.getOptions(self.__opts, all=True)
 				ConfigReader.merge_section(self, self.__name, self.__filter.getCombined(), 'known/')
@@ -208,26 +208,6 @@ class FormatterWithTraceBack(logging.Formatter):
 		return logging.Formatter.format(self, record)


-logging.exitOnIOError = False
-def __stopOnIOError(logSys=None, logHndlr=None): # pragma: no cover
-	if logSys and len(logSys.handlers):
-		logSys.removeHandler(logSys.handlers[0])
-	if logHndlr:
-		logHndlr.close = lambda: None
-	logging.StreamHandler.flush = lambda self: None
-	#sys.excepthook = lambda *args: None
-	if logging.exitOnIOError:
-		try:
-			sys.stderr.close()
-		except:
-			pass
-		sys.exit(0)
-
-try:
-	BrokenPipeError = BrokenPipeError
-except NameError: # pragma: 3.x no cover
-	BrokenPipeError = IOError
-
 __origLog = logging.Logger._log
 def __safeLog(self, level, msg, args, **kwargs):
 	"""Safe log inject to avoid possible errors by unsafe log-handlers, 
@@ -243,10 +223,6 @@ def __safeLog(self, level, msg, args, **kwargs):
 	try:
 		# if isEnabledFor(level) already called...
 		__origLog(self, level, msg, args, **kwargs)
-	except (BrokenPipeError, IOError) as e: # pragma: no cover
-		if e.errno == 32: # closed / broken pipe
-			__stopOnIOError(self)
-		raise
 	except Exception as e: # pragma: no cover - unreachable if log-handler safe in this python-version
 		try:
 			for args in (
@@ -261,18 +237,6 @@ def __safeLog(self, level, msg, args, **kwargs):
 			pass
 logging.Logger._log = __safeLog

-__origLogFlush = logging.StreamHandler.flush
-def __safeLogFlush(self):
-	"""Safe flush inject stopping endless logging on closed streams (redirected pipe).
-	"""
-	try:
-		__origLogFlush(self)
-	except (BrokenPipeError, IOError) as e: # pragma: no cover
-		if e.errno == 32: # closed / broken pipe
-			__stopOnIOError(None, self)
-		raise
-logging.StreamHandler.flush = __safeLogFlush
-
 def getLogger(name):
 	"""Get logging.Logger instance with Fail2Ban logger name convention
 	"""
@@ -303,7 +267,7 @@ def getVerbosityFormat(verbosity, fmt=' %(message)s', addtime=True, padding=True
 			if addtime:
 				fmt = ' %(asctime)-15s' + fmt
 	else: # default (not verbose):
-		fmt = "%(name)-24s[%(process)d]: %(levelname)-7s" + fmt
+		fmt = "%(name)-23.23s [%(process)d]: %(levelname)-7s" + fmt
 		if addtime:
 			fmt = "%(asctime)s " + fmt
 	# remove padding if not needed:
@@ -327,7 +291,7 @@ def splitwords(s):
 	"""
 	if not s:
 		return []
-	return filter(bool, map(lambda v: v.strip(), re.split('[ ,\n]+', s)))
+	return filter(bool, map(str.strip, re.split('[ ,\n]+', s)))

 if sys.version_info >= (3,5):
 	eval(compile(r'''if 1:
@@ -374,7 +338,7 @@ OPTION_EXTRACT_CRE = re.compile(
 	r'([\w\-_\.]+)=(?:"([^"]*)"|\'([^\']*)\'|([^,\]]*))(?:,|\]\s*\[|$)', re.DOTALL)
 # split by new-line considering possible new-lines within options [...]:
 OPTION_SPLIT_CRE = re.compile(
-	r'(?:[^\[\s]+(?:\s*\[\s*(?:[\w\-_\.]+=(?:"[^"]*"|\'[^\']*\'|[^,\]]*)\s*(?:,|\]\s*\[)?\s*)*\])?\s*|\S+)(?=\n\s*|\s+|$)', re.DOTALL)
+	r'(?:[^\[\n]+(?:\s*\[\s*(?:[\w\-_\.]+=(?:"[^"]*"|\'[^\']*\'|[^,\]]*)\s*(?:,|\]\s*\[)?\s*)*\])?\s*|[^\n]+)(?=\n\s*|$)', re.DOTALL)

 def extractOptions(option):
 	match = OPTION_CRE.match(option)
@@ -399,8 +363,8 @@ def splitWithOptions(option):
 # tags (<tag>) in tagged options.
 #

-# max tag replacement count (considering tag X in tag Y repeat):
-MAX_TAG_REPLACE_COUNT = 25
+# max tag replacement count:
+MAX_TAG_REPLACE_COUNT = 10

 # compiled RE for tag name (replacement name) 
 TAG_CRE = re.compile(r'<([^ <>]+)>')
@@ -434,7 +398,6 @@ def substituteRecursiveTags(inptags, conditional='',
 	done = set()
 	noRecRepl = hasattr(tags, "getRawItem")
 	# repeat substitution while embedded-recursive (repFlag is True)
-	repCounts = {}
 	while True:
 		repFlag = False
 		# substitute each value:
@@ -446,7 +409,7 @@ def substituteRecursiveTags(inptags, conditional='',
 			value = orgval = uni_string(tags[tag])
 			# search and replace all tags within value, that can be interpolated using other tags:
 			m = tre_search(value)
-			rplc = repCounts.get(tag, {})
+			refCounts = {}
 			#logSys.log(5, 'TAG: %s, value: %s' % (tag, value))
 			while m:
 				# found replacement tag:
@@ -456,13 +419,13 @@ def substituteRecursiveTags(inptags, conditional='',
 					m = tre_search(value, m.end())
 					continue
 				#logSys.log(5, 'found: %s' % rtag)
-				if rtag == tag or rplc.get(rtag, 1) > MAX_TAG_REPLACE_COUNT:
+				if rtag == tag or refCounts.get(rtag, 1) > MAX_TAG_REPLACE_COUNT:
 					# recursive definitions are bad
 					#logSys.log(5, 'recursion fail tag: %s value: %s' % (tag, value) )
 					raise ValueError(
 						"properties contain self referencing definitions "
 						"and cannot be resolved, fail tag: %s, found: %s in %s, value: %s" % 
-						(tag, rtag, rplc, value))
+						(tag, rtag, refCounts, value))
 				repl = None
 				if conditional:
 					repl = tags.get(rtag + '?' + conditional)
@@ -482,7 +445,7 @@ def substituteRecursiveTags(inptags, conditional='',
 				value = value.replace('<%s>' % rtag, repl)
 				#logSys.log(5, 'value now: %s' % value)
 				# increment reference count:
-				rplc[rtag] = rplc.get(rtag, 0) + 1
+				refCounts[rtag] = refCounts.get(rtag, 0) + 1
 				# the next match for replace:
 				m = tre_search(value, m.start())
 			#logSys.log(5, 'TAG: %s, newvalue: %s' % (tag, value))
@@ -490,7 +453,6 @@ def substituteRecursiveTags(inptags, conditional='',
 			if orgval != value:
 				# check still contains any tag - should be repeated (possible embedded-recursive substitution):
 				if tre_search(value):
-					repCounts[tag] = rplc
 					repFlag = True
 				# copy return tags dict to prevent modifying of inptags:
 				if id(tags) == id(inptags):
@@ -55,8 +55,6 @@ protocol = [
 ["stop", "stops all jails and terminate the server"], 
 ["unban --all", "unbans all IP addresses (in all jails and database)"],
 ["unban <IP> ... <IP>", "unbans <IP> (in all jails and database)"],
-["banned", "return jails with banned IPs as dictionary"],
-["banned <IP> ... <IP>]", "return list(s) of jails where given IP(s) are banned"],
 ["status", "gets the current status of the server"], 
 ["ping", "tests if the server is alive"],
 ["echo", "for internal usage, returns back and outputs a given string"],
@@ -122,8 +120,6 @@ protocol = [
 ["set <JAIL> action <ACT> <PROPERTY> <VALUE>", "sets the <VALUE> of <PROPERTY> for the action <ACT> for <JAIL>"],
 ["set <JAIL> action <ACT> <METHOD>[ <JSONKWARGS>]", "calls the <METHOD> with <JSONKWARGS> for the action <ACT> for <JAIL>"],
 ['', "JAIL INFORMATION", ""],
-["get <JAIL> banned", "return banned IPs of <JAIL>"],
-["get <JAIL> banned <IP> ... <IP>]", "return 1 if IP is banned in <JAIL> otherwise 0, or a list of 1/0 for multiple IPs"],
 ["get <JAIL> logpath", "gets the list of the monitored files for <JAIL>"],
 ["get <JAIL> logencoding", "gets the encoding of the log files for <JAIL>"],
 ["get <JAIL> journalmatch", "gets the journal filter match for <JAIL>"],
@@ -136,6 +132,7 @@ protocol = [
 ["get <JAIL> bantime", "gets the time a host is banned for <JAIL>"],
 ["get <JAIL> datepattern", "gets the patern used to match date/times for <JAIL>"],
 ["get <JAIL> usedns", "gets the usedns setting for <JAIL>"],
+["get <JAIL> banip [<SEP>|--with-time]", "gets the list of of banned IP addresses for <JAIL>. Optionally the separator character ('<SEP>', default is space) or the option '--with-time' (printing the times of ban) may be specified. The IPs are ordered by end of ban."],
 ["get <JAIL> maxretry", "gets the number of failures allowed for <JAIL>"],
 ["get <JAIL> maxmatches", "gets the max number of matches stored in memory per ticket in <JAIL>"], 
 ["get <JAIL> maxlines", "gets the number of lines to buffer for <JAIL>"],
@@ -262,6 +262,10 @@ class ActionBase(object):
 		"""
 		return self.ban(aInfo)

+	@property
+	def _prolongable(self): # pragma: no cover - abstract
+		return False
+
 	def unban(self, aInfo): # pragma: no cover - abstract
 		"""Executed when a ban expires.

@@ -274,6 +278,11 @@ class ActionBase(object):
 		pass


+WRAP_CMD_PARAMS = {
+	'timeout': 'str2seconds',
+	'bantime': 'ignore',
+}
+
 class CommandAction(ActionBase):
 	"""A action which executes OS shell commands.

@@ -346,7 +355,10 @@ class CommandAction(ActionBase):
 	def __setattr__(self, name, value):
 		if not name.startswith('_') and not self.__init and not callable(value):
 			# special case for some parameters:
-			if name in ('timeout', 'bantime'):
+			wrp = WRAP_CMD_PARAMS.get(name)
+			if wrp == 'ignore': # ignore (filter) dynamic parameters
+				return
+			elif wrp == 'str2seconds':
 				value = MyTime.str2seconds(value)
 			# parameters changed - clear properties and substitution cache:
 			self.__properties = None
@@ -392,13 +404,10 @@ class CommandAction(ActionBase):
 	def _getOperation(self, tag, family):
 		# replace operation tag (interpolate all values), be sure family is enclosed as conditional value
 		# (as lambda in addrepl so only if not overwritten in action):
-		cmd = self.replaceTag(tag, self._properties,
+		return self.replaceTag(tag, self._properties,
 			conditional=('family='+family if family else ''),
+			addrepl=(lambda tag:family if tag == 'family' else None),
 			cache=self.__substCache)
-		if '<' not in cmd or not family: return cmd
-		# replace family as dynamic tags, important - don't cache, no recursion and auto-escape here:
-		cmd = self.replaceDynamicTags(cmd, {'family':family})
-		return cmd

 	def _operationExecuted(self, tag, family, *args):
 		""" Get, set or delete command of operation considering family.
@@ -543,6 +552,26 @@ class CommandAction(ActionBase):
 			raise RuntimeError("Error banning %(ip)s" % aInfo)
 		self.__started[family] = self.__started.get(family, 0) | 3; # started and contains items

+	@property
+	def _prolongable(self):
+		return (hasattr(self, 'actionprolong') and self.actionprolong 
+			and not str(self.actionprolong).isspace())
+	
+	def prolong(self, aInfo):
+		"""Executes the "actionprolong" command.
+
+		Replaces the tags in the action command with actions properties
+		and ban information, and executes the resulting command.
+
+		Parameters
+		----------
+		aInfo : dict
+			Dictionary which includes information in relation to
+			the ban.
+		"""
+		if not self._processCmd('<actionprolong>', aInfo):
+			raise RuntimeError("Error prolonging %(ip)s" % aInfo)
+
 	def unban(self, aInfo):
 		"""Executes the "actionunban" command.

@@ -652,8 +681,10 @@ class CommandAction(ActionBase):
 					ret &= False
 		return ret

-	@staticmethod
-	def escapeTag(value):
+	ESCAPE_CRE = re.compile(r"""[\\#&;`|*?~<>^()\[\]{}$'"\n\r]""")
+	
+	@classmethod
+	def escapeTag(cls, value):
 		"""Escape characters which may be used for command injection.

 		Parameters
@@ -670,12 +701,15 @@ class CommandAction(ActionBase):
 		-----
 		The following characters are escaped::

-			\\#&;`|*?~<>^()[]{}$'"
+			\\#&;`|*?~<>^()[]{}$'"\n\r

 		"""
-		for c in '\\#&;`|*?~<>^()[]{}$\'"':
-			if c in value:
-				value = value.replace(c, '\\' + c)
+		_map2c = {'\n': 'n', '\r': 'r'}
+		def substChar(m):
+			c = m.group()
+			return '\\' + _map2c.get(c, c)
+		
+		value = cls.ESCAPE_CRE.sub(substChar, value)
 		return value

 	@classmethod
@@ -772,7 +806,7 @@ class CommandAction(ActionBase):
 	ESCAPE_VN_CRE = re.compile(r"\W")

 	@classmethod
-	def replaceDynamicTags(cls, realCmd, aInfo, escapeVal=None):
+	def replaceDynamicTags(cls, realCmd, aInfo):
 		"""Replaces dynamical tags in `query` with property values.

 		**Important**
@@ -797,17 +831,16 @@ class CommandAction(ActionBase):
 		# array for escaped vars:
 		varsDict = dict()

-		if not escapeVal:
-			def escapeVal(tag, value):
-				# if the value should be escaped:
-				if cls.ESCAPE_CRE.search(value):
-					# That one needs to be escaped since its content is
-					# out of our control
-					tag = 'f2bV_%s' % cls.ESCAPE_VN_CRE.sub('_', tag)
-					varsDict[tag] = value # add variable
-					value = '$'+tag	# replacement as variable
-				# replacement for tag:
-				return value
+		def escapeVal(tag, value):
+			# if the value should be escaped:
+			if cls.ESCAPE_CRE.search(value):
+				# That one needs to be escaped since its content is
+				# out of our control
+				tag = 'f2bV_%s' % cls.ESCAPE_VN_CRE.sub('_', tag)
+				varsDict[tag] = value # add variable
+				value = '$'+tag	# replacement as variable
+			# replacement for tag:
+			return value

 		# additional replacement as calling map:
 		ADD_REPL_TAGS_CM = CallingMap(ADD_REPL_TAGS)
@@ -975,7 +1008,8 @@ class CommandAction(ActionBase):
 		RuntimeError
 			If command execution times out.
 		"""
-		logSys.debug(realCmd)
+		if logSys.getEffectiveLevel() < logging.DEBUG:
+			logSys.log(9, realCmd)
 		if not realCmd:
 			logSys.debug("Nothing to do")
 			return True
@@ -39,6 +39,7 @@ from .ipdns import IPAddr
 from .jailthread import JailThread
 from .action import ActionBase, CommandAction, CallingMap
 from .mytime import MyTime
+from .observer import Observers
 from .utils import Utils
 from ..helpers import getLogger

@@ -210,13 +211,15 @@ class Actions(JailThread, Mapping):
 	def getBanTime(self):
 		return self.__banManager.getBanTime()

-	def getBanned(self, ids):
-		lst = self.__banManager.getBanList()
-		if not ids:
-			return lst
-		if len(ids) == 1:
-			return 1 if ids[0] in lst else 0
-		return map(lambda ip: 1 if ip in lst else 0, ids)
+	def getBanList(self, withTime=False):
+		"""Returns the list of banned IP addresses.
+
+		Returns
+		-------
+		list
+			The list of banned IP addresses.
+		"""
+		return self.__banManager.getBanList(ordered=True, withTime=withTime)

 	def addBannedIP(self, ip):
 		"""Ban an IP or list of IPs."""
@@ -251,7 +254,7 @@ class Actions(JailThread, Mapping):
 		if ip is None:
 			return self.__flushBan(db)
 		# Multiple IPs:
-		if isinstance(ip, (list, tuple)):
+		if isinstance(ip, list):
 			missed = []
 			cnt = 0
 			for i in ip:
@@ -273,14 +276,6 @@ class Actions(JailThread, Mapping):
 			# Unban the IP.
 			self.__unBan(ticket)
 		else:
-			# Multiple IPs by subnet or dns:
-			if not isinstance(ip, IPAddr):
-				ipa = IPAddr(ip)
-				if not ipa.isSingle: # subnet (mask/cidr) or raw (may be dns/hostname):
-					ips = filter(ipa.contains, self.__banManager.getBanList())
-					if ips:
-						return self.removeBannedIP(ips, db, ifexists)
-			# not found:
 			msg = "%s is not banned" % ip
 			logSys.log(logging.MSG, msg)
 			if ifexists:
@@ -327,33 +322,23 @@ class Actions(JailThread, Mapping):
 					self._jail.name, name, e,
 					exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
 		while self.active:
-			try:
-				if self.idle:
-					logSys.debug("Actions: enter idle mode")
-					Utils.wait_for(lambda: not self.active or not self.idle,
-						lambda: False, self.sleeptime)
-					logSys.debug("Actions: leave idle mode")
-					continue
-				# wait for ban (stop if gets inactive, pending ban or unban):
-				bancnt = 0
-				wt = min(self.sleeptime, self.__banManager._nextUnbanTime - MyTime.time())
-				logSys.log(5, "Actions: wait for pending tickets %s (default %s)", wt, self.sleeptime)
-				if Utils.wait_for(lambda: not self.active or self._jail.hasFailTickets, wt):
-					bancnt = self.__checkBan()
-					cnt += bancnt
-				# unban if nothing is banned not later than banned tickets >= banPrecedence
-				if not bancnt or cnt >= self.banPrecedence:
-					if self.active:
-						# let shrink the ban list faster
-						bancnt *= 2
-						logSys.log(5, "Actions: check-unban %s, bancnt %s, max: %s", bancnt if bancnt and bancnt < self.unbanMaxCount else self.unbanMaxCount, bancnt, self.unbanMaxCount)
-						self.__checkUnBan(bancnt if bancnt and bancnt < self.unbanMaxCount else self.unbanMaxCount)
-					cnt = 0
-			except Exception as e: # pragma: no cover
-				logSys.error("[%s] unhandled error in actions thread: %s",
-					self._jail.name, e,
-					exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
-
+			if self.idle:
+				logSys.debug("Actions: enter idle mode")
+				Utils.wait_for(lambda: not self.active or not self.idle,
+					lambda: False, self.sleeptime)
+				logSys.debug("Actions: leave idle mode")
+				continue
+			# wait for ban (stop if gets inactive):
+			bancnt = Utils.wait_for(lambda: not self.active or self.__checkBan(), self.sleeptime)
+			cnt += bancnt
+			# unban if nothing is banned not later than banned tickets >= banPrecedence
+			if not bancnt or cnt >= self.banPrecedence:
+				if self.active:
+					# let shrink the ban list faster
+					bancnt *= 2
+					self.__checkUnBan(bancnt if bancnt and bancnt < self.unbanMaxCount else self.unbanMaxCount)
+				cnt = 0
+		
 		self.__flushBan(stop=True)
 		self.stopActions()
 		return True
@@ -370,6 +355,8 @@ class Actions(JailThread, Mapping):
 			"fid":			lambda self: self.__ticket.getID(),
 			"failures":	lambda self: self.__ticket.getAttempt(),
 			"time":			lambda self: self.__ticket.getTime(),
+			"bantime":  lambda self: self._getBanTime(),
+			"bancount":  lambda self: self.__ticket.getBanCount(),
 			"matches":	lambda self: "\n".join(self.__ticket.getMatches()),
 			# to bypass actions, that should not be executed for restored tickets
 			"restored":	lambda self: (1 if self.__ticket.restored else 0),
@@ -396,6 +383,11 @@ class Actions(JailThread, Mapping):
 		def copy(self): # pragma: no cover
 			return self.__class__(self.__ticket, self.__jail, self.immutable, self.data.copy())

+		def _getBanTime(self):
+			btime = self.__ticket.getBanTime()
+			if btime is None: btime = self.__jail.actions.getBanTime()
+			return int(btime)
+
 		def _mi4ip(self, overalljails=False):
 			"""Gets bans merged once, a helper for lambda(s), prevents stop of executing action by any exception inside.

@@ -469,16 +461,22 @@ class Actions(JailThread, Mapping):
 			tickets = self.__getFailTickets(self.banPrecedence)
 		rebanacts = None
 		for ticket in tickets:
-			bTicket = BanManager.createBanTicket(ticket)
+
+			bTicket = BanTicket.wrap(ticket)
+			btime = ticket.getBanTime(self.__banManager.getBanTime())
 			ip = bTicket.getIP()
 			aInfo = self.__getActionInfo(bTicket)
 			reason = {}
 			if self.__banManager.addBanTicket(bTicket, reason=reason):
 				cnt += 1
+				# report ticket to observer, to check time should be increased and hereafter observer writes ban to database (asynchronous)
+				if Observers.Main is not None and not bTicket.restored:
+					Observers.Main.add('banFound', bTicket, self._jail, btime)
 				logSys.notice("[%s] %sBan %s", self._jail.name, ('' if not bTicket.restored else 'Restore '), ip)
+				# do actions :
 				for name, action in self._actions.iteritems():
 					try:
-						if bTicket.restored and getattr(action, 'norestored', False):
+						if ticket.restored and getattr(action, 'norestored', False):
 							continue
 						if not aInfo.immutable: aInfo.reset()
 						action.ban(aInfo)
@@ -493,7 +491,10 @@ class Actions(JailThread, Mapping):
 				if self.banEpoch: # be sure tickets always have the same ban epoch (default 0):
 					bTicket.banEpoch = self.banEpoch
 			else:
-				bTicket = reason['ticket']
+				if reason.get('expired', 0):
+					logSys.info('[%s] Ignore %s, expired bantime', self._jail.name, ip)
+					continue
+				bTicket = reason.get('ticket', bTicket)
 				# if already banned (otherwise still process some action)
 				if bTicket.banned:
 					# compare time of failure occurrence with time ticket was really banned:
@@ -521,12 +522,6 @@ class Actions(JailThread, Mapping):
 						cnt += self.__reBan(bTicket, actions=rebanacts)
 				else: # pragma: no cover - unexpected: ticket is not banned for some reasons - reban using all actions:
 					cnt += self.__reBan(bTicket)
-			# add ban to database:
-			if not bTicket.restored and self._jail.database is not None:
-				# ignore too old (repeated and ignored) tickets,
-				# [todo] replace it with inOperation later (once it gets back-ported):
-				if not reason and bTicket.getTime() >= MyTime.time() - 60:
-					self._jail.database.addBan(self._jail, bTicket)
 		if cnt:
 			logSys.debug("Banned %s / %s, %s ticket(s) in %r", cnt, 
 				self.__banManager.getBanTotal(), self.__banManager.size(), self._jail.name)
@@ -566,6 +561,29 @@ class Actions(JailThread, Mapping):
 			ticket.banEpoch = self.banEpoch
 		return 1

+	def _prolongBan(self, ticket):
+		# prevent to prolong ticket that was removed in-between,
+		# if it in ban list - ban time already prolonged (and it stays there):
+		if not self.__banManager._inBanList(ticket): return
+		# do actions :
+		aInfo = None
+		for name, action in self._actions.iteritems():
+			try:
+				if ticket.restored and getattr(action, 'norestored', False):
+					continue
+				if not action._prolongable:
+					continue
+				if aInfo is None:
+					aInfo = self.__getActionInfo(ticket)
+				if not aInfo.immutable: aInfo.reset()
+				action.prolong(aInfo)
+			except Exception as e:
+				logSys.error(
+					"Failed to execute ban jail '%s' action '%s' "
+					"info '%r': %s",
+					self._jail.name, name, aInfo, e,
+					exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
+
 	def __checkUnBan(self, maxCount=None):
 		"""Check for IP address to unban.

@@ -669,19 +687,13 @@ class Actions(JailThread, Mapping):
 		"""Status of current and total ban counts and current banned IP list.
 		"""
 		# TODO: Allow this list to be printed as 'status' output
-		supported_flavors = ["short", "basic", "cymru"]
+		supported_flavors = ["basic", "cymru"]
 		if flavor is None or flavor not in supported_flavors:
 			logSys.warning("Unsupported extended jail status flavor %r. Supported: %s" % (flavor, supported_flavors))
 		# Always print this information (basic)
-		if flavor != "short":
-			banned = self.__banManager.getBanList()
-			cnt = len(banned)
-		else:
-			cnt = self.__banManager.size()
-		ret = [("Currently banned", cnt),
-			   ("Total banned", self.__banManager.getBanTotal())]
-		if flavor != "short":
-			ret += [("Banned IP list", banned)]
+		ret = [("Currently banned", self.__banManager.size()),
+			   ("Total banned", self.__banManager.getBanTotal()),
+			   ("Banned IP list", self.__banManager.getBanList())]
 		if flavor == "cymru":
 			cymru_info = self.__banManager.getBanListExtendedCymruInfo()
 			ret += \
@@ -57,7 +57,7 @@ class BanManager:
 		## Total number of banned IP address
 		self.__banTotal = 0
 		## The time for next unban process (for performance and load reasons):
-		self._nextUnbanTime = BanTicket.MAX_TIME
+		self.__nextUnbanTime = BanTicket.MAX_TIME
 	
 	##
 	# Set the ban time.
@@ -66,7 +66,8 @@ class BanManager:
 	# @param value the time
 	
 	def setBanTime(self, value):
-		self.__banTime = int(value)
+		with self.__lock:
+			self.__banTime = int(value)
 	
 	##
 	# Get the ban time.
@@ -75,7 +76,8 @@ class BanManager:
 	# @return the time
 	
 	def getBanTime(self):
-		return self.__banTime
+		with self.__lock:
+			return self.__banTime
 	
 	##
 	# Set the total number of banned address.
@@ -83,7 +85,8 @@ class BanManager:
 	# @param value total number
 	
 	def setBanTotal(self, value):
-		self.__banTotal = value
+		with self.__lock:
+			self.__banTotal = value
 	
 	##
 	# Get the total number of banned address.
@@ -91,15 +94,30 @@ class BanManager:
 	# @return the total number
 	
 	def getBanTotal(self):
-		return self.__banTotal
+		with self.__lock:
+			return self.__banTotal

 	##
 	# Returns a copy of the IP list.
 	#
 	# @return IP list
 	
-	def getBanList(self):
-		return list(self.__banList.keys())
+	def getBanList(self, ordered=False, withTime=False):
+		with self.__lock:
+			if not ordered:
+				return self.__banList.keys()
+			lst = []
+			for ticket in self.__banList.itervalues():
+				eob = ticket.getEndOfBanTime(self.__banTime)
+				lst.append((ticket,eob))
+			lst.sort(key=lambda t: t[1])
+			t2s = MyTime.time2str
+			if withTime:
+				return ['%s \t%s + %d = %s' % (
+						t[0].getID(), 
+						t2s(t[0].getTime()), t[0].getBanTime(self.__banTime), t2s(t[1])
+					) for t in lst]
+			return [t[0].getID() for t in lst]

 	##
 	# Returns a iterator to ban list (used in reload, so idle).
@@ -107,8 +125,8 @@ class BanManager:
 	# @return ban list iterator
 	
 	def __iter__(self):
-		# ensure iterator is safe - traverse over the list in snapshot created within lock (GIL):
-		return iter(list(self.__banList.values()))
+		with self.__lock:
+			return self.__banList.itervalues()

 	##
 	# Returns normalized value
@@ -244,21 +262,6 @@ class BanManager:
 			logSys.exception(e)
 			return []

-	##
-	# Create a ban ticket.
-	#
-	# Create a BanTicket from a FailTicket. The timestamp of the BanTicket
-	# is the current time. This is a static method.
-	# @param ticket the FailTicket
-	# @return a BanTicket
-	
-	@staticmethod
-	def createBanTicket(ticket):
-		# we should always use correct time to calculate correct end time (ban time is variable now, 
-		# + possible double banning by restore from database and from log file)
-		# so use as lastTime always time from ticket.
-		return BanTicket(ticket=ticket)
-	
 	##
 	# Add a ban ticket.
 	#
@@ -268,6 +271,9 @@ class BanManager:
 	
 	def addBanTicket(self, ticket, reason={}):
 		eob = ticket.getEndOfBanTime(self.__banTime)
+		if eob < MyTime.time():
+			reason['expired'] = 1
+			return False
 		with self.__lock:
 			# check already banned
 			fid = ticket.getID()
@@ -289,9 +295,10 @@ class BanManager:
 			# not yet banned - add new one:
 			self.__banList[fid] = ticket
 			self.__banTotal += 1
+			ticket.incrBanCount()
 			# correct next unban time:
-			if self._nextUnbanTime > eob:
-				self._nextUnbanTime = eob
+			if self.__nextUnbanTime > eob:
+				self.__nextUnbanTime = eob
 			return True

 	##
@@ -322,8 +329,12 @@ class BanManager:
 	
 	def unBanList(self, time, maxCount=0x7fffffff):
 		with self.__lock:
+			# Permanent banning
+			if self.__banTime < 0:
+				return list()
+
 			# Check next unban time:
-			nextUnbanTime = self._nextUnbanTime
+			nextUnbanTime = self.__nextUnbanTime
 			if nextUnbanTime > time:
 				return list()

@@ -336,12 +347,12 @@ class BanManager:
 				if time > eob:
 					unBanList[fid] = ticket
 					if len(unBanList) >= maxCount: # stop search cycle, so reset back the next check time
-						nextUnbanTime = self._nextUnbanTime
+						nextUnbanTime = self.__nextUnbanTime
 						break
 				elif nextUnbanTime > eob:
 					nextUnbanTime = eob

-			self._nextUnbanTime = nextUnbanTime
+			self.__nextUnbanTime = nextUnbanTime
 			# Removes tickets.
 			if len(unBanList):
 				if len(unBanList) / 2.0 <= len(self.__banList) / 3.0:
@@ -138,7 +138,7 @@ class Fail2BanDb(object):
 	filename
 	purgeage
 	"""
-	__version__ = 2
+	__version__ = 4
 	# Note all SCRIPTS strings must end in ';' for py26 compatibility
 	_CREATE_SCRIPTS = (
 		 ('fail2banDb', "CREATE TABLE IF NOT EXISTS fail2banDb(version INTEGER);")
@@ -166,21 +166,36 @@ class Fail2BanDb(object):
 			"jail TEXT NOT NULL, " \
 			"ip TEXT, " \
 			"timeofban INTEGER NOT NULL, " \
+			"bantime INTEGER NOT NULL, " \
+			"bancount INTEGER NOT NULL default 1, " \
 			"data JSON, " \
 			"FOREIGN KEY(jail) REFERENCES jails(name) " \
 			");" \
 			"CREATE INDEX IF NOT EXISTS bans_jail_timeofban_ip ON bans(jail, timeofban);" \
 			"CREATE INDEX IF NOT EXISTS bans_jail_ip ON bans(jail, ip);" \
 			"CREATE INDEX IF NOT EXISTS bans_ip ON bans(ip);")
+		,('bips', "CREATE TABLE IF NOT EXISTS bips(" \
+			"ip TEXT NOT NULL, " \
+			"jail TEXT NOT NULL, " \
+			"timeofban INTEGER NOT NULL, " \
+			"bantime INTEGER NOT NULL, " \
+			"bancount INTEGER NOT NULL default 1, " \
+			"data JSON, " \
+			"PRIMARY KEY(ip, jail), " \
+			"FOREIGN KEY(jail) REFERENCES jails(name) " \
+			");" \
+			"CREATE INDEX IF NOT EXISTS bips_timeofban ON bips(timeofban);" \
+			"CREATE INDEX IF NOT EXISTS bips_ip ON bips(ip);")
 	)
 	_CREATE_TABS = dict(_CREATE_SCRIPTS)


-	def __init__(self, filename, purgeAge=24*60*60):
+	def __init__(self, filename, purgeAge=24*60*60, outDatedFactor=3):
 		self.maxMatches = 10
 		self._lock = RLock()
 		self._dbFilename = filename
 		self._purgeAge = purgeAge
+		self._outDatedFactor = outDatedFactor;
 		self._connectDB()

 	def _connectDB(self, checkIntegrity=False):
@@ -366,13 +381,32 @@ class Fail2BanDb(object):

 			if version < 2 and self._tableExists(cur, "logs"):
 				cur.executescript("BEGIN TRANSACTION;"
-						"CREATE TEMPORARY TABLE logs_temp AS SELECT * FROM logs;"
-						"DROP TABLE logs;"
-						"%s;"
-						"INSERT INTO logs SELECT * from logs_temp;"
-						"DROP TABLE logs_temp;"
-						"UPDATE fail2banDb SET version = 2;"
-						"COMMIT;" % Fail2BanDb._CREATE_TABS['logs'])
+							"CREATE TEMPORARY TABLE logs_temp AS SELECT * FROM logs;"
+							"DROP TABLE logs;"
+							"%s;"
+							"INSERT INTO logs SELECT * from logs_temp;"
+							"DROP TABLE logs_temp;"
+							"UPDATE fail2banDb SET version = 2;"
+							"COMMIT;" % Fail2BanDb._CREATE_TABS['logs'])
+
+			if version < 3 and self._tableExists(cur, "bans"):
+				# set ban-time to -2 (note it means rather unknown, as persistent, will be fixed by restore):
+				cur.executescript("BEGIN TRANSACTION;"
+							"CREATE TEMPORARY TABLE bans_temp AS SELECT jail, ip, timeofban, -2 as bantime, 1 as bancount, data FROM bans;"
+							"DROP TABLE bans;"
+							"%s;\n"
+							"INSERT INTO bans SELECT * from bans_temp;"
+							"DROP TABLE bans_temp;"
+							"COMMIT;" % Fail2BanDb._CREATE_TABS['bans'])
+			if version < 4 and not self._tableExists(cur, "bips"):
+				cur.executescript("BEGIN TRANSACTION;"
+							"%s;\n"
+							"UPDATE fail2banDb SET version = 4;"
+							"COMMIT;" % Fail2BanDb._CREATE_TABS['bips'])
+				if self._tableExists(cur, "bans"):
+					cur.execute(
+							"INSERT OR REPLACE INTO bips(ip, jail, timeofban, bantime, bancount, data)"
+							"  SELECT ip, jail, timeofban, bantime, bancount, data FROM bans order by timeofban")

 			cur.execute("SELECT version FROM fail2banDb LIMIT 1")
 			return cur.fetchone()[0]
@@ -455,24 +489,22 @@ class Fail2BanDb(object):
 			If log was already present in database, value of last position
 			in the log file; else `None`
 		"""
-		return self._addLog(cur, jail, container.getFileName(), container.getPos(), container.getHash())
-
-	def _addLog(self, cur, jail, name, pos=0, md5=None):
 		lastLinePos = None
 		cur.execute(
 			"SELECT firstlinemd5, lastfilepos FROM logs "
 				"WHERE jail=? AND path=?",
-			(jail.name, name))
+			(jail.name, container.getFileName()))
 		try:
 			firstLineMD5, lastLinePos = cur.fetchone()
 		except TypeError:
-			firstLineMD5 = None
+			firstLineMD5 = False

-		if not firstLineMD5 and (pos or md5):
-			cur.execute(
-					"INSERT OR REPLACE INTO logs(jail, path, firstlinemd5, lastfilepos) "
-						"VALUES(?, ?, ?, ?)", (jail.name, name, md5, pos))
-		if md5 is not None and md5 != firstLineMD5:
+		cur.execute(
+				"INSERT OR REPLACE INTO logs(jail, path, firstlinemd5, lastfilepos) "
+					"VALUES(?, ?, ?, ?)",
+				(jail.name, container.getFileName(),
+					container.getHash(), container.getPos()))
+		if container.getHash() != firstLineMD5:
 			lastLinePos = None
 		return lastLinePos

@@ -501,7 +533,7 @@ class Fail2BanDb(object):
 		return set(row[0] for row in cur.fetchmany())

 	@commitandrollback
-	def updateLog(self, cur, jail, container):
+	def updateLog(self, cur, *args, **kwargs):
 		"""Updates hash and last position in log file.

 		Parameters
@@ -511,48 +543,14 @@ class Fail2BanDb(object):
 		container : FileContainer
 			File container of the log file being updated.
 		"""
-		self._updateLog(cur, jail, container.getFileName(), container.getPos(), container.getHash())
+		self._updateLog(cur, *args, **kwargs)

-	def _updateLog(self, cur, jail, name, pos, md5):
+	def _updateLog(self, cur, jail, container):
 		cur.execute(
 			"UPDATE logs SET firstlinemd5=?, lastfilepos=? "
-				"WHERE jail=? AND path=?", (md5, pos, jail.name, name))
-		# be sure it is set (if not available):
-		if not cur.rowcount:
-			cur.execute(
-					"INSERT OR REPLACE INTO logs(jail, path, firstlinemd5, lastfilepos) "
-						"VALUES(?, ?, ?, ?)", (jail.name, name, md5, pos))
-
-	@commitandrollback
-	def getJournalPos(self, cur, jail, name, time=0, iso=None):
-		"""Get journal position from database.
-
-		Parameters
-		----------
-		jail : Jail
-			Jail of which the journal belongs to.
-		name, time, iso :
-			Journal name (typically systemd-journal) and last known time.
-
-		Returns
-		-------
-		int (or float)
-			Last position (as time) if it was already present in database; else `None`
-		"""
-		return self._addLog(cur, jail, name, time, iso); # no hash, just time as iso
-
-	@commitandrollback
-	def updateJournal(self, cur, jail, name, time, iso):
-		"""Updates last position (as time) of journal.
-
-		Parameters
-		----------
-		jail : Jail
-			Jail of which the journal belongs to.
-		name, time, iso :
-			Journal name (typically systemd-journal) and last known time.
-		"""
-		self._updateLog(cur, jail, name, time, iso); # no hash, just time as iso
+				"WHERE jail=? AND path=?",
+			(container.getHash(), container.getPos(),
+				jail.name, container.getFileName()))

 	@commitandrollback
 	def addBan(self, cur, jail, ticket):
@@ -585,8 +583,13 @@ class Fail2BanDb(object):
 			data = data.copy()
 			del data['matches']
 		cur.execute(
-			"INSERT INTO bans(jail, ip, timeofban, data) VALUES(?, ?, ?, ?)",
-			(jail.name, ip, int(round(ticket.getTime())), data))
+			"INSERT INTO bans(jail, ip, timeofban, bantime, bancount, data) VALUES(?, ?, ?, ?, ?, ?)",
+			(jail.name, ip, int(round(ticket.getTime())), ticket.getBanTime(jail.actions.getBanTime()), ticket.getBanCount(),
+				data))
+		cur.execute(
+			"INSERT OR REPLACE INTO bips(ip, jail, timeofban, bantime, bancount, data) VALUES(?, ?, ?, ?, ?, ?)",
+			(ip, jail.name, int(round(ticket.getTime())), ticket.getBanTime(jail.actions.getBanTime()), ticket.getBanCount(),
+				data))

 	@commitandrollback
 	def delBan(self, cur, jail, *args):
@@ -599,16 +602,20 @@ class Fail2BanDb(object):
 		args : list of IP
 			IPs to be removed, if not given all tickets of jail will be removed.
 		"""
-		query = "DELETE FROM bans WHERE jail = ?"
+		query1 = "DELETE FROM bips WHERE jail = ?"
+		query2 = "DELETE FROM bans WHERE jail = ?"
 		queryArgs = [jail.name];
 		if not len(args):
-			cur.execute(query, queryArgs);
+			cur.execute(query1, queryArgs);
+			cur.execute(query2, queryArgs);
 			return
-		query += " AND ip = ?"
+		query1 += " AND ip = ?"
+		query2 += " AND ip = ?"
 		queryArgs.append('');
 		for ip in args:
 			queryArgs[1] = str(ip);
-			cur.execute(query, queryArgs);
+			cur.execute(query1, queryArgs);
+			cur.execute(query2, queryArgs);

 	@commitandrollback
 	def _getBans(self, cur, jail=None, bantime=None, ip=None):
@@ -726,18 +733,41 @@ class Fail2BanDb(object):
 				self._bansMergedCache[cacheKey] = tickets if ip is None else ticket
 			return tickets if ip is None else ticket

+	@commitandrollback
+	def getBan(self, cur, ip, jail=None, forbantime=None, overalljails=None, fromtime=None):
+		ip = str(ip)
+		if not overalljails:
+			query = "SELECT bancount, timeofban, bantime FROM bips"
+		else:
+			query = "SELECT sum(bancount), max(timeofban), sum(bantime) FROM bips"
+		query += " WHERE ip = ?"
+		queryArgs = [ip]
+		if not overalljails and jail is not None:
+			query += " AND jail=?"
+			queryArgs.append(jail.name)
+		if forbantime is not None:
+			query += " AND timeofban > ?"
+			queryArgs.append(MyTime.time() - forbantime)
+		if fromtime is not None:
+			query += " AND timeofban > ?"
+			queryArgs.append(fromtime)
+		if overalljails or jail is None:
+			query += " GROUP BY ip ORDER BY timeofban DESC LIMIT 1"
+		cur = self._db.cursor()
+		return cur.execute(query, queryArgs)
+
 	def _getCurrentBans(self, cur, jail = None, ip = None, forbantime=None, fromtime=None):
-		if fromtime is None:
-			fromtime = MyTime.time()
 		queryArgs = []
 		if jail is not None:
-			query = "SELECT ip, timeofban, data FROM bans WHERE jail=?"
+			query = "SELECT ip, timeofban, bantime, bancount, data FROM bips WHERE jail=?"
 			queryArgs.append(jail.name)
 		else:
-			query = "SELECT ip, max(timeofban), data FROM bans WHERE 1"
+			query = "SELECT ip, max(timeofban), bantime, bancount, data FROM bips WHERE 1"
 		if ip is not None:
 			query += " AND ip=?"
 			queryArgs.append(ip)
+		query += " AND (timeofban + bantime > ? OR bantime <= -1)"
+		queryArgs.append(fromtime)
 		if forbantime not in (None, -1): # not specified or persistent (all)
 			query += " AND timeofban > ?"
 			queryArgs.append(fromtime - forbantime)
@@ -748,33 +778,90 @@ class Fail2BanDb(object):
 		cur = self._db.cursor()
 		return cur.execute(query, queryArgs)

-	def getCurrentBans(self, jail = None, ip = None, forbantime=None, fromtime=None, maxmatches=None):
+	@commitandrollback
+	def getCurrentBans(self, cur, jail=None, ip=None, forbantime=None, fromtime=None,
+		correctBanTime=True, maxmatches=None
+	):
+		"""Reads tickets (with merged info) currently affected from ban from the database.
+		
+		There are all the tickets corresponding parameters jail/ip, forbantime,
+		fromtime (normally now).
+		
+		If correctBanTime specified (default True) it will fix the restored ban-time 
+		(and therefore endOfBan) of the ticket (normally it is ban-time of jail as maximum)
+		for all tickets with ban-time greater (or persistent).
+		"""
+		if fromtime is None:
+			fromtime = MyTime.time()
 		tickets = []
 		ticket = None
+		if correctBanTime is True:
+			correctBanTime = jail.getMaxBanTime() if jail is not None else None
+			# don't change if persistent allowed:
+			if correctBanTime == -1: correctBanTime = None

-		with self._lock:
-			results = list(self._getCurrentBans(self._db.cursor(), 
-				jail=jail, ip=ip, forbantime=forbantime, fromtime=fromtime))
-
-		if results:
-			for banip, timeofban, data in results:
-				# logSys.debug('restore ticket   %r, %r, %r', banip, timeofban, data)
-				ticket = FailTicket(banip, timeofban, data=data)
-				# filter matches if expected (current count > as maxmatches specified):
-				if maxmatches is None:
-					maxmatches = self.maxMatches
-				if maxmatches:
-					matches = ticket.getMatches()
-					if matches and len(matches) > maxmatches:
-						ticket.setMatches(matches[-maxmatches:])
-				else:
-					ticket.setMatches(None)
-				# logSys.debug('restored ticket: %r', ticket)
-				if ip is not None: return ticket
-				tickets.append(ticket)
+		for ticket in self._getCurrentBans(cur, jail=jail, ip=ip, 
+			forbantime=forbantime, fromtime=fromtime
+		):
+			# can produce unpack error (database may return sporadical wrong-empty row):
+			try:
+				banip, timeofban, bantime, bancount, data = ticket
+				# additionally check for empty values:
+				if banip is None or banip == "": # pragma: no cover
+					raise ValueError('unexpected value %r' % (banip,))
+				# if bantime unknown (after upgrade-db from earlier version), just use min known ban-time:
+				if bantime == -2: # todo: remove it in future version
+					bantime = jail.actions.getBanTime() if jail is not None else (
+						correctBanTime if correctBanTime else 600)
+				elif correctBanTime and correctBanTime >= 0:
+					# if persistent ban (or greater as max), use current max-bantime of the jail:
+					if bantime == -1 or bantime > correctBanTime:
+						bantime = correctBanTime
+				# after correction check the end of ban again:
+				if bantime != -1 and timeofban + bantime <= fromtime:
+					# not persistent and too old - ignore it:
+					logSys.debug("ignore ticket (with new max ban-time %r): too old %r <= %r, ticket: %r",
+						bantime, timeofban + bantime, fromtime, ticket)
+					continue
+			except ValueError as e: # pragma: no cover
+				logSys.debug("get current bans: ignore row %r - %s", ticket, e)
+				continue
+			# logSys.debug('restore ticket   %r, %r, %r', banip, timeofban, data)
+			ticket = FailTicket(banip, timeofban, data=data)
+			# filter matches if expected (current count > as maxmatches specified):
+			if maxmatches is None:
+				maxmatches = self.maxMatches
+			if maxmatches:
+				matches = ticket.getMatches()
+				if matches and len(matches) > maxmatches:
+					ticket.setMatches(matches[-maxmatches:])
+			else:
+				ticket.setMatches(None)
+			# logSys.debug('restored ticket: %r', ticket)
+			ticket.setBanTime(bantime)
+			ticket.setBanCount(bancount)
+			if ip is not None: return ticket
+			tickets.append(ticket)

 		return tickets

+	def _cleanjails(self, cur):
+		"""Remove empty jails jails and log files from database.
+		"""
+		cur.execute(
+			"DELETE FROM jails WHERE enabled = 0 "
+				"AND NOT EXISTS(SELECT * FROM bans WHERE jail = jails.name) "
+				"AND NOT EXISTS(SELECT * FROM bips WHERE jail = jails.name)")
+
+	def _purge_bips(self, cur):
+		"""Purge old bad ips (jails and log files from database).
+		Currently it is timed out IP, whose time since last ban is several times out-dated (outDatedFactor is default 3).
+		Permanent banned ips will be never removed.
+		"""
+		cur.execute(
+			"DELETE FROM bips WHERE timeofban < ? and bantime != -1 and (timeofban + (bantime * ?)) < ?",
+			(int(MyTime.time()) - self._purgeAge, self._outDatedFactor, int(MyTime.time()) - self._purgeAge))
+
 	@commitandrollback
 	def purge(self, cur):
 		"""Purge old bans, jails and log files from database.
@@ -783,7 +870,6 @@ class Fail2BanDb(object):
 		cur.execute(
 			"DELETE FROM bans WHERE timeofban < ?",
 			(MyTime.time() - self._purgeAge, ))
-		cur.execute(
-			"DELETE FROM jails WHERE enabled = 0 "
-				"AND NOT EXISTS(SELECT * FROM bans WHERE jail = jails.name)")
+		self._purge_bips(cur)
+		self._cleanjails(cur)

@@ -282,8 +282,6 @@ class DateDetector(object):
 				elif "{DATE}" in key:
 					self.addDefaultTemplate(preMatch=pattern, allDefaults=False)
 					return
-				elif key == "{NONE}":
-					template = _getPatternTemplate('{UNB}^', key)
 				else:
 					template = _getPatternTemplate(pattern, key)

@@ -339,76 +337,65 @@ class DateDetector(object):
 		# if no templates specified - default templates should be used:
 		if not len(self.__templates):
 			self.addDefaultTemplate()
-		log = logSys.log if logSys.getEffectiveLevel() <= logLevel else lambda *args: None
-		log(logLevel-1, "try to match time for line: %.120s", line)
-
-		# first try to use last template with same start/end position:
+		logSys.log(logLevel-1, "try to match time for line: %.120s", line)
 		match = None
-		found = None, 0x7fffffff, 0x7fffffff, -1
+		# first try to use last template with same start/end position:
 		ignoreBySearch = 0x7fffffff
 		i = self.__lastTemplIdx
 		if i < len(self.__templates):
 			ddtempl = self.__templates[i]
 			template = ddtempl.template
 			if template.flags & (DateTemplate.LINE_BEGIN|DateTemplate.LINE_END):
-				log(logLevel-1, "  try to match last anchored template #%02i ...", i)
+				if logSys.getEffectiveLevel() <= logLevel-1: # pragma: no cover - very-heavy debug
+					logSys.log(logLevel-1, "  try to match last anchored template #%02i ...", i)
 				match = template.matchDate(line)
 				ignoreBySearch = i
 			else:
 				distance, endpos = self.__lastPos[0], self.__lastEndPos[0]
-				log(logLevel-1, "  try to match last template #%02i (from %r to %r): ...%r==%r %s %r==%r...",
-					i, distance, endpos,
-					line[distance-1:distance], self.__lastPos[1],
-					line[distance:endpos],
-					line[endpos:endpos+1], self.__lastEndPos[2])
-				# check same boundaries left/right, outside fully equal, inside only if not alnum (e. g. bound RE
-				# with space or some special char), otherwise possible collision/pattern switch:
-				if ((
-					line[distance-1:distance] == self.__lastPos[1] or
-						(line[distance] == self.__lastPos[2] and not self.__lastPos[2].isalnum())
-					) and (
-					line[endpos:endpos+1] == self.__lastEndPos[2] or
-						(line[endpos-1] == self.__lastEndPos[1] and not self.__lastEndPos[1].isalnum())
-				)):
-					# search in line part only:
-					log(logLevel-1, "  boundaries are correct, search in part %r", line[distance:endpos])
+				if logSys.getEffectiveLevel() <= logLevel-1:
+					logSys.log(logLevel-1, "  try to match last template #%02i (from %r to %r): ...%r==%r %s %r==%r...",
+						i, distance, endpos, 
+						line[distance-1:distance], self.__lastPos[1],
+						line[distance:endpos],
+						line[endpos:endpos+1], self.__lastEndPos[1])
+				# check same boundaries left/right, otherwise possible collision/pattern switch:
+				if (line[distance-1:distance] == self.__lastPos[1] and 
+						line[endpos:endpos+1] == self.__lastEndPos[1]
+				):
 					match = template.matchDate(line, distance, endpos)
-				else:
-					log(logLevel-1, "  boundaries show conflict, try whole search")
-					match = template.matchDate(line)
-					ignoreBySearch = i
 			if match:
 				distance = match.start()
 				endpos = match.end()
 				# if different position, possible collision/pattern switch:
 				if (
-					len(self.__templates) == 1 or # single template:
 					template.flags & (DateTemplate.LINE_BEGIN|DateTemplate.LINE_END) or 
 					(distance == self.__lastPos[0] and endpos == self.__lastEndPos[0])
 				):
-					log(logLevel, "  matched last time template #%02i", i)
+					logSys.log(logLevel, "  matched last time template #%02i", i)
 				else:
-					log(logLevel, "  ** last pattern collision - pattern change, reserve & search ...")
-					found = match, distance, endpos, i; # save current best alternative
+					logSys.log(logLevel, "  ** last pattern collision - pattern change, search ...")
 					match = None
 			else:
-				log(logLevel, "  ** last pattern not found - pattern change, search ...")
+				logSys.log(logLevel, "  ** last pattern not found - pattern change, search ...")
 		# search template and better match:
 		if not match:
-			log(logLevel, " search template (%i) ...", len(self.__templates))
+			logSys.log(logLevel, " search template (%i) ...", len(self.__templates))
+			found = None, 0x7fffffff, 0x7fffffff, -1
 			i = 0
 			for ddtempl in self.__templates:
+				if logSys.getEffectiveLevel() <= logLevel-1:
+					logSys.log(logLevel-1, "  try template #%02i: %s", i, ddtempl.name)
 				if i == ignoreBySearch:
 					i += 1
 					continue
-				log(logLevel-1, "  try template #%02i: %s", i, ddtempl.name)
 				template = ddtempl.template
 				match = template.matchDate(line)
 				if match:
 					distance = match.start()
 					endpos = match.end()
-					log(logLevel, "  matched time template #%02i (at %r <= %r, %r) %s",
-						i, distance, ddtempl.distance, self.__lastPos[0], template.name)
+					if logSys.getEffectiveLevel() <= logLevel:
+						logSys.log(logLevel, "  matched time template #%02i (at %r <= %r, %r) %s",
+							i, distance, ddtempl.distance, self.__lastPos[0], template.name)
 					## last (or single) template - fast stop:
 					if i+1 >= len(self.__templates):
 						break
@@ -421,7 +408,7 @@ class DateDetector(object):
 					## [grave] if distance changed, possible date-match was found somewhere 
 					## in body of message, so save this template, and search further:
 					if distance > ddtempl.distance or distance > self.__lastPos[0]:
-						log(logLevel, "  ** distance collision - pattern change, reserve")
+						logSys.log(logLevel, "  ** distance collision - pattern change, reserve")
 						## shortest of both:
 						if distance < found[1]:
 							found = match, distance, endpos, i
@@ -435,7 +422,7 @@ class DateDetector(object):
 			# check other template was found (use this one with shortest distance):
 			if not match and found[0]:
 				match, distance, endpos, i = found
-				log(logLevel, "  use best time template #%02i", i)
+				logSys.log(logLevel, "  use best time template #%02i", i)
 				ddtempl = self.__templates[i]
 				template = ddtempl.template
 		# we've winner, incr hits, set distance, usage, reorder, etc:
@@ -445,8 +432,8 @@ class DateDetector(object):
 			ddtempl.distance = distance
 			if self.__firstUnused == i:
 				self.__firstUnused += 1
-			self.__lastPos = distance, line[distance-1:distance], line[distance]
-			self.__lastEndPos = endpos, line[endpos-1], line[endpos:endpos+1]
+			self.__lastPos = distance, line[distance-1:distance]
+			self.__lastEndPos = endpos, line[endpos:endpos+1]
 			# if not first - try to reorder current template (bubble up), they will be not sorted anymore:
 			if i and i != self.__lastTemplIdx:
 				i = self._reorderTemplate(i)
@@ -455,7 +442,7 @@ class DateDetector(object):
 			return (match, template)

 		# not found:
-		log(logLevel, " no template.")
+		logSys.log(logLevel, " no template.")
 		return (None, None)

 	@property
@@ -36,16 +36,15 @@ logSys = getLogger(__name__)
 RE_GROUPED = re.compile(r'(?<!(?:\(\?))(?<!\\)\((?!\?)')
 RE_GROUP = ( re.compile(r'^((?:\(\?\w+\))?\^?(?:\(\?\w+\))?)(.*?)(\$?)$'), r"\1(\2)\3" )

-RE_EXLINE_NO_BOUNDS = re.compile(r'^\{UNB\}')
 RE_EXLINE_BOUND_BEG = re.compile(r'^\{\^LN-BEG\}')
-RE_EXSANC_BOUND_BEG = re.compile(r'^\((?:\?:)?\^\|\\b\|\\W\)')
+RE_EXSANC_BOUND_BEG = re.compile(r'^\(\?:\^\|\\b\|\\W\)')
 RE_EXEANC_BOUND_BEG = re.compile(r'\(\?=\\b\|\\W\|\$\)$')
-RE_NO_WRD_BOUND_BEG = re.compile(r'^\(*(?:\(\?\w+\))?(?:\^|\(*\*\*|\((?:\?:)?\^)')
+RE_NO_WRD_BOUND_BEG = re.compile(r'^\(*(?:\(\?\w+\))?(?:\^|\(*\*\*|\(\?:\^)')
 RE_NO_WRD_BOUND_END = re.compile(r'(?<!\\)(?:\$\)?|\\b|\\s|\*\*\)*)$')
 RE_DEL_WRD_BOUNDS = ( re.compile(r'^\(*(?:\(\?\w+\))?\(*\*\*|(?<!\\)\*\*\)*$'), 
 	                    lambda m: m.group().replace('**', '') )

-RE_LINE_BOUND_BEG = re.compile(r'^(?:\(\?\w+\))?(?:\^|\((?:\?:)?\^(?!\|))')
+RE_LINE_BOUND_BEG = re.compile(r'^(?:\(\?\w+\))?(?:\^|\(\?:\^(?!\|))')
 RE_LINE_BOUND_END = re.compile(r'(?<![\\\|])(?:\$\)?)$')

 RE_ALPHA_PATTERN = re.compile(r'(?<!\%)\%[aAbBpc]')
@@ -120,7 +119,7 @@ class DateTemplate(object):
 		if boundBegin:
 			self.flags |= DateTemplate.WORD_BEGIN if wordBegin != 'start' else DateTemplate.LINE_BEGIN
 			if wordBegin != 'start':
-				regex = r'(?=^|\b|\W)' + regex
+				regex = r'(?:^|\b|\W)' + regex
 			else:
 				regex = r"^(?:\W{0,2})?" + regex
 				if not self.name.startswith('{^LN-BEG}'):
@@ -129,10 +128,8 @@ class DateTemplate(object):
 		if boundEnd:
 			self.flags |= DateTemplate.WORD_END
 			regex += r'(?=\b|\W|$)'
-		if not (self.flags & DateTemplate.LINE_BEGIN) and RE_LINE_BOUND_BEG.search(regex):
-			self.flags |= DateTemplate.LINE_BEGIN
-		if not (self.flags & DateTemplate.LINE_END) and RE_LINE_BOUND_END.search(regex):
-			self.flags |= DateTemplate.LINE_END
+		if RE_LINE_BOUND_BEG.search(regex): self.flags |= DateTemplate.LINE_BEGIN
+		if RE_LINE_BOUND_END.search(regex): self.flags |= DateTemplate.LINE_END
 		# remove possible special pattern "**" in front and end of regex:
 		regex = RE_DEL_WRD_BOUNDS[0].sub(RE_DEL_WRD_BOUNDS[1], regex)
 		self._regex = regex
@@ -191,7 +188,7 @@ class DateTemplate(object):
 	def unboundPattern(pattern):
 		return RE_EXEANC_BOUND_BEG.sub('',
 			RE_EXSANC_BOUND_BEG.sub('',
-				RE_EXLINE_BOUND_BEG.sub('', RE_EXLINE_NO_BOUNDS.sub('', pattern))
+				RE_EXLINE_BOUND_BEG.sub('', pattern)
 			)
 		)

@@ -300,10 +297,6 @@ class DatePatternRegex(DateTemplate):
 	def setRegex(self, pattern, wordBegin=True, wordEnd=True):
 		# original pattern:
 		self._pattern = pattern
-		# if unbound signalled - reset boundaries left and right:
-		if RE_EXLINE_NO_BOUNDS.search(pattern):
-			pattern = RE_EXLINE_NO_BOUNDS.sub('', pattern)
-			wordBegin = wordEnd = False
 		# if explicit given {^LN-BEG} - remove it from pattern and set 'start' in wordBegin:
 		if wordBegin and RE_EXLINE_BOUND_BEG.search(pattern):
 			pattern = RE_EXLINE_BOUND_BEG.sub('', pattern)
@@ -27,7 +27,7 @@ __license__ = "GPL"
 from threading import Lock
 import logging

-from .ticket import FailTicket
+from .ticket import FailTicket, BanTicket
 from ..helpers import getLogger, BgService

 # Gets the instance of the logger.
@@ -43,20 +43,26 @@ class FailManager:
 		self.__maxRetry = 3
 		self.__maxTime = 600
 		self.__failTotal = 0
-		self.maxMatches = 5
+		self.maxMatches = 50
 		self.__bgSvc = BgService()
 	
 	def setFailTotal(self, value):
-		self.__failTotal = value
+		with self.__lock:
+			self.__failTotal = value
 		
 	def getFailTotal(self):
-		return self.__failTotal
+		with self.__lock:
+			return self.__failTotal
 	
 	def getFailCount(self):
 		# may be slow on large list of failures, should be used for test purposes only...
 		with self.__lock:
 			return len(self.__failList), sum([f.getRetry() for f in self.__failList.values()])

+	def getFailTotal(self):
+		with self.__lock:
+			return self.__failTotal
+
 	def setMaxRetry(self, value):
 		self.__maxRetry = value
 	
@@ -69,7 +75,7 @@ class FailManager:
 	def getMaxTime(self):
 		return self.__maxTime

-	def addFailure(self, ticket, count=1):
+	def addFailure(self, ticket, count=1, observed=False):
 		attempts = 1
 		with self.__lock:
 			fid = ticket.getID()
@@ -86,7 +92,10 @@ class FailManager:
 					if attempt <= 0:
 						attempt += 1
 				unixTime = ticket.getTime()
-				fData.adjustTime(unixTime, self.__maxTime)
+				fData.setLastTime(unixTime)
+				if fData.getLastReset() < unixTime - self.__maxTime:
+					fData.setLastReset(unixTime)
+					fData.setRetry(0)
 				fData.inc(matches, attempt, count)
 				# truncate to maxMatches:
 				if self.maxMatches:
@@ -96,11 +105,14 @@ class FailManager:
 				else:
 					fData.setMatches(None)
 			except KeyError:
+				# not found - already banned - prevent to add failure if comes from observer:
+				if observed or isinstance(ticket, BanTicket):
+					return ticket.getRetry()
 				# if already FailTicket - add it direct, otherwise create (using copy all ticket data):
 				if isinstance(ticket, FailTicket):
 					fData = ticket;
 				else:
-					fData = FailTicket(ticket=ticket)
+					fData = FailTicket.wrap(ticket)
 				if count > ticket.getAttempt():
 					fData.setRetry(count)
 				self.__failList[fid] = fData
@@ -121,12 +133,13 @@ class FailManager:
 		return attempts
 	
 	def size(self):
-		return len(self.__failList)
+		with self.__lock:
+			return len(self.__failList)
 	
 	def cleanup(self, time):
 		with self.__lock:
 			todelete = [fid for fid,item in self.__failList.iteritems() \
-				if item.getTime() + self.__maxTime <= time]
+				if item.getLastTime() + self.__maxTime <= time]
 			if len(todelete) == len(self.__failList):
 				# remove all:
 				self.__failList = dict()
@@ -140,7 +153,7 @@ class FailManager:
 			else:
 				# create new dictionary without items to be deleted:
 				self.__failList = dict((fid,item) for fid,item in self.__failList.iteritems() \
-					if item.getTime() + self.__maxTime > time)
+					if item.getLastTime() + self.__maxTime > time)
 		self.__bgSvc.service()
 	
 	def delFailure(self, fid):
@@ -138,8 +138,6 @@ class Regex:
 		except sre_constants.error:
 			raise RegexException("Unable to compile regular expression '%s'" %
 								 regex)
-		# set fetch handler depending on presence of alternate tags:
-		self.getGroups = self._getGroupsWithAlt if self._altValues else self._getGroups

 	def __str__(self):
 		return "%s(%r)" % (self.__class__.__name__, self._regex)
@@ -279,12 +277,11 @@ class Regex:
 	# Returns all matched groups.
 	#

-	def _getGroups(self):
-		return self._matchCache.groupdict()
-
-	def _getGroupsWithAlt(self):
-		fail = self._matchCache.groupdict()
+	def getGroups(self):
+		if not self._altValues:
+			return self._matchCache.groupdict()
 		# merge alternate values (e. g. 'alt_user_1' -> 'user' or 'alt_host' -> 'host'):
+		fail = self._matchCache.groupdict()
 		#fail = fail.copy()
 		for k,n in self._altValues:
 			v = fail.get(k)
@@ -292,9 +289,6 @@ class Regex:
 				fail[n] = v
 		return fail

-	def getGroups(self): # pragma: no cover - abstract function (replaced in __init__)
-		pass
-
 	##
 	# Returns skipped lines.
 	#
@@ -33,6 +33,7 @@ import time
 from .actions import Actions
 from .failmanager import FailManagerEmpty, FailManager
 from .ipdns import DNSUtils, IPAddr
+from .observer import Observers
 from .ticket import FailTicket
 from .jailthread import JailThread
 from .datedetector import DateDetector, validateTimeZone
@@ -80,7 +81,6 @@ class Filter(JailThread):
 		## Ignore own IPs flag:
 		self.__ignoreSelf = True
 		## The ignore IP list.
-		self.__ignoreIpSet = set()
 		self.__ignoreIpList = []
 		## External command
 		self.__ignoreCommand = False
@@ -106,16 +106,8 @@ class Filter(JailThread):
 		self.returnRawHost = False
 		## check each regex (used for test purposes):
 		self.checkAllRegex = False
-		## avoid finding of pending failures (without ID/IP, used in fail2ban-regex):
-		self.ignorePending = True
-		## callback called on ignoreregex match :
-		self.onIgnoreRegex = None
 		## if true ignores obsolete failures (failure time < now - findTime):
 		self.checkFindTime = True
-		## shows that filter is in operation mode (processing new messages):
-		self.inOperation = True
-		## if true prevents against retarded banning in case of RC by too many failures (disabled only for test purposes):
-		self.banASAP = True
 		## Ticks counter
 		self.ticks = 0
 		## Thread name:
@@ -177,7 +169,7 @@ class Filter(JailThread):
 	# @param value the regular expression

 	def addFailRegex(self, value):
-		multiLine = self.__lineBufferSize > 1
+		multiLine = self.getMaxLines() > 1
 		try:
 			regex = FailRegex(value, prefRegex=self.__prefRegex, multiline=multiLine,
 				useDns=self.__useDns)
@@ -460,10 +452,10 @@ class Filter(JailThread):
 		logSys.info(
 			"[%s] Attempt %s - %s", self.jailName, ip, datetime.datetime.fromtimestamp(unixTime).strftime("%Y-%m-%d %H:%M:%S")
 		)
-		attempts = self.failManager.addFailure(ticket, len(matches) or 1)
+		self.failManager.addFailure(ticket, len(matches) or 1)
+
 		# Perform the ban if this attempt is resulted to:
-		if attempts >= self.failManager.getMaxRetry():
-			self.performBan(ip)
+		self.performBan(ip)

 		return 1

@@ -492,36 +484,28 @@ class Filter(JailThread):
 		# Create IP address object
 		ip = IPAddr(ipstr)
 		# Avoid exact duplicates
-		if ip in self.__ignoreIpSet or ip in self.__ignoreIpList:
-			logSys.log(logging.MSG, "  Ignore duplicate %r (%r), already in ignore list", ip, ipstr)
+		if ip in self.__ignoreIpList:
+			logSys.warn("  Ignore duplicate %r (%r), already in ignore list", ip, ipstr)
 			return
 		# log and append to ignore list
 		logSys.debug("  Add %r to ignore list (%r)", ip, ipstr)
-		# if single IP (not DNS or a subnet) add to set, otherwise to list:
-		if ip.isSingle:
-			self.__ignoreIpSet.add(ip)
-		else:
-			self.__ignoreIpList.append(ip)
+		self.__ignoreIpList.append(ip)

 	def delIgnoreIP(self, ip=None):
 		# clear all:
 		if ip is None:
-			self.__ignoreIpSet.clear()
 			del self.__ignoreIpList[:]
 			return
 		# delete by ip:
 		logSys.debug("  Remove %r from ignore list", ip)
-		if ip in self.__ignoreIpSet:
-			self.__ignoreIpSet.remove(ip)
-		else:
-			self.__ignoreIpList.remove(ip)
+		self.__ignoreIpList.remove(ip)

 	def logIgnoreIp(self, ip, log_ignore, ignore_source="unknown source"):
 		if log_ignore:
 			logSys.info("[%s] Ignore %s by %s", self.jailName, ip, ignore_source)

 	def getIgnoreIP(self):
-		return self.__ignoreIpList + list(self.__ignoreIpSet)
+		return self.__ignoreIpList

 	##
 	# Check if IP address/DNS is in the ignore list.
@@ -561,11 +545,8 @@ class Filter(JailThread):
 			if self.__ignoreCache: c.set(key, True)
 			return True

-		# check if the IP is covered by ignore IP (in set or in subnet/dns):
-		if ip in self.__ignoreIpSet:
-			self.logIgnoreIp(ip, log_ignore, ignore_source="ip")
-			return True
 		for net in self.__ignoreIpList:
+			# check if the IP is covered by ignore IP
 			if ip.isInNet(net):
 				self.logIgnoreIp(ip, log_ignore, ignore_source=("ip" if net.isValid else "dns"))
 				if self.__ignoreCache: c.set(key, True)
@@ -588,89 +569,29 @@ class Filter(JailThread):
 		if self.__ignoreCache: c.set(key, False)
 		return False

-	def _logWarnOnce(self, nextLTM, *args):
-		"""Log some issue as warning once per day, otherwise level 7"""
-		if MyTime.time() < getattr(self, nextLTM, 0):
-			if logSys.getEffectiveLevel() <= 7: logSys.log(7, *(args[0]))
-		else:
-			setattr(self, nextLTM, MyTime.time() + 24*60*60)
-			for args in args:
-				logSys.warning('[%s] ' + args[0], self.jailName, *args[1:])
-
 	def processLine(self, line, date=None):
 		"""Split the time portion from log msg and return findFailures on them
 		"""
-		logSys.log(7, "Working on line %r", line)
-
-		noDate = False
 		if date:
 			tupleLine = line
-			self.__lastTimeText = tupleLine[1]
-			self.__lastDate = date
 		else:
-			# try to parse date:
-			timeMatch = self.dateDetector.matchTime(line)
-			m = timeMatch[0]
-			if m:
-				s = m.start(1)
-				e = m.end(1)
-				m = line[s:e]
-				tupleLine = (line[:s], m, line[e:])
-				if m: # found and not empty - retrive date:
-					date = self.dateDetector.getTime(m, timeMatch)
-					if date is not None:
-						# Lets get the time part
-						date = date[0]
-						self.__lastTimeText = m
-						self.__lastDate = date
-					else:
-						logSys.error("findFailure failed to parse timeText: %s", m)
-				# matched empty value - date is optional or not available - set it to last known or now:
-				elif self.__lastDate and self.__lastDate > MyTime.time() - 60:
-					# set it to last known:
-					tupleLine = ("", self.__lastTimeText, line)
-					date = self.__lastDate
-				else:
-					# set it to now:
-					date = MyTime.time()
+			l = line.rstrip('\r\n')
+			logSys.log(7, "Working on line %r", line)
+
+			(timeMatch, template) = self.dateDetector.matchTime(l)
+			if timeMatch:
+				tupleLine  = (
+					l[:timeMatch.start(1)],
+					l[timeMatch.start(1):timeMatch.end(1)],
+					l[timeMatch.end(1):],
+					(timeMatch, template)
+				)
 			else:
-				tupleLine = ("", "", line)
-			# still no date - try to use last known:
-			if date is None:
-				noDate = True
-				if self.__lastDate and self.__lastDate > MyTime.time() - 60:
-					tupleLine = ("", self.__lastTimeText, line)
-					date = self.__lastDate
-		
-		if self.checkFindTime:
-			# if in operation (modifications have been really found):
-			if self.inOperation:
-				# if weird date - we'd simulate now for timeing issue (too large deviation from now):
-				if (date is None or date < MyTime.time() - 60 or date > MyTime.time() + 60):
-					# log time zone issue as warning once per day:
-					self._logWarnOnce("_next_simByTimeWarn",
-						("Simulate NOW in operation since found time has too large deviation %s ~ %s +/- %s",
-							date, MyTime.time(), 60),
-						("Please check jail has possibly a timezone issue. Line with odd timestamp: %s", 
-							line))
-					# simulate now as date:
-					date = MyTime.time()
-					self.__lastDate = date
-			else:
-				# in initialization (restore) phase, if too old - ignore:
-				if date is not None and date < MyTime.time() - self.getFindTime():
-					# log time zone issue as warning once per day:
-					self._logWarnOnce("_next_ignByTimeWarn",
-						("Ignore line since time %s < %s - %s",
-							date, MyTime.time(), self.getFindTime()),
-						("Please check jail has possibly a timezone issue. Line with odd timestamp: %s", 
-							line))
-					# ignore - too old (obsolete) entry:
-					return []
+				tupleLine = (l, "", "", None)

 		# save last line (lazy convert of process line tuple to string on demand):
 		self.processedLine = lambda: "".join(tupleLine[::2])
-		return self.findFailure(tupleLine, date, noDate=noDate)
+		return self.findFailure(tupleLine, date)

 	def processLineAndAdd(self, line, date=None):
 		"""Processes the line for failures and populates failManager
@@ -682,20 +603,16 @@ class Filter(JailThread):
 				fail = element[3]
 				logSys.debug("Processing line with time:%s and ip:%s", 
 						unixTime, ip)
-				# ensure the time is not in the future, e. g. by some estimated (assumed) time:
-				if self.checkFindTime and unixTime > MyTime.time():
-					unixTime = MyTime.time()
 				tick = FailTicket(ip, unixTime, data=fail)
 				if self._inIgnoreIPList(ip, tick):
 					continue
 				logSys.info(
-					"[%s] Found %s - %s", self.jailName, ip, datetime.datetime.fromtimestamp(unixTime).strftime("%Y-%m-%d %H:%M:%S")
+					"[%s] Found %s - %s", self.jailName, ip, MyTime.time2str(unixTime)
 				)
-				attempts = self.failManager.addFailure(tick)
-				# avoid RC on busy filter (too many failures) - if attempts for IP/ID reached maxretry,
-				# we can speedup ban, so do it as soon as possible:
-				if self.banASAP and attempts >= self.failManager.getMaxRetry():
-					self.performBan(ip)
+				self.failManager.addFailure(tick)
+				# report to observer - failure was found, for possibly increasing of it retry counter (asynchronous)
+				if Observers.Main is not None:
+					Observers.Main.add('failureFound', self.failManager, self.jail, tick)
 			# reset (halve) error counter (successfully processed line):
 			if self._errors:
 				self._errors //= 2
@@ -715,26 +632,20 @@ class Filter(JailThread):
 			self._errors //= 2
 			self.idle = True

-	def _ignoreLine(self, buf, orgBuffer, failRegex=None):
-		# if multi-line buffer - use matched only, otherwise (single line) - original buf:
-		if failRegex and self.__lineBufferSize > 1:
-			orgBuffer = failRegex.getMatchedTupleLines()
-			buf = Regex._tupleLinesBuf(orgBuffer)
-		# search ignored:
-		fnd = None
+	##
+	# Returns true if the line should be ignored.
+	#
+	# Uses ignoreregex.
+	# @param line: the line
+	# @return: a boolean
+
+	def ignoreLine(self, tupleLines):
+		buf = Regex._tupleLinesBuf(tupleLines)
 		for ignoreRegexIndex, ignoreRegex in enumerate(self.__ignoreRegex):
-			ignoreRegex.search(buf, orgBuffer)
+			ignoreRegex.search(buf, tupleLines)
 			if ignoreRegex.hasMatched():
-				fnd = ignoreRegexIndex
-				logSys.log(7, "  Matched ignoreregex %d and was ignored", fnd)
-				if self.onIgnoreRegex: self.onIgnoreRegex(fnd, ignoreRegex)
-				# remove ignored match:
-				if not self.checkAllRegex or self.__lineBufferSize > 1:
-					# todo: check ignoreRegex.getUnmatchedTupleLines() would be better (fix testGetFailuresMultiLineIgnoreRegex):
-					if failRegex:
-						self.__lineBuffer = failRegex.getUnmatchedTupleLines()
-				if not self.checkAllRegex: break
-		return fnd
+				return ignoreRegexIndex
+		return None

 	def _updateUsers(self, fail, user=()):
 		users = fail.get('users')
@@ -744,31 +655,54 @@ class Filter(JailThread):
 				fail['users'] = users = set()
 			users.add(user)
 			return users
-		return users
+		return None
+
+	# # ATM incremental (non-empty only) merge deactivated ...
+	# @staticmethod
+	# def _updateFailure(self, mlfidGroups, fail):
+	# 	# reset old failure-ids when new types of id available in this failure:
+	# 	fids = set()
+	# 	for k in ('fid', 'ip4', 'ip6', 'dns'):
+	# 		if fail.get(k):
+	# 			fids.add(k)
+	# 	if fids:
+	# 		for k in ('fid', 'ip4', 'ip6', 'dns'):
+	# 			if k not in fids:
+	# 				try:
+	# 					del mlfidGroups[k]
+	# 				except:
+	# 					pass
+	# 	# update not empty values:
+	# 	mlfidGroups.update(((k,v) for k,v in fail.iteritems() if v))

 	def _mergeFailure(self, mlfid, fail, failRegex):
 		mlfidFail = self.mlfidCache.get(mlfid) if self.__mlfidCache else None
 		users = None
 		nfflgs = 0
 		if fail.get("mlfgained"):
-			nfflgs |= (8|1)
+			nfflgs |= 9
 			if not fail.get('nofail'):
 				fail['nofail'] = fail["mlfgained"]
 		elif fail.get('nofail'): nfflgs |= 1
-		if fail.pop('mlfforget', None): nfflgs |= 2
+		if fail.get('mlfforget'): nfflgs |= 2
 		# if multi-line failure id (connection id) known:
 		if mlfidFail:
 			mlfidGroups = mlfidFail[1]
 			# update users set (hold all users of connect):
 			users = self._updateUsers(mlfidGroups, fail.get('user'))
 			# be sure we've correct current state ('nofail' and 'mlfgained' only from last failure)
-			if mlfidGroups.pop('nofail', None): nfflgs |= 4
-			if mlfidGroups.pop('mlfgained', None): nfflgs |= 4
-			# if we had no pending failures then clear the matches (they are already provided):
-			if (nfflgs & 4) == 0 and not mlfidGroups.get('mlfpending', 0):
-				mlfidGroups.pop("matches", None)
+			try:
+				del mlfidGroups['nofail']
+				del mlfidGroups['mlfgained']
+			except KeyError:
+				pass
+			# # ATM incremental (non-empty only) merge deactivated (for future version only),
+			# # it can be simulated using alternate value tags, like <F-ALT_VAL>...</F-ALT_VAL>,
+			# # so previous value 'val' will be overwritten only if 'alt_val' is not empty...		
+			# _updateFailure(mlfidGroups, fail)
+			#
 			# overwrite multi-line failure with all values, available in fail:
-			mlfidGroups.update(((k,v) for k,v in fail.iteritems() if v is not None))
+			mlfidGroups.update(fail)
 			# new merged failure data:
 			fail = mlfidGroups
 			# if forget (disconnect/reset) - remove cached entry:
@@ -779,19 +713,24 @@ class Filter(JailThread):
 			mlfidFail = [self.__lastDate, fail]
 			self.mlfidCache.set(mlfid, mlfidFail)
 		# check users in order to avoid reset failure by multiple logon-attempts:
-		if fail.pop('mlfpending', 0) or users and len(users) > 1:
-			# we've pending failures or new user, reset 'nofail' because of failures or multiple users attempts:
-			fail.pop('nofail', None)
-			fail.pop('mlfgained', None)
-			nfflgs &= ~(8|1) # reset nofail and gained
+		if users and len(users) > 1:
+			# we've new user, reset 'nofail' because of multiple users attempts:
+			try:
+				del fail['nofail']
+				nfflgs &= ~1 # reset nofail
+			except KeyError:
+				pass
 		# merge matches:
-		if (nfflgs & 1) == 0: # current nofail state (corresponding users)
-			m = fail.pop("nofail-matches", [])
-			m += fail.get("matches", [])
-			if (nfflgs & 8) == 0: # no gain signaled
+		if not (nfflgs & 1): # current nofail state (corresponding users)
+			try:
+				m = fail.pop("nofail-matches")
+				m += fail.get("matches", [])
+			except KeyError:
+				m = fail.get("matches", [])
+			if not (nfflgs & 8): # no gain signaled
 				m += failRegex.getMatchedTupleLines()
 			fail["matches"] = m
-		elif (nfflgs & 3) == 1: # not mlfforget and nofail:
+		elif not (nfflgs & 2) and (nfflgs & 1): # not mlfforget and nofail:
 			fail["nofail-matches"] = fail.get("nofail-matches", []) + failRegex.getMatchedTupleLines()
 		# return merged:
 		return fail
@@ -804,7 +743,7 @@ class Filter(JailThread):
 	# to find the logging time.
 	# @return a dict with IP and timestamp.

-	def findFailure(self, tupleLine, date, noDate=False):
+	def findFailure(self, tupleLine, date=None):
 		failList = list()

 		ll = logSys.getEffectiveLevel()
@@ -814,33 +753,62 @@ class Filter(JailThread):
 			returnRawHost = True
 			cidr = IPAddr.CIDR_RAW

-		if self.__lineBufferSize > 1:
-			self.__lineBuffer.append(tupleLine)
-			orgBuffer = self.__lineBuffer = self.__lineBuffer[-self.__lineBufferSize:]
-		else:
-			orgBuffer = self.__lineBuffer = [tupleLine]
-		if ll <= 5: logSys.log(5, "Looking for match of %r", orgBuffer)
-		buf = Regex._tupleLinesBuf(orgBuffer)
+		# Checks if we mut ignore this line.
+		if self.ignoreLine([tupleLine[::2]]) is not None:
+			# The ignoreregex matched. Return.
+			if ll <= 7: logSys.log(7, "Matched ignoreregex and was \"%s\" ignored",
+				"".join(tupleLine[::2]))
+			return failList

-		# Checks if we must ignore this line (only if fewer ignoreregex than failregex).
-		if self.__ignoreRegex and len(self.__ignoreRegex) < len(self.__failRegex) - 2:
-			if self._ignoreLine(buf, orgBuffer) is not None:
-				# The ignoreregex matched. Return.
-				return failList
+		timeText = tupleLine[1]
+		if date:
+			self.__lastTimeText = timeText
+			self.__lastDate = date
+		elif timeText:
+
+			dateTimeMatch = self.dateDetector.getTime(timeText, tupleLine[3])
+
+			if dateTimeMatch is None:
+				logSys.error("findFailure failed to parse timeText: %s", timeText)
+				date = self.__lastDate
+
+			else:
+				# Lets get the time part
+				date = dateTimeMatch[0]
+
+				self.__lastTimeText = timeText
+				self.__lastDate = date
+		else:
+			timeText = self.__lastTimeText or "".join(tupleLine[::2])
+			date = self.__lastDate
+
+		if self.checkFindTime and date is not None and date < MyTime.time() - self.getFindTime():
+			if ll <= 5: logSys.log(5, "Ignore line since time %s < %s - %s", 
+				date, MyTime.time(), self.getFindTime())
+			return failList
+
+		if self.__lineBufferSize > 1:
+			orgBuffer = self.__lineBuffer = (
+				self.__lineBuffer + [tupleLine[:3]])[-self.__lineBufferSize:]
+		else:
+			orgBuffer = self.__lineBuffer = [tupleLine[:3]]
+		if ll <= 5: logSys.log(5, "Looking for match of %r", self.__lineBuffer)
+		buf = Regex._tupleLinesBuf(self.__lineBuffer)

 		# Pre-filter fail regex (if available):
 		preGroups = {}
 		if self.__prefRegex:
 			if ll <= 5: logSys.log(5, "  Looking for prefregex %r", self.__prefRegex.getRegex())
-			self.__prefRegex.search(buf, orgBuffer)
+			self.__prefRegex.search(buf, self.__lineBuffer)
 			if not self.__prefRegex.hasMatched():
 				if ll <= 5: logSys.log(5, "  Prefregex not matched")
 				return failList
 			preGroups = self.__prefRegex.getGroups()
 			if ll <= 7: logSys.log(7, "  Pre-filter matched %s", preGroups)
-			repl = preGroups.pop('content', None)
+			repl = preGroups.get('content')
 			# Content replacement:
 			if repl:
+				del preGroups['content']
 				self.__lineBuffer, buf = [('', '', repl)], None

 		# Iterates over all the regular expressions.
@@ -858,21 +826,28 @@ class Filter(JailThread):
 				# The failregex matched.
 				if ll <= 7: logSys.log(7, "  Matched failregex %d: %s", failRegexIndex, fail)
 				# Checks if we must ignore this match.
-				if self.__ignoreRegex and self._ignoreLine(buf, orgBuffer, failRegex) is not None:
+				if self.ignoreLine(failRegex.getMatchedTupleLines()) \
+						is not None:
 					# The ignoreregex matched. Remove ignored match.
-					buf = None
+					self.__lineBuffer, buf = failRegex.getUnmatchedTupleLines(), None
+					if ll <= 7: logSys.log(7, "  Matched ignoreregex and was ignored")
 					if not self.checkAllRegex:
 						break
+					else:
+						continue
+				if date is None:
+					logSys.warning(
+						"Found a match for %r but no valid date/time "
+						"found for %r. Please try setting a custom "
+						"date pattern (see man page jail.conf(5)). "
+						"If format is complex, please "
+						"file a detailed issue on"
+						" https://github.com/fail2ban/fail2ban/issues "
+						"in order to get support for this format.",
+						 "\n".join(failRegex.getMatchedLines()), timeText)
 					continue
-				if noDate:
-					self._logWarnOnce("_next_noTimeWarn",
-						("Found a match but no valid date/time found for %r.", tupleLine[1]),
-						("Match without a timestamp: %s", "\n".join(failRegex.getMatchedLines())),
-						("Please try setting a custom date pattern (see man page jail.conf(5)).",)
-					)
-					if date is None and self.checkFindTime: continue
 				# we should check all regex (bypass on multi-line, otherwise too complex):
-				if not self.checkAllRegex or self.__lineBufferSize > 1:
+				if not self.checkAllRegex or self.getMaxLines() > 1:
 					self.__lineBuffer, buf = failRegex.getUnmatchedTupleLines(), None
 				# merge data if multi-line failure:
 				raw = returnRawHost
@@ -917,8 +892,7 @@ class Filter(JailThread):
 				if host is None:
 					if ll <= 7: logSys.log(7, "No failure-id by mlfid %r in regex %s: %s",
 						mlfid, failRegexIndex, fail.get('mlfforget', "waiting for identifier"))
-					fail['mlfpending'] = 1; # mark failure is pending
-					if not self.checkAllRegex and self.ignorePending: return failList
+					if not self.checkAllRegex: return failList
 					ips = [None]
 				# if raw - add single ip or failure-id,
 				# otherwise expand host to multiple ips using dns (or ignore it if not valid):
@@ -931,9 +905,6 @@ class Filter(JailThread):
 				# otherwise, try to use dns conversion:
 				else:
 					ips = DNSUtils.textToIp(host, self.__useDns)
-				# if checkAllRegex we must make a copy (to be sure next RE doesn't change merged/cached failure):
-				if self.checkAllRegex and mlfid is not None:
-					fail = fail.copy()
 				# append failure with match to the list:
 				for ip in ips:
 					failList.append([failRegexIndex, ip, date, fail])
@@ -979,7 +950,7 @@ class FileFilter(Filter):
 					log.setPos(lastpos)
 			self.__logs[path] = log
 			logSys.info("Added logfile: %r (pos = %s, hash = %s)" , path, log.getPos(), log.getHash())
-			if autoSeek and not tail:
+			if autoSeek:
 				self.__autoSeek[path] = autoSeek
 			self._addLogPath(path)			# backend specific

@@ -1063,7 +1034,7 @@ class FileFilter(Filter):
 	# MyTime.time()-self.findTime. When a failure is detected, a FailTicket
 	# is created and is added to the FailManager.

-	def getFailures(self, filename, inOperation=None):
+	def getFailures(self, filename):
 		log = self.getLog(filename)
 		if log is None:
 			logSys.error("Unable to get failures in %s", filename)
@@ -1108,15 +1079,10 @@ class FileFilter(Filter):
 			if has_content:
 				while not self.idle:
 					line = log.readline()
-					if not self.active: break; # jail has been stopped
-					if not line:
-						# The jail reached the bottom, simply set in operation for this log
-						# (since we are first time at end of file, growing is only possible after modifications):
-						log.inOperation = True
+					if not line or not self.active:
+						# The jail reached the bottom or has been stopped
 						break
-					# acquire in operation from log and process:
-					self.inOperation = inOperation if inOperation is not None else log.inOperation
-					self.processLineAndAdd(line.rstrip('\r\n'))
+					self.processLineAndAdd(line)
 		finally:
 			log.close()
 		db = self.jail.database
@@ -1132,7 +1098,7 @@ class FileFilter(Filter):
 		fs = container.getFileSize()
 		if logSys.getEffectiveLevel() <= logging.DEBUG:
 			logSys.debug("Seek to find time %s (%s), file size %s", date, 
-				datetime.datetime.fromtimestamp(date).strftime("%Y-%m-%d %H:%M:%S"), fs)
+				MyTime.time2str(date), fs)
 		minp = container.getPos()
 		maxp = fs
 		tryPos = minp
@@ -1211,7 +1177,7 @@ class FileFilter(Filter):
 		container.setPos(foundPos)
 		if logSys.getEffectiveLevel() <= logging.DEBUG:
 			logSys.debug("Position %s from %s, found time %s (%s) within %s seeks", lastPos, fs, foundTime, 
-				(datetime.datetime.fromtimestamp(foundTime).strftime("%Y-%m-%d %H:%M:%S") if foundTime is not None else ''), cntr)
+				(MyTime.time2str(foundTime) if foundTime is not None else ''), cntr)
 		
 	def status(self, flavor="basic"):
 		"""Status of Filter plus files being monitored.
@@ -1254,7 +1220,7 @@ except ImportError: # pragma: no cover

 class FileContainer:

-	def __init__(self, filename, encoding, tail=False):
+	def __init__(self, filename, encoding, tail = False):
 		self.__filename = filename
 		self.setEncoding(encoding)
 		self.__tail = tail
@@ -1275,8 +1241,6 @@ class FileContainer:
 				self.__pos = 0
 		finally:
 			handler.close()
-		## shows that log is in operation mode (expecting new messages only from here):
-		self.inOperation = tail

 	def getFileName(self):
 		return self.__filename
@@ -1350,17 +1314,16 @@ class FileContainer:
 			return line.decode(enc, 'strict')
 		except (UnicodeDecodeError, UnicodeEncodeError) as e:
 			global _decode_line_warn
-			lev = 7
-			if not _decode_line_warn.get(filename, 0):
+			lev = logging.DEBUG
+			if _decode_line_warn.get(filename, 0) <= MyTime.time():
 				lev = logging.WARNING
-				_decode_line_warn.set(filename, 1)
+				_decode_line_warn[filename] = MyTime.time() + 24*60*60
 			logSys.log(lev,
-				"Error decoding line from '%s' with '%s'.", filename, enc)
-			if logSys.getEffectiveLevel() <= lev:
-				logSys.log(lev, "Consider setting logencoding=utf-8 (or another appropriate"
-					" encoding) for this jail. Continuing"
-					" to process line ignoring invalid characters: %r",
-					line)
+				"Error decoding line from '%s' with '%s'."
+				" Consider setting logencoding=utf-8 (or another appropriate"
+				" encoding) for this jail. Continuing"
+				" to process line ignoring invalid characters: %r",
+				filename, enc, line)
 			# decode with replacing error chars:
 			line = line.decode(enc, 'replace')
 		return line
@@ -1381,7 +1344,7 @@ class FileContainer:
 		## print "D: Closed %s with pos %d" % (handler, self.__pos)
 		## sys.stdout.flush()

-_decode_line_warn = Utils.Cache(maxCount=1000, maxTime=24*60*60);
+_decode_line_warn = {}


 ##
@@ -79,8 +79,7 @@ class FilterGamin(FileFilter):
 		this is a common logic and must be shared/provided by FileFilter
 		"""
 		self.getFailures(path)
-		if not self.banASAP: # pragma: no cover
-			self.performBan()
+		self.performBan()
 		self.__modified = False

 	##
@@ -111,16 +111,13 @@ class FilterPoll(FileFilter):
 				modlst = []
 				Utils.wait_for(lambda: not self.active or self.getModified(modlst),
 					self.sleeptime)
-				if not self.active: # pragma: no cover - timing
-					break
 				for filename in modlst:
 					self.getFailures(filename)
 					self.__modified = True

 				self.ticks += 1
 				if self.__modified:
-					if not self.banASAP: # pragma: no cover
-						self.performBan()
+					self.performBan()
 					self.__modified = False
 			except Exception as e: # pragma: no cover
 				if not self.active: # if not active - error by stop...
@@ -142,7 +139,7 @@ class FilterPoll(FileFilter):
 		try:
 			logStats = os.stat(filename)
 			stats = logStats.st_mtime, logStats.st_ino, logStats.st_size
-			pstats = self.__prevStats.get(filename, (0,))
+			pstats = self.__prevStats.get(filename, (0))
 			if logSys.getEffectiveLevel() <= 4:
 				# we do not want to waste time on strftime etc if not necessary
 				dt = logStats.st_mtime - pstats[0]
@@ -140,8 +140,7 @@ class FilterPyinotify(FileFilter):
 		"""
 		if not self.idle:
 			self.getFailures(path)
-			if not self.banASAP: # pragma: no cover
-				self.performBan()
+			self.performBan()
 			self.__modified = False

 	def _addPending(self, path, reason, isDir=False):
@@ -188,8 +187,7 @@ class FilterPyinotify(FileFilter):
 		for path, isDir in found.iteritems():
 			self._delPending(path)
 			# refresh monitoring of this:
-			if isDir is not None:
-				self._refreshWatcher(path, isDir=isDir)
+			self._refreshWatcher(path, isDir=isDir)
 			if isDir:
 				# check all files belong to this dir:
 				for logpath in self.__watchFiles:
@@ -272,13 +270,7 @@ class FilterPyinotify(FileFilter):

 	def _addLogPath(self, path):
 		self._addFileWatcher(path)
-		# initial scan:
-		if self.active:
-			# we can execute it right now:
-			self._process_file(path)
-		else:
-			# retard until filter gets started, isDir=None signals special case: process file only (don't need to refresh monitor):
-			self._addPending(path, ('INITIAL', path), isDir=None)
+		self._process_file(path)

    ##
 	# Delete a log path
@@ -286,9 +278,9 @@ class FilterPyinotify(FileFilter):
 	# @param path the log file to delete

 	def _delLogPath(self, path):
-		self._delPending(path)
 		if not self._delFileWatcher(path): # pragma: no cover
 			logSys.error("Failed to remove watch on path: %s", path)
+		self._delPending(path)

 		path_dir = dirname(path)
 		for k in self.__watchFiles:
@@ -298,8 +290,8 @@ class FilterPyinotify(FileFilter):
 		if path_dir:
 			# Remove watches for the directory
 			# since there is no other monitored file under this directory
-			self._delPending(path_dir)
 			self._delDirWatcher(path_dir)
+			self._delPending(path_dir)

 	# pyinotify.ProcessEvent default handler:
 	def __process_default(self, event):
@@ -190,13 +190,6 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 	def getJournalReader(self):
 		return self.__journal

-	def getJrnEntTime(self, logentry):
-		""" Returns time of entry as tuple (ISO-str, Posix)."""
-		date = logentry.get('_SOURCE_REALTIME_TIMESTAMP')
-		if date is None:
-				date = logentry.get('__REALTIME_TIMESTAMP')
-		return (date.isoformat(), time.mktime(date.timetuple()) + date.microsecond/1.0E6)
-
 	##
 	# Format journal log entry into syslog style
 	#
@@ -229,8 +222,9 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 				logelements[-1] += v
 			logelements[-1] += ":"
 			if logelements[-1] == "kernel:":
-				monotonic = logentry.get('_SOURCE_MONOTONIC_TIMESTAMP')
-				if monotonic is None:
+				if '_SOURCE_MONOTONIC_TIMESTAMP' in logentry:
+					monotonic = logentry.get('_SOURCE_MONOTONIC_TIMESTAMP')
+				else:
 					monotonic = logentry.get('__MONOTONIC_TIMESTAMP')[0]
 				logelements.append("[%12.6f]" % monotonic.total_seconds())
 		msg = logentry.get('MESSAGE','')
@@ -241,11 +235,13 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover

 		logline = " ".join(logelements)

-		date = self.getJrnEntTime(logentry)
+		date = logentry.get('_SOURCE_REALTIME_TIMESTAMP',
+				logentry.get('__REALTIME_TIMESTAMP'))
 		logSys.log(5, "[%s] Read systemd journal entry: %s %s", self.jailName,
-			date[0], logline)
+			date.isoformat(), logline)
 		## use the same type for 1st argument:
-		return ((logline[:0], date[0], logline.replace('\n', '\\n')), date[1])
+		return ((logline[:0], date.isoformat(), logline.replace('\n', '\\n')),
+			time.mktime(date.timetuple()) + date.microsecond/1.0E6)

 	def seekToTime(self, date):
 		if not isinstance(date, datetime.datetime):
@@ -266,12 +262,9 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 				"Jail regexs will be checked against all journal entries, "
 				"which is not advised for performance reasons.")

-		# Try to obtain the last known time (position of journal)
-		start_time = 0
-		if self.jail.database is not None:
-			start_time = self.jail.database.getJournalPos(self.jail, 'systemd-journal') or 0
-		# Seek to max(last_known_time, now - findtime) in journal
-		start_time = max( start_time, MyTime.time() - int(self.getFindTime()) )
+		# Seek to now - findtime in journal
+		start_time = datetime.datetime.now() - \
+				datetime.timedelta(seconds=int(self.getFindTime()))
 		self.seekToTime(start_time)
 		# Move back one entry to ensure do not end up in dead space
 		# if start time beyond end of journal
@@ -310,20 +303,16 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover
 							e, exc_info=logSys.getEffectiveLevel() <= logging.DEBUG)
 					self.ticks += 1
 					if logentry:
-						line = self.formatJournalEntry(logentry)
-						self.processLineAndAdd(*line)
+						self.processLineAndAdd(
+							*self.formatJournalEntry(logentry))
 						self.__modified += 1
 						if self.__modified >= 100: # todo: should be configurable
 							break
 					else:
 						break
 				if self.__modified:
-					if not self.banASAP: # pragma: no cover
-						self.performBan()
+					self.performBan()
 					self.__modified = 0
-					# update position in log (time and iso string):
-					if self.jail.database is not None:
-						self.jail.database.updateJournal(self.jail, 'systemd-journal', line[1], line[0][1])
 			except Exception as e: # pragma: no cover
 				if not self.active: # if not active - error by stop...
 					break
@@ -379,12 +379,6 @@ class IPAddr(object):
 		"""
 		return self._family != socket.AF_UNSPEC

-	@property
-	def isSingle(self):
-		"""Returns whether the object is a single IP address (not DNS and subnet)
-		"""
-		return self._plen == {socket.AF_INET: 32, socket.AF_INET6: 128}.get(self._family, -1000)
-
 	def __eq__(self, other):
 		if self._family == IPAddr.CIDR_RAW and not isinstance(other, IPAddr):
 			return self._raw == other
@@ -517,11 +511,6 @@ class IPAddr(object):
 		
 		return (self.addr & mask) == net.addr

-	def contains(self, ip):
-		"""Return whether the object (as network) contains given IP
-		"""
-		return isinstance(ip, IPAddr) and (ip == self or ip.isInNet(self))
-
 	# Pre-calculated map: addr to maskplen
 	def __getMaskMap():
 		m6 = (1 << 128)-1
@@ -24,10 +24,13 @@ __copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2011-2012 Lee Clemens, 2012 Y
 __license__ = "GPL"

 import logging
+import math
+import random
 import Queue

 from .actions import Actions
-from ..helpers import getLogger, extractOptions, MyTime
+from ..helpers import getLogger, _as_bool, extractOptions, MyTime
+from .mytime import MyTime

 # Gets the instance of the logger.
 logSys = getLogger(__name__)
@@ -75,6 +78,8 @@ class Jail(object):
 		self.__name = name
 		self.__queue = Queue.Queue()
 		self.__filter = None
+		# Extra parameters for increase ban time
+		self._banExtra = {};
 		logSys.info("Creating new jail '%s'" % self.name)
 		if backend is not None:
 			self._setBackend(backend)
@@ -156,10 +161,6 @@ class Jail(object):
 		"""
 		return self.__db

-	@database.setter
-	def database(self, value):
-		self.__db = value;
-
 	@property
 	def filter(self):
 		"""The filter which the jail is using to monitor log files.
@@ -191,18 +192,14 @@ class Jail(object):
 			("Actions", self.actions.status(flavor=flavor)),
 			]

-	@property
-	def hasFailTickets(self):
-		"""Retrieve whether queue has tickets to ban.
-		"""
-		return not self.__queue.empty()
-
 	def putFailTicket(self, ticket):
 		"""Add a fail ticket to the jail.

 		Used by filter to add a failure for banning.
 		"""
 		self.__queue.put(ticket)
+		# add ban to database moved to observer (should previously check not already banned 
+		# and increase ticket time if "bantime.increment" set)

 	def getFailTicket(self):
 		"""Get a fail ticket from the jail.
@@ -215,16 +212,80 @@ class Jail(object):
 		except Queue.Empty:
 			return False

-	def restoreCurrentBans(self):
+	def setBanTimeExtra(self, opt, value):
+		# merge previous extra with new option:
+		be = self._banExtra;
+		if value == '':
+			value = None
+		if value is not None:
+			be[opt] = value;
+		elif opt in be:
+			del be[opt]
+		logSys.info('Set banTime.%s = %s', opt, value)
+		if opt == 'increment':
+			be[opt] = _as_bool(value)
+			if be.get(opt) and self.database is None:
+				logSys.warning("ban time increment is not available as long jail database is not set")
+		if opt in ['maxtime', 'rndtime']:
+			if not value is None:
+				be[opt] = MyTime.str2seconds(value)
+		# prepare formula lambda:
+		if opt in ['formula', 'factor', 'maxtime', 'rndtime', 'multipliers'] or be.get('evformula', None) is None:
+			# split multifiers to an array begins with 0 (or empty if not set):
+			if opt == 'multipliers':
+				be['evmultipliers'] = [int(i) for i in (value.split(' ') if value is not None and value != '' else [])]
+			# if we have multifiers - use it in lambda, otherwise compile and use formula within lambda
+			multipliers = be.get('evmultipliers', [])
+			banFactor = eval(be.get('factor', "1"))
+			if len(multipliers):
+				evformula = lambda ban, banFactor=banFactor: (
+					ban.Time * banFactor * multipliers[ban.Count if ban.Count < len(multipliers) else -1]
+				)
+			else:
+				formula = be.get('formula', 'ban.Time * (1<<(ban.Count if ban.Count<20 else 20)) * banFactor')
+				formula = compile(formula, '~inline-conf-expr~', 'eval')
+				evformula = lambda ban, banFactor=banFactor, formula=formula: max(ban.Time, eval(formula))
+			# extend lambda with max time :
+			if not be.get('maxtime', None) is None:
+				maxtime = be['maxtime']
+				evformula = lambda ban, evformula=evformula: min(evformula(ban), maxtime)
+			# mix lambda with random time (to prevent bot-nets to calculate exact time IP can be unbanned):
+			if not be.get('rndtime', None) is None:
+				rndtime = be['rndtime']
+				evformula = lambda ban, evformula=evformula: (evformula(ban) + random.random() * rndtime)
+			# set to extra dict:
+			be['evformula'] = evformula
+		#logSys.info('banTimeExtra : %s' % json.dumps(be))
+
+	def getBanTimeExtra(self, opt=None):
+		if opt is not None:
+			return self._banExtra.get(opt, None)
+		return self._banExtra
+
+	def getMaxBanTime(self):
+		"""Returns max possible ban-time of jail.
+		"""
+		return self._banExtra.get("maxtime", -1) \
+			if self._banExtra.get('increment') else self.actions.getBanTime()
+
+	def restoreCurrentBans(self, correctBanTime=True):
 		"""Restore any previous valid bans from the database.
 		"""
 		try:
 			if self.database is not None:
-				forbantime = self.actions.getBanTime()
-				for ticket in self.database.getCurrentBans(jail=self, 
-						forbantime=forbantime, maxmatches=self.filter.failManager.maxMatches):
-					#logSys.debug('restored ticket: %s', ticket)
-					if not self.filter.inIgnoreIPList(ticket.getIP(), log_ignore=True):
+				if self._banExtra.get('increment'):
+					forbantime = None;
+					if correctBanTime:
+						correctBanTime = self.getMaxBanTime()
+				else:
+					# use ban time as search time if we have not enabled a increasing:
+					forbantime = self.actions.getBanTime()
+				for ticket in self.database.getCurrentBans(jail=self, forbantime=forbantime,
+					correctBanTime=correctBanTime, maxmatches=self.filter.failManager.maxMatches
+				):
+					try:
+						#logSys.debug('restored ticket: %s', ticket)
+						if self.filter.inIgnoreIPList(ticket.getIP(), log_ignore=True): continue
 						# mark ticked was restored from database - does not put it again into db:
 						ticket.restored = True
 						# correct start time / ban time (by the same end of ban):
@@ -235,11 +296,13 @@ class Jail(object):
 						# ignore obsolete tickets:
 						if btm != -1 and btm <= 0:
 							continue
-						ticket.setTime(MyTime.time())
-						ticket.setBanTime(btm)
 						self.putFailTicket(ticket)
+					except Exception as e: # pragma: no cover
+						logSys.error('Restore ticket failed: %s', e, 
+							exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
 		except Exception as e: # pragma: no cover
-			logSys.error('%s', e, exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
+			logSys.error('Restore bans failed: %s', e,
+				exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)

 	def start(self):
 		"""Start the jail, by starting filter and actions threads.
@@ -120,6 +120,3 @@ class JailThread(Thread):
 ## python 2.x replace binding of private __bootstrap method:
 if sys.version_info < (3,): # pragma: 3.x no cover
 	JailThread._Thread__bootstrap = JailThread._JailThread__bootstrap
-## python 3.9, restore isAlive method:
-elif not hasattr(JailThread, 'isAlive'): # pragma: 2.x no cover
-	 JailThread.isAlive = JailThread.is_alive
@@ -113,6 +113,16 @@ class MyTime:
 			return time.localtime(x)
 		else:
 			return time.localtime(MyTime.myTime)
+	
+	@staticmethod
+	def time2str(unixTime, format="%Y-%m-%d %H:%M:%S"):
+		"""Convert time to a string representing as date and time using given format.
+		Default format is ISO 8601, YYYY-MM-DD HH:MM:SS without microseconds.
+
+		@return ISO-capable string representation of given unixTime
+		"""
+		return datetime.datetime.fromtimestamp(
+			unixTime).replace(microsecond=0).strftime(format)

 	## precreate/precompile primitives used in str2seconds:

@@ -0,0 +1,529 @@
+# emacs: -*- mode: python; py-indent-offset: 4; indent-tabs-mode: t -*-
+# vi: set ft=python sts=4 ts=4 sw=4 noet :
+
+# This file is part of Fail2Ban.
+#
+# Fail2Ban is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# Fail2Ban is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Fail2Ban; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+
+# Author: Serg G. Brester (sebres)
+# 
+# This module was written as part of ban time increment feature.
+
+__author__ = "Serg G. Brester (sebres)"
+__copyright__ = "Copyright (c) 2014 Serg G. Brester"
+__license__ = "GPL"
+
+import threading
+from .jailthread import JailThread
+from .failmanager import FailManagerEmpty
+import os, logging, time, datetime, math, json, random
+import sys
+from ..helpers import getLogger
+from .mytime import MyTime
+from .utils import Utils
+
+# Gets the instance of the logger.
+logSys = getLogger(__name__)
+
+class ObserverThread(JailThread):
+	"""Handles observing a database, managing bad ips and ban increment.
+
+	Parameters
+	----------
+
+	Attributes
+	----------
+	daemon
+	ident
+	name
+	status
+	active : bool
+		Control the state of the thread.
+	idle : bool
+		Control the idle state of the thread.
+	sleeptime : int
+		The time the thread sleeps for in the loop.
+	"""
+
+	# observer is event driven and it sleep organized incremental, so sleep intervals can be shortly:
+	DEFAULT_SLEEP_INTERVAL = Utils.DEFAULT_SLEEP_INTERVAL / 10
+
+	def __init__(self):
+    # init thread
+		super(ObserverThread, self).__init__(name='f2b/observer')
+		# before started - idle:
+		self.idle = True
+		## Event queue
+		self._queue_lock = threading.RLock()
+		self._queue = []
+		## Event, be notified if anything added to event queue
+		self._notify = threading.Event()
+		## Sleep for max 60 seconds, it possible to specify infinite to always sleep up to notifying via event, 
+		## but so we can later do some service "events" occurred infrequently directly in main loop of observer (not using queue)
+		self.sleeptime = 60
+		#
+		self._timers = {}
+		self._paused = False
+		self.__db = None
+		self.__db_purge_interval = 60*60
+		# observer is a not main thread:
+		self.daemon = True
+
+	def __getitem__(self, i):
+		try:
+			return self._queue[i]
+		except KeyError:
+			raise KeyError("Invalid event index : %s" % i)
+
+	def __delitem__(self, name):
+		try:
+			del self._queue[i]
+		except KeyError:
+			raise KeyError("Invalid event index: %s" % i)
+
+	def __iter__(self):
+		return iter(self._queue)
+
+	def __len__(self):
+		return len(self._queue)
+
+	def __eq__(self, other): # Required for Threading
+		return False
+
+	def __hash__(self): # Required for Threading
+		return id(self)
+
+	def add_named_timer(self, name, starttime, *event):
+		"""Add a named timer event to queue will start (and wake) in 'starttime' seconds
+		
+		Previous timer event with same name will be canceled and trigger self into 
+		queue after new 'starttime' value
+		"""
+		t = self._timers.get(name, None)
+		if t is not None:
+			t.cancel()
+		t = threading.Timer(starttime, self.add, event)
+		self._timers[name] = t
+		t.start()
+
+	def add_timer(self, starttime, *event):
+		"""Add a timer event to queue will start (and wake) in 'starttime' seconds
+		"""
+		# in testing we should wait (looping) for the possible time drifts:
+		if MyTime.myTime is not None and starttime:
+			# test time after short sleep:
+			t = threading.Timer(Utils.DEFAULT_SLEEP_INTERVAL, self._delayedEvent,
+				(MyTime.time() + starttime, time.time() + starttime, event)
+			)
+			t.start()
+			return
+		# add timer event:
+		t = threading.Timer(starttime, self.add, event)
+		t.start()
+
+	def _delayedEvent(self, endMyTime, endTime, event):
+		if MyTime.time() >= endMyTime or time.time() >= endTime:
+			self.add_timer(0, *event)
+			return
+		# repeat after short sleep:
+		t = threading.Timer(Utils.DEFAULT_SLEEP_INTERVAL, self._delayedEvent,
+			(endMyTime, endTime, event)
+		)
+		t.start()
+
+	def pulse_notify(self):
+		"""Notify wakeup (sets /and resets/ notify event)
+		"""
+		if not self._paused and self._notify:
+			self._notify.set()
+			#self._notify.clear()
+
+	def add(self, *event):
+		"""Add a event to queue and notify thread to wake up.
+		"""
+		## lock and add new event to queue:
+		with self._queue_lock:
+			self._queue.append(event)
+		self.pulse_notify()
+
+	def add_wn(self, *event):
+		"""Add a event to queue withouth notifying thread to wake up.
+		"""
+		## lock and add new event to queue:
+		with self._queue_lock:
+			self._queue.append(event)
+
+	def call_lambda(self, l, *args):
+		l(*args)
+
+	def run(self):
+		"""Main loop for Threading.
+
+		This function is the main loop of the thread.
+
+		Returns
+		-------
+		bool
+			True when the thread exits nicely.
+		"""
+		logSys.info("Observer start...")
+		## first time create named timer to purge database each hour (clean old entries) ...
+		self.add_named_timer('DB_PURGE', self.__db_purge_interval, 'db_purge')
+		## Mapping of all possible event types of observer:
+		__meth = {
+			# universal lambda:
+			'call': self.call_lambda,
+			# system and service events:
+			'db_set': self.db_set,
+			'db_purge': self.db_purge,
+			# service events of observer self:
+			'is_alive' : self.isAlive,
+			'is_active': self.isActive,
+			'start': self.start,
+			'stop': self.stop,
+			'nop': lambda:(),
+			'shutdown': lambda:()
+		}
+		try:
+			## check it self with sending is_alive event
+			self.add('is_alive')
+			## if we should stop - break a main loop
+			while self.active:
+				self.idle = False
+				## check events available and execute all events from queue
+				while not self._paused:
+					## lock, check and pop one from begin of queue:
+					try:
+						ev = None
+						with self._queue_lock:
+							if len(self._queue):
+								ev = self._queue.pop(0)
+						if ev is None:
+							break
+						## retrieve method by name
+						meth = ev[0]
+						if not callable(ev[0]): meth = __meth.get(meth) or getattr(self, meth)
+						## execute it with rest of event as variable arguments
+						meth(*ev[1:])
+					except Exception as e:
+						#logSys.error('%s', e, exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
+						logSys.error('%s', e, exc_info=True)
+				## going sleep, wait for events (in queue)
+				n = self._notify
+				if n:
+					self.idle = True
+					n.wait(self.sleeptime)
+					## wake up - reset signal now (we don't need it so long as we reed from queue)
+					n.clear()
+					if self._paused:
+						continue
+				else:
+					## notify event deleted (shutdown) - just sleep a litle bit (waiting for shutdown events, prevent high cpu usage)
+					time.sleep(ObserverThread.DEFAULT_SLEEP_INTERVAL)
+					## stop by shutdown and empty queue :
+					if not self.is_full:
+						break
+				## end of main loop - exit
+			logSys.info("Observer stopped, %s events remaining.", len(self._queue))
+			#print("Observer stopped, %s events remaining." % len(self._queue))
+		except Exception as e:
+			logSys.error('Observer stopped after error: %s', e, exc_info=True)
+			#print("Observer stopped with error: %s" % str(e))
+		# clear all events - exit, for possible calls of wait_empty:
+		with self._queue_lock:
+			self._queue = []
+		self.idle = True
+		return True
+
+	def isAlive(self):
+		#logSys.debug("Observer alive...")
+		return True
+
+	def isActive(self, fromStr=None):
+		# logSys.info("Observer alive, %s%s", 
+		# 	'active' if self.active else 'inactive', 
+		# 	'' if fromStr is None else (", called from '%s'" % fromStr))
+		return self.active
+
+	def start(self):
+		with self._queue_lock:
+			if not self.active:
+				super(ObserverThread, self).start()
+
+	def stop(self):
+		if self.active and self._notify:
+			wtime = 5
+			logSys.info("Observer stop ... try to end queue %s seconds", wtime)
+			#print("Observer stop ....")
+			# just add shutdown job to make possible wait later until full (events remaining)
+			with self._queue_lock:
+				self.add_wn('shutdown')
+				#don't pulse - just set, because we will delete it hereafter (sometimes not wakeup)
+				n = self._notify
+				self._notify.set()
+				#self.pulse_notify()
+				self._notify = None
+			# wait max wtime seconds until full (events remaining)
+			self.wait_empty(wtime)
+			n.clear()
+			self.active = False
+			self.wait_idle(0.5)
+
+	@property
+	def is_full(self):
+		with self._queue_lock:
+			return True if len(self._queue) else False
+
+	def wait_empty(self, sleeptime=None):
+		"""Wait observer is running and returns if observer has no more events (queue is empty)
+		"""
+		time.sleep(ObserverThread.DEFAULT_SLEEP_INTERVAL)
+		if sleeptime is not None:
+			e = MyTime.time() + sleeptime
+		# block queue with not operation to be sure all really jobs are executed if nop goes from queue :
+		if self._notify is not None:
+			self.add_wn('nop')
+			if self.is_full and self.idle:
+				self.pulse_notify()
+		while self.is_full:
+			if sleeptime is not None and MyTime.time() > e:
+				break
+			time.sleep(ObserverThread.DEFAULT_SLEEP_INTERVAL)
+		# wait idle to be sure the last queue element is processed (because pop event before processing it) :
+		self.wait_idle(0.001)
+		return not self.is_full
+
+
+	def wait_idle(self, sleeptime=None):
+		"""Wait observer is running and returns if observer idle (observer sleeps)
+		"""
+		time.sleep(ObserverThread.DEFAULT_SLEEP_INTERVAL)
+		if self.idle:
+			return True
+		if sleeptime is not None:
+			e = MyTime.time() + sleeptime
+		while not self.idle:
+			if sleeptime is not None and MyTime.time() > e:
+				break
+			time.sleep(ObserverThread.DEFAULT_SLEEP_INTERVAL)
+		return self.idle
+
+	@property
+	def paused(self):
+		return self._paused;
+
+	@paused.setter
+	def paused(self, pause):
+		if self._paused == pause:
+			return
+		self._paused = pause
+		# wake after pause ended
+		self.pulse_notify()
+
+
+	@property
+	def status(self):
+		"""Status of observer to be implemented. [TODO]
+		"""
+		return ('', '')
+
+	## -----------------------------------------
+	## [Async] database service functionality ...
+	## -----------------------------------------
+
+	def db_set(self, db):
+		self.__db = db
+
+	def db_purge(self):
+		logSys.debug("Purge database event occurred")
+		if self.__db is not None:
+			self.__db.purge()
+		# trigger timer again ...
+		self.add_named_timer('DB_PURGE', self.__db_purge_interval, 'db_purge')
+
+	## -----------------------------------------
+	## [Async] ban time increment functionality ...
+	## -----------------------------------------
+
+	def failureFound(self, failManager, jail, ticket):
+		""" Notify observer a failure for ip was found
+
+		Observer will check ip was known (bad) and possibly increase an retry count
+		"""
+		# check jail active :
+		if not jail.isAlive() or not jail.getBanTimeExtra("increment"):
+			return
+		ip = ticket.getIP()
+		unixTime = ticket.getTime()
+		logSys.debug("[%s] Observer: failure found %s", jail.name, ip)
+		# increase retry count for known (bad) ip, corresponding banCount of it (one try will count than 2, 3, 5, 9 ...)  :
+		banCount = 0
+		retryCount = 1
+		timeOfBan = None
+		try:
+			maxRetry = failManager.getMaxRetry()
+			db = jail.database
+			if db is not None:
+				for banCount, timeOfBan, lastBanTime in db.getBan(ip, jail):
+					banCount = max(banCount, ticket.getBanCount())
+					retryCount = ((1 << (banCount if banCount < 20 else 20))/2 + 1)
+					# if lastBanTime == -1 or timeOfBan + lastBanTime * 2 > MyTime.time():
+					# 	retryCount = maxRetry
+					break
+				retryCount = min(retryCount, maxRetry)
+				# check this ticket already known (line was already processed and in the database and will be restored from there):
+				if timeOfBan is not None and unixTime <= timeOfBan:
+					logSys.debug("[%s] Ignore failure %s before last ban %s < %s, restored",
+								jail.name, ip, unixTime, timeOfBan)
+					return
+			# for not increased failures observer should not add it to fail manager, because was already added by filter self
+			if retryCount <= 1:
+				return
+			# retry counter was increased - add it again:
+			logSys.info("[%s] Found %s, bad - %s, %s # -> %s%s", jail.name, ip, 
+				MyTime.time2str(unixTime), banCount, retryCount,
+				(', Ban' if retryCount >= maxRetry else ''))
+			# retryCount-1, because a ticket was already once incremented by filter self
+			retryCount = failManager.addFailure(ticket, retryCount - 1, True)
+			ticket.setBanCount(banCount)
+			# after observe we have increased attempt count, compare it >= maxretry ...
+			if retryCount >= maxRetry:
+				# perform the banning of the IP now (again)
+				# [todo]: this code part will be used multiple times - optimize it later.
+				try: # pragma: no branch - exception is the only way out
+					while True:
+						ticket = failManager.toBan(ip)
+						jail.putFailTicket(ticket)
+				except FailManagerEmpty:
+					failManager.cleanup(MyTime.time())
+
+		except Exception as e:
+			logSys.error('%s', e, exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
+
+
+	class BanTimeIncr:
+		def __init__(self, banTime, banCount):
+			self.Time = banTime
+			self.Count = banCount
+
+	def calcBanTime(self, jail, banTime, banCount):
+		be = jail.getBanTimeExtra()
+		return be['evformula'](self.BanTimeIncr(banTime, banCount))
+
+	def incrBanTime(self, jail, banTime, ticket):
+		"""Check for IP address to increment ban time (if was already banned).
+
+		Returns
+		-------
+		float
+			new ban time.
+		"""
+		# check jail active :
+		if not jail.isAlive() or not jail.database:
+			return banTime
+		be = jail.getBanTimeExtra()
+		ip = ticket.getIP()
+		orgBanTime = banTime
+		# check ip was already banned (increment time of ban):
+		try:
+			if banTime > 0 and be.get('increment', False):
+				# search IP in database and increase time if found:
+				for banCount, timeOfBan, lastBanTime in \
+					jail.database.getBan(ip, jail, overalljails=be.get('overalljails', False)) \
+				:
+					# increment count in ticket (if still not increased from banmanager, test-cases?):
+					if banCount >= ticket.getBanCount():
+						ticket.setBanCount(banCount+1)
+					logSys.debug('IP %s was already banned: %s #, %s', ip, banCount, timeOfBan);
+					# calculate new ban time
+					if banCount > 0:
+						banTime = be['evformula'](self.BanTimeIncr(banTime, banCount))
+					ticket.setBanTime(banTime)
+					# check current ticket time to prevent increasing for twice read tickets (restored from log file besides database after restart)
+					if ticket.getTime() > timeOfBan:
+						logSys.info('[%s] IP %s is bad: %s # last %s - incr %s to %s' % (jail.name, ip, banCount, 
+							MyTime.time2str(timeOfBan), 
+							datetime.timedelta(seconds=int(orgBanTime)), datetime.timedelta(seconds=int(banTime))));
+					else:
+						ticket.restored = True
+					break
+		except Exception as e:
+			logSys.error('%s', e, exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
+		return banTime
+
+	def banFound(self, ticket, jail, btime):
+		""" Notify observer a ban occured for ip
+
+		Observer will check ip was known (bad) and possibly increase/prolong a ban time
+		Secondary we will actualize the bans and bips (bad ip) in database
+		"""
+		if ticket.restored: # pragma: no cover (normally not resored tickets only)
+			return
+		try:
+			oldbtime = btime
+			ip = ticket.getIP()
+			logSys.debug("[%s] Observer: ban found %s, %s", jail.name, ip, btime)
+			# if not permanent and ban time was not set - check time should be increased:
+			if btime != -1 and ticket.getBanTime() is None:
+				btime = self.incrBanTime(jail, btime, ticket)
+				# if we should prolong ban time:
+				if btime == -1 or btime > oldbtime:
+					ticket.setBanTime(btime)
+			# if not permanent
+			if btime != -1:
+				bendtime = ticket.getTime() + btime
+				logtime = (datetime.timedelta(seconds=int(btime)),
+					MyTime.time2str(bendtime))
+				# check ban is not too old :
+				if bendtime < MyTime.time():
+					logSys.debug('Ignore old bantime %s', logtime[1])
+					return False
+			else:
+				logtime = ('permanent', 'infinite')
+			# if ban time was prolonged - log again with new ban time:
+			if btime != oldbtime:
+				logSys.notice("[%s] Increase Ban %s (%d # %s -> %s)", jail.name, 
+					ip, ticket.getBanCount(), *logtime)
+				# delayed prolonging ticket via actions that expected this (not later than 10 sec):
+				logSys.log(5, "[%s] Observer: prolong %s in %s", jail.name, ip, (btime, oldbtime))
+				self.add_timer(min(10, max(0, btime - oldbtime - 5)), self.prolongBan, ticket, jail)
+			# add ticket to database, but only if was not restored (not already read from database):
+			if jail.database is not None and not ticket.restored:
+				# add to database always only after ban time was calculated an not yet already banned:
+				jail.database.addBan(jail, ticket)
+		except Exception as e:
+			logSys.error('%s', e, exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
+
+	def prolongBan(self, ticket, jail):
+		""" Notify observer a ban occured for ip
+
+		Observer will check ip was known (bad) and possibly increase/prolong a ban time
+		Secondary we will actualize the bans and bips (bad ip) in database
+		"""
+		try:
+			btime = ticket.getBanTime()
+			ip = ticket.getIP()
+			logSys.debug("[%s] Observer: prolong %s, %s", jail.name, ip, btime)
+			# prolong ticket via actions that expected this:
+			jail.actions._prolongBan(ticket)
+		except Exception as e:
+			logSys.error('%s', e, exc_info=logSys.getEffectiveLevel()<=logging.DEBUG)
+
+# Global observer initial created in server (could be later rewriten via singleton)
+class _Observers:
+	def __init__(self):
+		self.Main = None
+
+Observers = _Observers()
--- a/Show More
+++ b/Show More