Merge branch 'bf-common-zzz'

* bf-common-zzz: BF: do not rely on long relative path to upstairs config - symlink common.conf
Added missing files to MANIFEST
2016-07-14 22:25:07 -04:00 · 2016-07-14 22:19:10 -04:00 · 2016-07-14 22:15:22 -04:00 · 2016-07-14 21:35:49 -04:00 · 2016-07-14 21:31:18 -04:00 · 2016-07-14 21:23:09 -04:00
38 changed files with 1170 additions and 888 deletions
@@ -16,6 +16,8 @@ config/action.d/firewallcmd-allports.conf
 config/action.d/firewallcmd-ipset.conf
 config/action.d/firewallcmd-multiport.conf
 config/action.d/firewallcmd-new.conf
+config/action.d/firewallcmd-rich-logging.conf
+config/action.d/firewallcmd-rich-rules.conf
 config/action.d/hostsdeny.conf
 config/action.d/ipfilter.conf
 config/action.d/ipfw.conf
@@ -124,6 +126,7 @@ config/filter.d/sendmail-auth.conf
 config/filter.d/sendmail-reject.conf
 config/filter.d/sendmail-spam.conf
 config/filter.d/sieve.conf
+config/filter.d/slapd.conf
 config/filter.d/sogo-auth.conf
 config/filter.d/solid-pop3d.conf
 config/filter.d/squid.conf
@@ -205,6 +208,7 @@ fail2ban/tests/config/fail2ban.conf
 fail2ban/tests/config/filter.d/simple.conf
 fail2ban/tests/config/filter.d/test.conf
 fail2ban/tests/config/filter.d/test.local
+fail2ban/tests/config/filter.d/zzz-generic-example.conf
 fail2ban/tests/config/jail.conf
 fail2ban/tests/config/paths-common.conf
 fail2ban/tests/config/paths-debian.conf
@@ -306,6 +310,7 @@ fail2ban/tests/files/logs/sendmail-auth
 fail2ban/tests/files/logs/sendmail-reject
 fail2ban/tests/files/logs/sendmail-spam
 fail2ban/tests/files/logs/sieve
+fail2ban/tests/files/logs/slapd
 fail2ban/tests/files/logs/sogo-auth
 fail2ban/tests/files/logs/solid-pop3d
 fail2ban/tests/files/logs/squid
@@ -320,6 +325,7 @@ fail2ban/tests/files/logs/vsftpd
 fail2ban/tests/files/logs/webmin-auth
 fail2ban/tests/files/logs/wuftpd
 fail2ban/tests/files/logs/xinetd-fail
+fail2ban/tests/files/logs/zzz-generic-example
 fail2ban/tests/files/testcase01.log
 fail2ban/tests/files/testcase02.log
 fail2ban/tests/files/testcase03.log
@@ -2,7 +2,7 @@
                        / _|__ _(_) |_  ) |__  __ _ _ _  
                       |  _/ _` | | |/ /| '_ \/ _` | ' \ 
                       |_| \__,_|_|_/___|_.__/\__,_|_||_|
-                       v0.9.4.dev0             2016/??/??
+                       v0.9.5                  2016/07/15

 ## Fail2Ban: ban hosts that cause multiple authentication errors

@@ -39,8 +39,8 @@ Optional:

 To install, just do:

-    tar xvfj fail2ban-0.9.4.tar.bz2
-    cd fail2ban-0.9.4
+    tar xvfj fail2ban-0.9.5.tar.bz2
+    cd fail2ban-0.9.5
    python setup.py install

 This will install Fail2Ban into the python library directory. The executable
@@ -53,7 +53,7 @@ Preparation

  or an alternative for comparison with previous release

-     git diff 0.9.4 | grep -B2 'index 0000000..' | grep -B1 'new file mode' | sed -n -e '/^diff /s,.* b/,,gp' >> MANIFEST
+     git diff 0.9.5 | grep -B2 'index 0000000..' | grep -B1 'new file mode' | sed -n -e '/^diff /s,.* b/,,gp' >> MANIFEST
     sort MANIFEST | uniq | sponge MANIFEST

 * Run::
@@ -66,24 +66,24 @@ Preparation

  * Which indicates that testcases/files/logs/mysqld.log has been moved or is a directory::

-      tar -C /tmp -jxf dist/fail2ban-0.9.4.tar.bz2
+      tar -C /tmp -jxf dist/fail2ban-0.9.6.tar.bz2

-* clean up current direcory::
+* clean up current directory::

-    diff -rul --exclude \*.pyc . /tmp/fail2ban-0.9.4/
+    diff -rul --exclude \*.pyc . /tmp/fail2ban-0.9.5/

  * Only differences should be files that you don't want distributed.


 * Ensure the tests work from the tarball::

-    cd /tmp/fail2ban-0.9.4/ && bin/fail2ban-testcases
+    cd /tmp/fail2ban-0.9.6/ && bin/fail2ban-testcases

 * Add/finalize the corresponding entry in the ChangeLog

  * To generate a list of committers use e.g.::

-      git shortlog -sn 0.9.4.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g'
+      git shortlog -sn 0.9.5.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g'

  * Ensure the top of the ChangeLog has the right version and current date.
  * Ensure the top entry of the ChangeLog has the right version and current date.
@@ -106,7 +106,7 @@ Preparation
 * Tag the release by using a signed (and annotated) tag.  Cut/paste
  release ChangeLog entry as tag annotation::

-    git tag -s 0.9.4
+    git tag -s 0.9.5

 Pre Release
 ===========
@@ -190,14 +190,14 @@ Post Release

 Add the following to the top of the ChangeLog::

-  ver. 0.9.6 (2016/XX/XXX) - wanna-be-released
+  ver. 0.9.7 (2016/XX/XXX) - wanna-be-released
  -----------
  
-  - Fixes:
+  ### Fixes
  
-  - New Features:
+  ### New Features
  
-  - Enhancements:
+  ### Enhancements

 Alter the git shortlog command in the previous section to refer to the just
 released version.
@@ -80,14 +80,17 @@ class BadIPsAction(ActionBase):
 		If invalid `category`, `score`, `banaction` or `updateperiod`.
 	"""

+	TIMEOUT = 10
 	_badips = "http://www.badips.com"
 	def _Request(self, url, **argv):
 		return Request(url, headers={'User-Agent': self.agent}, **argv)

 	def __init__(self, jail, name, category, score=3, age="24h", key=None,
-		banaction=None, bancategory=None, bankey=None, updateperiod=900, agent="Fail2Ban"):
+		banaction=None, bancategory=None, bankey=None, updateperiod=900, agent="Fail2Ban", 
+		timeout=TIMEOUT):
 		super(BadIPsAction, self).__init__(jail, name)

+		self.timeout = timeout
 		self.agent = agent
 		self.category = category
 		self.score = score
@@ -119,7 +122,7 @@ class BadIPsAction(ActionBase):
 		"""
 		try:
 			response = urlopen(
-				self._Request("/".join([self._badips, "get", "categories"])), None, 3)
+				self._Request("/".join([self._badips, "get", "categories"])), timeout=self.timeout)
 		except HTTPError as response:
 			messages = json.loads(response.read().decode('utf-8'))
 			self._logSys.error(
@@ -173,7 +176,7 @@ class BadIPsAction(ActionBase):
 				urlencode({'age': age})])
 			if key:
 				url = "&".join([url, urlencode({'key': key})])
-			response = urlopen(self._Request(url))
+			response = urlopen(self._Request(url), timeout=self.timeout)
 		except HTTPError as response:
 			messages = json.loads(response.read().decode('utf-8'))
 			self._logSys.error(
@@ -358,7 +361,7 @@ class BadIPsAction(ActionBase):
 			url = "/".join([self._badips, "add", self.category, aInfo['ip']])
 			if self.key:
 				url = "?".join([url, urlencode({'key': self.key})])
-			response = urlopen(self._Request(url))
+			response = urlopen(self._Request(url), timeout=self.timeout)
 		except HTTPError as response:
 			messages = json.loads(response.read().decode('utf-8'))
 			self._logSys.error(
@@ -16,17 +16,18 @@ __pid_re = (?:\[\d+\])
 iso8601 = \d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d+[+-]\d{4}

 # All Asterisk log messages begin like this:
-log_prefix= (?:NOTICE|SECURITY)%(__pid_re)s:?(?:\[C-[\da-f]*\])? \S+:\d*( in \w+:)?
+log_prefix= (?:NOTICE|SECURITY|WARNING)%(__pid_re)s:?(?:\[C-[\da-f]*\])? [^:]+:\d*( in \w+:)?

-failregex = ^(%(__prefix_line)s|\[\]\s*)%(log_prefix)s Registration from '[^']*' failed for '<HOST>(:\d+)?' - (Wrong password|Username/auth name mismatch|No matching peer found|Not a local domain|Device does not match ACL|Peer is not supposed to register|ACL error \(permit/deny\)|Not a local domain)$
-            ^(%(__prefix_line)s|\[\]\s*)%(log_prefix)s Call from '[^']*' \(<HOST>:\d+\) to extension '[^']*' rejected because extension not found in context
-            ^(%(__prefix_line)s|\[\]\s*)%(log_prefix)s Host <HOST> failed to authenticate as '[^']*'$
-            ^(%(__prefix_line)s|\[\]\s*)%(log_prefix)s No registration for peer '[^']*' \(from <HOST>\)$
-            ^(%(__prefix_line)s|\[\]\s*)%(log_prefix)s Host <HOST> failed MD5 authentication for '[^']*' \([^)]+\)$
-            ^(%(__prefix_line)s|\[\]\s*)%(log_prefix)s Failed to authenticate (user|device) [^@]+@<HOST>\S*$
-            ^(%(__prefix_line)s|\[\]\s*)%(log_prefix)s hacking attempt detected '<HOST>'$
-            ^(%(__prefix_line)s|\[\]\s*)%(log_prefix)s SecurityEvent="(FailedACL|InvalidAccountID|ChallengeResponseFailed|InvalidPassword)",EventTV="([\d-]+|%(iso8601)s)",Severity="[\w]+",Service="[\w]+",EventVersion="\d+",AccountID="(\d*|<unknown>)",SessionID=".+",LocalAddress="IPV[46]/(UDP|TCP|WS)/[\da-fA-F:.]+/\d+",RemoteAddress="IPV[46]/(UDP|TCP|WS)/<HOST>/\d+"(,Challenge="[\w/]+")?(,ReceivedChallenge="\w+")?(,Response="\w+",ExpectedResponse="\w*")?(,ReceivedHash="[\da-f]+")?(,ACLName="\w+")?$
-            ^(%(__prefix_line)s|\[\]\s*WARNING%(__pid_re)s:?(?:\[C-[\da-f]*\])? )Ext\. s: "Rejecting unknown SIP connection from <HOST>"$
+failregex = ^%(__prefix_line)s%(log_prefix)s Registration from '[^']*' failed for '<HOST>(:\d+)?' - (Wrong password|Username/auth name mismatch|No matching peer found|Not a local domain|Device does not match ACL|Peer is not supposed to register|ACL error \(permit/deny\)|Not a local domain)$
+            ^%(__prefix_line)s%(log_prefix)s Call from '[^']*' \(<HOST>:\d+\) to extension '[^']*' rejected because extension not found in context
+            ^%(__prefix_line)s%(log_prefix)s Host <HOST> failed to authenticate as '[^']*'$
+            ^%(__prefix_line)s%(log_prefix)s No registration for peer '[^']*' \(from <HOST>\)$
+            ^%(__prefix_line)s%(log_prefix)s Host <HOST> failed MD5 authentication for '[^']*' \([^)]+\)$
+            ^%(__prefix_line)s%(log_prefix)s Failed to authenticate (user|device) [^@]+@<HOST>\S*$
+            ^%(__prefix_line)s%(log_prefix)s hacking attempt detected '<HOST>'$
+            ^%(__prefix_line)s%(log_prefix)s SecurityEvent="(FailedACL|InvalidAccountID|ChallengeResponseFailed|InvalidPassword)",EventTV="([\d-]+|%(iso8601)s)",Severity="[\w]+",Service="[\w]+",EventVersion="\d+",AccountID="(\d*|<unknown>)",SessionID=".+",LocalAddress="IPV[46]/(UDP|TCP|WS)/[\da-fA-F:.]+/\d+",RemoteAddress="IPV[46]/(UDP|TCP|WS)/<HOST>/\d+"(,Challenge="[\w/]+")?(,ReceivedChallenge="\w+")?(,Response="\w+",ExpectedResponse="\w*")?(,ReceivedHash="[\da-f]+")?(,ACLName="\w+")?$
+            ^%(__prefix_line)s%(log_prefix)s "Rejecting unknown SIP connection from <HOST>"$
+            ^%(__prefix_line)s%(log_prefix)s Request (?:'[^']*' )?from '[^']*' failed for '<HOST>(?::\d+)?'\s\(callid: [^\)]*\) - (?:No matching endpoint found|Not match Endpoint(?: Contact)? ACL|(?:Failed|Error) to authenticate)\s*$

 ignoreregex =

@@ -26,7 +26,7 @@ __daemon_re = [\[\(]?%(_daemon)s(?:\(\S+\))?[\]\)]?:?

 # extra daemon info
 # EXAMPLE: [ID 800047 auth.info]
-__daemon_extra_re = (?:\[ID \d+ \S+\])
+__daemon_extra_re = \[ID \d+ \S+\]

 # Combinations of daemon name and PID
 # EXAMPLES: sshd[31607], pop(pam_unix)[4920]
@@ -44,14 +44,18 @@ __md5hex = (?:[\da-f]{2}:){15}[\da-f]{2}

 # bsdverbose is where syslogd is started with -v or -vv and results in <4.3> or
 # <auth.info> appearing before the host as per testcases/files/logs/bsd/*.
-__bsd_syslog_verbose = (<[^.]+\.[^.]+>)
+__bsd_syslog_verbose = <[^.]+\.[^.]+>
+
+__vserver = @vserver_\S+
+
+__date_ambit = (?:\[\])

 # Common line prefixes (beginnings) which could be used in filters
 #
 #      [bsdverbose]? [hostname] [vserver tag] daemon_id spaces
 #
 # This can be optional (for instance if we match named native log files)
-__prefix_line = \s*%(__bsd_syslog_verbose)s?\s*(?:%(__hostname)s )?(?:%(__kernel_prefix)s )?(?:@vserver_\S+ )?%(__daemon_combs_re)s?\s%(__daemon_extra_re)s?\s*
+__prefix_line = %(__date_ambit)s?\s*(?:%(__bsd_syslog_verbose)s\s+)?(?:%(__hostname)s\s+)?(?:%(__kernel_prefix)s\s+)?(?:%(__vserver)s\s+)?(?:%(__daemon_combs_re)s\s+)?(?:%(__daemon_extra_re)s\s+)?

 # PAM authentication mechanism check for failures, e.g.: pam_unix, pam_sss,
 # pam_ldap
@@ -13,7 +13,7 @@ before = common.conf
 _daemon = courieresmtpd

 failregex = ^%(__prefix_line)serror,relay=<HOST>,.*: 550 User (<.*> )?unknown\.?$
-            ^%(__prefix_line)serror,relay=<HOST>,msg="535 Authentication failed\.",cmd:( AUTH \S+)?( [0-9a-zA-Z\+/=]+)?$
+            ^%(__prefix_line)serror,relay=<HOST>,msg="535 Authentication failed\.",cmd:( AUTH \S+)?( [0-9a-zA-Z\+/=]+)?(?: \S+)$

 ignoreregex = 

@@ -9,8 +9,8 @@ after = exim-common.local

 [Definition]

-host_info = H=([\w.-]+ )?(\(\S+\) )?\[<HOST>\](:\d+)? (I=\[\S+\]:\d+ )?(U=\S+ )?(P=e?smtp )?
-pid = ( \[\d+\])?
+host_info = (?:H=([\w.-]+ )?(?:\(\S+\) )?)?\[<HOST>\](?::\d+)? (?:I=\[\S+\](:\d+)? )?(?:U=\S+ )?(?:P=e?smtp )?
+pid = (?: \[\d+\])?

 # DEV Notes:
 # From exim source code: ./src/receive.c:add_host_info_for_log
@@ -14,13 +14,13 @@ before = exim-common.conf
 [Definition]

 failregex = ^%(pid)s %(host_info)ssender verify fail for <\S+>: (?:Unknown user|Unrouteable address|all relevant MX records point to non-existent hosts)\s*$
-            ^%(pid)s \w+ authenticator failed for (\S+ )?\(\S+\) \[<HOST>\](:\d+)?( I=\[\S+\](:\d+)?)?: 535 Incorrect authentication data( \(set_id=.*\)|: \d+ Time\(s\))?\s*$
-            ^%(pid)s %(host_info)sF=(<>|[^@]+@\S+) rejected RCPT [^@]+@\S+: (relay not permitted|Sender verify failed|Unknown user)\s*$
-            ^%(pid)s SMTP protocol synchronization error \([^)]*\): rejected (connection from|"\S+") %(host_info)s(next )?input=".*"\s*$
-            ^%(pid)s SMTP call from \S+ \[<HOST>\](:\d+)? (I=\[\S+\](:\d+)? )?dropped: too many nonmail commands \(last was "\S+"\)\s*$
-            ^%(pid)s SMTP protocol error in "AUTH \S*(| \S*)" H=(|\S* )(|\(\S*\) )\[<HOST>\]\:\d+ I=\[\S*\]\:\d+ AUTH command used when not advertised\s*$
-            ^%(pid)s no MAIL in SMTP connection from (|\S* )(|\(\S*\) )\[<HOST>\]\:\d+ I=\[\S*\]\:\d+ D=\d+s(| C=\S*)\s*$
-            ^%(pid)s \S+ SMTP connection from (|\S* )(|\(\S*\) )\[<HOST>\]\:\d+ I=\[\S*\]\:\d+ closed by DROP in ACL\s*$
+            ^%(pid)s \w+ authenticator failed for (\S+ )?\(\S+\) \[<HOST>\](?::\d+)?(?: I=\[\S+\](:\d+)?)?: 535 Incorrect authentication data( \(set_id=.*\)|: \d+ Time\(s\))?\s*$
+            ^%(pid)s %(host_info)sF=(?:<>|[^@]+@\S+) rejected RCPT [^@]+@\S+: (?:relay not permitted|Sender verify failed|Unknown user)\s*$
+            ^%(pid)s SMTP protocol synchronization error \([^)]*\): rejected (?:connection from|"\S+") %(host_info)s(?:next )?input=".*"\s*$
+            ^%(pid)s SMTP call from \S+ %(host_info)sdropped: too many nonmail commands \(last was "\S+"\)\s*$
+            ^%(pid)s SMTP protocol error in "AUTH \S*(?: \S*)?" %(host_info)sAUTH command used when not advertised\s*$
+            ^%(pid)s no MAIL in SMTP connection from (?:\S* )?(?:\(\S*\) )?%(host_info)sD=\d+s(?: C=\S*)?\s*$
+            ^%(pid)s \S+ SMTP connection from (?:\S* )?(?:\(\S*\) )?%(host_info)sclosed by DROP in ACL\s*$

 ignoreregex = 

@@ -22,7 +22,7 @@ _daemon = nsd
 #          (?:::f{4,6}:)?(?P<host>[\w\-.^_]+)
 # Values:  TEXT

-failregex =  ^\[\]%(__prefix_line)sinfo: ratelimit block .* query <HOST> TYPE255$
-              ^\[\]%(__prefix_line)sinfo: .* <HOST> refused, no acl matches\.$
+failregex =  ^%(__prefix_line)sinfo: ratelimit block .* query <HOST> TYPE255$
+             ^%(__prefix_line)sinfo: .* <HOST> refused, no acl matches\.$

 ignoreregex =
@@ -0,0 +1,25 @@
+# slapd (Stand-alone LDAP Daemon) openldap daemon filter
+#
+# Detecting invalid credentials: error code 49
+# http://www.openldap.org/doc/admin24/appendix-ldap-result-codes.html#invalidCredentials (49)
+
+[INCLUDES]
+
+# Read common prefixes. If any customizations available -- read them from
+# common.local
+before = common.conf
+
+[Definition]
+
+_daemon = slapd
+
+failregex = ^(?P<__prefix>%(__prefix_line)s)conn=(?P<_conn_>\d+) fd=\d+ ACCEPT from IP=<HOST>:\d{1,5} \(IP=\S+\)\s*<SKIPLINES>(?P=__prefix)conn=(?P=_conn_) op=\d+ RESULT(?:\s(?!err)\S+=\S*)* err=49 text=[\w\s]*$
+
+ignoreregex =
+
+[Init]
+
+# "maxlines" is number of log lines to buffer for multi-line regex searches
+maxlines = 20
+
+# Author: Andrii Melnyk
@@ -94,6 +94,7 @@ backend = auto
 #        but it will be logged as a warning.
 # no:    if a hostname is encountered, will not be used for banning,
 #        but it will be logged as info.
+# raw:   use raw value (no hostname), allow use it for no-host filters/actions (example user)
 usedns = warn

 # "logencoding" specifies the encoding of the log files handled by the jail
@@ -839,3 +840,8 @@ logencoding = utf-8
 # See "haproxy-http-auth" filter for a brief cautionary note when setting
 # maxretry and findtime.
 logpath  = /var/log/haproxy.log
+
+[slapd]
+port    = ldap,ldaps
+filter  = slapd
+logpath = /var/log/slapd.log
@@ -40,7 +40,8 @@ lighttpd_error_log = /var/log/lighttpd/error.log
 # http://www.hardened-php.net/suhosin/configuration.html#suhosin.log.syslog.facility
 # syslog_user is the default. Lighttpd also hooks errors into its log.

-suhosin_log = %(syslog_user)s %(lighttpd_error_log)s
+suhosin_log = %(syslog_user)s
+              %(lighttpd_error_log)s

 # defaults to ftp or local2 if ftp doesn't exist
 proftpd_log = %(syslog_ftp)s
@@ -126,6 +126,8 @@ Report bugs to https://github.com/fail2ban/fail2ban/issues
 			   help="set custom pattern used to match date/times"),
 		Option("-e", "--encoding",
 			   help="File encoding. Default: system locale"),
+		Option("-r", "--raw", action='store_true',
+			   help="Raw hosts, don't resolve dns"),
 		Option("-L", "--maxlines", type=int, default=0,
 			   help="maxlines for multi-line regex"),
 		Option("-m", "--journalmatch",
@@ -239,6 +241,7 @@ class Fail2banRegex(object):
 			self.encoding = opts.encoding
 		else:
 			self.encoding = locale.getpreferredencoding()
+		self.raw = True if opts.raw else False

 	def decode_line(self, line):
 		return FileContainer.decode_line('<LOG>', self.encoding, line)
@@ -335,7 +338,7 @@ class Fail2banRegex(object):
 		orgLineBuffer = self._filter._Filter__lineBuffer
 		fullBuffer = len(orgLineBuffer) >= self._filter.getMaxLines()
 		try:
-			line, ret = self._filter.processLine(line, date, checkAllRegex=True)
+			line, ret = self._filter.processLine(line, date, checkAllRegex=True, returnRawHost=self.raw)
 			for match in ret:
 				# Append True/False flag depending if line was matched by
 				# more than one regex
@@ -34,7 +34,7 @@ from .filterreader import FilterReader
 from .actionreader import ActionReader
 from ..version import version
 from ..helpers import getLogger
-from ..helpers import splitcommaspace
+from ..helpers import splitwords

 # Gets the instance of the logger.
 logSys = getLogger(__name__)
@@ -214,7 +214,7 @@ class JailReader(ConfigReader):
 			elif opt == "maxretry":
 				stream.append(["set", self.__name, "maxretry", self.__opts[opt]])
 			elif opt == "ignoreip":
-				for ip in splitcommaspace(self.__opts[opt]):
+				for ip in splitwords(self.__opts[opt]):
 					stream.append(["set", self.__name, "addignoreip", ip])
 			elif opt == "findtime":
 				stream.append(["set", self.__name, "findtime", self.__opts[opt]])
@@ -128,12 +128,12 @@ def excepthook(exctype, value, traceback):
 		"Unhandled exception in Fail2Ban:", exc_info=True)
 	return sys.__excepthook__(exctype, value, traceback)

-def splitcommaspace(s):
-	"""Helper to split on any comma or space
+def splitwords(s):
+	"""Helper to split words on any comma, space, or a new line

 	Returns empty list if input is empty (or None) and filters
 	out empty entries
 	"""
 	if not s:
 		return []
-	return filter(bool, re.split('[ ,]', s))
+	return filter(bool, map(str.strip, re.split('[ ,\n]+', s)))
@@ -55,6 +55,9 @@ _RETCODE_HINTS = {
 signame = dict((num, name)
 	for name, num in signal.__dict__.iteritems() if name.startswith("SIG"))

+# max tag replacement count:
+MAX_TAG_REPLACE_COUNT = 10
+

 class CallingMap(MutableMapping):
 	"""A Mapping type which returns the result of callable values.
@@ -390,22 +393,22 @@ class CommandAction(ActionBase):
 		"""
 		t = re.compile(r'<([^ <>]+)>')
 		# repeat substitution while embedded-recursive (repFlag is True)
+		done = cls._escapedTags.copy()
 		while True:
 			repFlag = False
 			# substitute each value:
 			for tag in tags.iterkeys():
-				if tag in cls._escapedTags:
-					# Escaped so won't match
-					continue
+				# ignore escaped or already done:
+				if tag in done: continue
 				value = str(tags[tag])
 				# search and replace all tags within value, that can be interpolated using other tags:
 				m = t.search(value)
-				done = []
+				refCounts = {}
 				#logSys.log(5, 'TAG: %s, value: %s' % (tag, value))
 				while m:
 					found_tag = m.group(1)
 					#logSys.log(5, 'found: %s' % found_tag)
-					if found_tag == tag or found_tag in done:
+					if found_tag == tag or refCounts.get(found_tag, 1) > MAX_TAG_REPLACE_COUNT:
 						# recursive definitions are bad
 						#logSys.log(5, 'recursion fail tag: %s value: %s' % (tag, value) )
 						return False
@@ -417,7 +420,9 @@ class CommandAction(ActionBase):
 						continue
 					value = value.replace('<%s>' % found_tag , tags[found_tag])
 					#logSys.log(5, 'value now: %s' % value)
-					done.append(found_tag)
+					# increment reference count:
+					refCounts[found_tag] = refCounts.get(found_tag, 0) + 1
+					# the next match for replace:
 					m = t.search(value, m.start())
 				#logSys.log(5, 'TAG: %s, newvalue: %s' % (tag, value))
 				# was substituted?
@@ -426,6 +431,9 @@ class CommandAction(ActionBase):
 					if t.search(value):
 						repFlag = True
 					tags[tag] = value
+				# no more sub tags (and no possible composite), add this tag to done set (just to be faster):
+				if '<' not in value: done.add(tag)
+			# stop interpolation, if no replacements anymore:
 			if not repFlag:
 				break
 		return tags
@@ -181,8 +181,23 @@ class Fail2BanDb(object):
 				filename, e.args[0])
 			raise

+		# differentiate pypy: switch journal mode later (save it during the upgrade), 
+		# to prevent errors like "database table is locked":
+		try:
+			import __pypy__
+			pypy = True
+		except ImportError:
+			pypy = False
+
 		cur = self._db.cursor()
-		cur.execute("PRAGMA foreign_keys = ON;")
+		cur.execute("PRAGMA foreign_keys = ON")
+		# speedup: write data through OS without syncing (no wait):
+		cur.execute("PRAGMA synchronous = OFF")
+		# speedup: transaction log in memory, alternate using OFF (disable, rollback will be impossible):
+		if not pypy:
+			cur.execute("PRAGMA journal_mode = MEMORY")
+		# speedup: temporary tables and indices are kept in memory:
+		cur.execute("PRAGMA temp_store = MEMORY")

 		try:
 			cur.execute("SELECT version FROM fail2banDb LIMIT 1")
@@ -202,6 +217,9 @@ class Fail2BanDb(object):
 						Fail2BanDb.__version__, version, newversion)
 					raise RuntimeError('Failed to fully update')
 		finally:
+			# pypy: set journal mode after possible upgrade db:
+			if pypy:
+				cur.execute("PRAGMA journal_mode = MEMORY")
 			cur.close()

 	@property
@@ -244,13 +262,14 @@ class Fail2BanDb(object):

 		A timestamped backup is also created prior to attempting the update.
 		"""
-		self._dbBackupFilename = self.filename + '.' + time.strftime('%Y%m%d-%H%M%S', MyTime.gmtime())
-		shutil.copyfile(self.filename, self._dbBackupFilename)
-		logSys.info("Database backup created: %s", self._dbBackupFilename)
 		if version > Fail2BanDb.__version__:
 			raise NotImplementedError(
 						"Attempt to travel to future version of database ...how did you get here??")

+		self._dbBackupFilename = self.filename + '.' + time.strftime('%Y%m%d-%H%M%S', MyTime.gmtime())
+		shutil.copyfile(self.filename, self._dbBackupFilename)
+		logSys.info("Database backup created: %s", self._dbBackupFilename)
+
 		if version < 2:
 			cur.executescript("BEGIN TRANSACTION;"
 						"CREATE TEMPORARY TABLE logs_temp AS SELECT * FROM logs;"
@@ -168,7 +168,7 @@ class Filter(JailThread):
 		if isinstance(value, bool):
 			value = {True: 'yes', False: 'no'}[value]
 		value = value.lower()			  # must be a string by now
-		if not (value in ('yes', 'no', 'warn')):
+		if not (value in ('yes', 'warn', 'no', 'raw')):
 			logSys.error("Incorrect value %r specified for usedns. "
 						 "Using safe 'no'" % (value,))
 			value = 'no'
@@ -523,7 +523,7 @@ class Filter(JailThread):
 					self.__lineBuffer = failRegex.getUnmatchedTupleLines()
 					try:
 						host = failRegex.getHost()
-						if returnRawHost:
+						if returnRawHost or self.__useDns == "raw":
 							failList.append([failRegexIndex, host, date,
 								 failRegex.getMatchedLines()])
 							if not checkAllRegex:
@@ -37,6 +37,7 @@ if sys.version_info >= (2,7):
 			self.jail.actions.add("badips", pythonModule, initOpts={
 				'category': "ssh",
 				'banaction': "test",
+				'timeout': 30,
 				})
 			self.action = self.jail.actions["badips"]

@@ -29,6 +29,7 @@ import time
 import tempfile

 from ..server.action import CommandAction, CallingMap
+from ..server.actions import OrderedDict

 from .utils import LogCaptureTestCase
 from .utils import pid_exists
@@ -58,6 +59,62 @@ class CommandActionTest(LogCaptureTestCase):
 		# Unresolveable substition
 		self.assertFalse(CommandAction.substituteRecursiveTags({'A': 'to=<B> fromip=<IP>', 'C': '<B>', 'B': '<C>', 'D': ''}))
 		self.assertFalse(CommandAction.substituteRecursiveTags({'failregex': 'to=<honeypot> fromip=<IP>', 'sweet': '<honeypot>', 'honeypot': '<sweet>', 'ignoreregex': ''}))
+		# We need here an ordered, because the sequence of iteration is very important for this test
+		if OrderedDict:
+			# No cyclic recursion, just multiple replacement of tag <T>, should be successful:
+			self.assertEqual(CommandAction.substituteRecursiveTags( OrderedDict(
+					(('X', 'x=x<T>'), ('T', '1'), ('Z', '<X> <T> <Y>'), ('Y', 'y=y<T>')))
+				), {'X': 'x=x1', 'T': '1', 'Y': 'y=y1', 'Z': 'x=x1 1 y=y1'}
+			)
+			# No cyclic recursion, just multiple replacement of tag <T> in composite tags, should be successful:
+			self.assertEqual(CommandAction.substituteRecursiveTags( OrderedDict(
+				  (('X', 'x=x<T> <Z> <<R1>> <<R2>>'), ('R1', 'Z'), ('R2', 'Y'), ('T', '1'), ('Z', '<T> <Y>'), ('Y', 'y=y<T>')))
+				), {'X': 'x=x1 1 y=y1 1 y=y1 y=y1', 'R1': 'Z', 'R2': 'Y', 'T': '1', 'Z': '1 y=y1', 'Y': 'y=y1'}
+			)
+			# No cyclic recursion, just multiple replacement of same tags, should be successful:
+			self.assertEqual(CommandAction.substituteRecursiveTags( OrderedDict((
+					('actionstart', 'ipset create <ipmset> hash:ip timeout <bantime> family <ipsetfamily>\n<iptables> -I <chain> <actiontype>'),
+					('ipmset', 'f2b-<name>'),
+					('name', 'any'),
+					('bantime', '600'),
+					('ipsetfamily', 'inet'),
+					('iptables', 'iptables <lockingopt>'),
+					('lockingopt', '-w'),
+					('chain', 'INPUT'),
+					('actiontype', '<multiport>'),
+					('multiport', '-p <protocol> -m multiport --dports <port> -m set --match-set <ipmset> src -j <blocktype>'),
+					('protocol', 'tcp'),
+					('port', 'ssh'),
+					('blocktype', 'REJECT',),
+				))
+				), OrderedDict((
+					('actionstart', 'ipset create f2b-any hash:ip timeout 600 family inet\niptables -w -I INPUT -p tcp -m multiport --dports ssh -m set --match-set f2b-any src -j REJECT'),
+					('ipmset', 'f2b-any'),
+					('name', 'any'),
+					('bantime', '600'),
+					('ipsetfamily', 'inet'),
+					('iptables', 'iptables -w'),
+					('lockingopt', '-w'),
+					('chain', 'INPUT'),
+					('actiontype', '-p tcp -m multiport --dports ssh -m set --match-set f2b-any src -j REJECT'),
+					('multiport', '-p tcp -m multiport --dports ssh -m set --match-set f2b-any src -j REJECT'),
+					('protocol', 'tcp'),
+					('port', 'ssh'),
+					('blocktype', 'REJECT')
+				))
+			)
+			# Cyclic recursion by composite tag creation, tags "create" another tag, that closes cycle:
+			self.assertFalse(CommandAction.substituteRecursiveTags( OrderedDict((
+					('A', '<<B><C>>'),
+					('B', 'D'), ('C', 'E'),
+					('DE', 'cycle <A>'),
+			)) ))
+			self.assertFalse(CommandAction.substituteRecursiveTags( OrderedDict((
+					('DE', 'cycle <A>'),
+					('A', '<<B><C>>'),
+					('B', 'D'), ('C', 'E'),
+			)) ))
+			
 		# missing tags are ok
 		self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<C>'}), {'A': '<C>'})
 		self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<C> <D> <X>','X':'fun'}), {'A': '<C> <D> fun', 'X':'fun'})
@@ -125,6 +182,7 @@ class CommandActionTest(LogCaptureTestCase):
 				CallingMap(matches=lambda: str(10))),
 			"09 10 11")

+	def testReplaceNoTag(self):
 		# As tag not present, therefore callable should not be called
 		# Will raise ValueError if it is
 		self.assertEqual(
@@ -0,0 +1 @@
+../../../../config/filter.d/common.conf
@@ -0,0 +1,18 @@
+# Fail2Ban generic example resp. test filter 
+#
+# Author: Serg G. Brester (sebres)
+#
+
+[INCLUDES]
+
+# Read common prefixes. If any customizations available -- read them from
+# common.local.  common.conf is a symlink to the original common.conf and
+# should be copied (dereferenced) during installation
+before = common.conf
+
+[Definition]
+
+_daemon = test-demo
+
+failregex = ^%(__prefix_line)sF2B: failure from <HOST>$
+ignoreregex = 
@@ -132,6 +132,15 @@ class Fail2banRegexTest(LogCaptureTestCase):
 		self.assertLogged('Dez 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 193.168.0.128')
 		self.assertLogged('Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 87.142.124.10')

+	def testDirectRE_1raw(self):
+		(opts, args, fail2banRegex) = _Fail2banRegex(
+			"--print-all-matched", "--raw",
+			Fail2banRegexTest.FILENAME_01, 
+			Fail2banRegexTest.RE_00
+		)
+		self.assertTrue(fail2banRegex.start(opts, args))
+		self.assertLogged('Lines: 19 lines, 0 ignored, 16 matched, 3 missed')
+
 	def testDirectRE_2(self):
 		(opts, args, fail2banRegex) = _Fail2banRegex(
 			"--print-all-matched",
@@ -67,3 +67,18 @@ Nov 4 18:30:40 localhost asterisk[32229]: NOTICE[32257]: chan_sip.c:23417 in han
 [2016-01-28 10:34:31] NOTICE[3477][C-000003c3] chan_sip.c: Call from '' (1.2.3.4:10836) to extension '0+441772285407' rejected because extension not found in context 'default'.
 # failJSON: { "time": "2016-01-28T10:34:33", "match": true , "host": "1.2.3.4" }
 [2016-01-28 10:34:33] NOTICE[3477][C-000003c3] chan_sip.c: Call from '' (1.2.3.4:10836) to extension '' rejected because extension not found in context 'my-context'.
+
+# Failed authentication with pjsip on Asterisk 13+
+# failJSON: { "time": "2016-05-23T10:18:16", "match": true , "host": "1.2.3.4" }
+[2016-05-23 10:18:16] NOTICE[19388] res_pjsip/pjsip_distributor.c: Request from '"1000" <sip:1000@10.0.0.1>' failed for '1.2.3.4:48336' (callid: 276666022) - No matching endpoint found
+# failJSON: { "time": "2016-05-23T10:18:16", "match": true , "host": "1.2.3.4" }
+[2016-05-23 10:18:16] NOTICE[19388] res_pjsip/pjsip_distributor.c: Request from '"1000" <sip:1000@10.0.0.1>' failed for '1.2.3.4:48336' (callid: 276666022) - Not match Endpoint ACL
+# failJSON: { "time": "2016-05-23T10:18:16", "match": true , "host": "1.2.3.4" }
+[2016-05-23 10:18:16] NOTICE[19388] res_pjsip/pjsip_distributor.c: Request from '"1000" <sip:1000@10.0.0.1>' failed for '1.2.3.4:48336' (callid: 276666022) - Not match Endpoint Contact ACL
+# failJSON: { "time": "2016-05-23T10:18:16", "match": true , "host": "1.2.3.4" }
+[2016-05-23 10:18:16] NOTICE[19388] res_pjsip/pjsip_distributor.c: Request from '"1000" <sip:1000@10.0.0.1>' failed for '1.2.3.4:48336' (callid: 276666022) - Failed to authenticate
+# failJSON: { "time": "2016-05-23T10:18:16", "match": true , "host": "1.2.3.4" }
+[2016-05-23 10:18:16] NOTICE[19388] res_pjsip/pjsip_distributor.c: Request from '"1000" <sip:1000@10.0.0.1>' failed for '1.2.3.4:48336' (callid: 276666022) - Error to authenticate
+# Failed authentication with pjsip on Asterisk 13+
+# failJSON: { "time": "2016-06-08T23:40:26", "match": true , "host": "2.3.4.5" }
+[2016-06-08 23:40:26] NOTICE[32497] res_pjsip/pjsip_distributor.c: Request from '"317" <sip:317@1.2.3.4>' failed for '2.3.4.5:5089' (callid: 206f178f-896564cb-57573f49@1.2.3.4) - No matching endpoint found
@@ -10,3 +10,5 @@ Jul  6 03:42:28 whistler courieresmtpd: error,relay=::ffff:1.2.3.4,from=<>,to=<a
 Nov 21 23:16:17 server courieresmtpd: error,relay=::ffff:1.2.3.4,from=<>,to=<>: 550 User unknown.
 # failJSON: { "time": "2004-08-14T12:51:04", "match": true , "host": "1.2.3.4" }
 Aug 14 12:51:04 HOSTNAME courieresmtpd: error,relay=::ffff:1.2.3.4,from=<firozquarl@aclunc.org>,to=<BOGUSUSER@HOSTEDDOMAIN.org>: 550 User unknown.
+# failJSON: { "time": "2004-08-14T12:51:04", "match": true , "host": "1.2.3.4" }
+Aug 14 12:51:04 mail.server courieresmtpd[26762]: error,relay=::ffff:1.2.3.4,msg="535 Authentication failed.",cmd: AUTH PLAIN AAAAABBBBCCCCWxlZA== admin
@@ -48,10 +48,14 @@
 2016-03-18 00:34:06 [7513] SMTP protocol error in "AUTH LOGIN" H=(ylmf-pc) [45.32.34.167]:60723 I=[172.89.0.6]:587 AUTH command used when not advertised
 # failJSON: { "time": "2016-03-19T18:40:44", "match": true , "host": "92.45.204.170" }
 2016-03-19 18:40:44 [26221] SMTP protocol error in "AUTH LOGIN aW5mb0BtYW5iYXQub3Jn" H=([127.0.0.1]) [92.45.204.170]:14243 I=[172.89.0.6]:587 AUTH command used when not advertised
+# failJSON: { "time": "2016-05-17T06:25:27", "match": true , "host": "69.10.61.61", "desc": "from gh-1430" }
+2016-05-17 06:25:27 SMTP protocol error in "AUTH LOGIN" H=(ylmf-pc) [69.10.61.61] AUTH command used when not advertised
 # failJSON: { "time": "2016-03-21T06:38:05", "match": true , "host": "49.212.207.15" }
 2016-03-21 06:38:05 [5718] no MAIL in SMTP connection from www3005.sakura.ne.jp [49.212.207.15]:28890 I=[172.89.0.6]:25 D=21s C=EHLO,STARTTLS
 # failJSON: { "time": "2016-03-21T06:57:36", "match": true , "host": "122.165.71.116" }
 2016-03-21 06:57:36 [5908] no MAIL in SMTP connection from [122.165.71.116]:2056 I=[172.89.0.6]:25 D=10s
+# failJSON: { "time": "2016-03-21T06:57:36", "match": true , "host": "122.165.71.116" }
+2016-03-21 06:57:36 [5908] no MAIL in SMTP connection from [122.165.71.116] I=[172.89.0.6]:25 D=10s
 # failJSON: { "time": "2016-03-21T04:07:49", "match": true , "host": "174.137.147.204" }
 2016-03-21 04:07:49 [25874] 1ahr79-0006jK-G9 SMTP connection from (voyeur.webair.com) [174.137.147.204]:44884 I=[172.89.0.6]:25 closed by DROP in ACL
 # failJSON: { "time": "2016-03-21T04:33:13", "match": true , "host": "206.214.71.53" }
@@ -0,0 +1,18 @@
+# failJSON: { "match": false }
+Jul  8 01:47:19 ldap-server slapd[1183]: conn=1022 fd=21 ACCEPT from IP=8.8.8.8:45011 (IP=0.0.0.0:636)
+# failJSON: { "match": false }
+Jul  8 01:47:19 ldap-server slapd[1183]: conn=1022 fd=21 TLS established tls_ssf=256 ssf=256
+# failJSON: { "match": false }
+Jul  8 01:47:19 ldap-server slapd[1183]: conn=1022 op=0 EXT oid=1.3.6.1.4.1.6.1
+# failJSON: { "match": false }
+Jul  8 01:47:19 ldap-server slapd[1183]: conn=1022 op=0 STARTTLS
+# failJSON: { "match": false }
+Jul  8 01:47:19 ldap-server slapd[1183]: conn=1022 op=0 RESULT oid= err=1 text=TLS already started
+# failJSON: { "match": false }
+Jul  8 01:47:20 ldap-server slapd[1183]: conn=1022 op=1 BIND dn="uid=gipson,ou=people,dc=example,dc=com" method=128
+# failJSON: { "time": "2005-07-08T01:47:20", "match": true , "host": "8.8.8.8", "desc": "Multiline match for invalid credentials" }
+Jul  8 01:47:20 ldap-server slapd[1183]: conn=1022 op=1 RESULT tag=97 err=49 text=
+# failJSON: { "match": false }
+Jul  8 01:47:20 ldap-server slapd[1183]: conn=1022 op=2 UNBIND
+# failJSON: { "match": false }
+Jul  8 01:47:20 ldap-server slapd[1183]: conn=1022 fd=21 closed
@@ -0,0 +1,31 @@
+# -- _daemon with __pid_re, without __hostname --
+# failJSON: { "time": "2005-06-21T16:47:46", "match": true , "host": "192.0.2.1" }
+Jun 21 16:47:46 machine test-demo[13709]: F2B: failure from 192.0.2.1
+# -- _daemon with __pid_re --
+# failJSON: { "time": "2005-06-21T16:47:48", "match": true , "host": "192.0.2.1" }
+Jun 21 16:47:48 test-demo[13709]: F2B: failure from 192.0.2.1
+
+# -- __kernel_prefix --
+# failJSON: { "time": "2005-06-21T16:47:50", "match": true , "host": "192.0.2.2" }
+Jun 21 16:47:50 machine kernel: [  970.699396] F2B: failure from 192.0.2.2
+
+# -- _daemon_re with and without __pid_re --
+# failJSON: { "time": "2005-06-21T16:47:52", "match": true , "host": "192.0.2.3" }
+Jun 21 16:47:52 machine [test-demo] F2B: failure from 192.0.2.3
+# failJSON: { "time": "2005-06-21T16:47:53", "match": true , "host": "192.0.2.3" }
+Jun 21 16:47:53 machine [test-demo][13709] F2B: failure from 192.0.2.3
+# failJSON: { "time": "2005-06-21T16:50:00", "match": true , "host": "192.0.2.3" }
+Jun 21 16:50:00 machine test-demo(pam_unix) F2B: failure from 192.0.2.3
+# failJSON: { "time": "2005-06-21T16:50:02", "match": true , "host": "192.0.2.3" }
+Jun 21 16:50:02 machine test-demo(pam_unix)[13709] F2B: failure from 192.0.2.3
+
+
+# -- all common definitions together (bsdverbose hostname kernel_prefix vserver tag daemon_id space) --
+# failJSON: { "time": "2005-06-21T16:55:01", "match": true , "host": "192.0.2.3" }
+Jun 21 16:55:01 <auth.info> machine kernel: [  970.699396] @vserver_demo test-demo(pam_unix)[13709] [ID 255 test] F2B: failure from 192.0.2.3
+# -- the same as above with additional spaces around --
+# failJSON: { "time": "2005-06-21T16:55:02", "match": true , "host": "192.0.2.3" }
+Jun 21 16:55:02   <auth.info> machine kernel: [  970.699396] @vserver_demo test-demo(pam_unix)[13709] [ID 255 test]    F2B: failure from 192.0.2.3
+# -- the same as above with brackets as date ambit --
+# failJSON: { "time": "2005-06-21T16:55:03", "match": true , "host": "192.0.2.3" }
+[Jun 21 16:55:03] <auth.info> machine kernel: [  970.699396] @vserver_demo test-demo(pam_unix)[13709] [ID 255 test] F2B: failure from 192.0.2.3
@@ -33,7 +33,7 @@ from glob import glob
 from StringIO import StringIO

 from ..helpers import formatExceptionInfo, mbasename, TraceBack, FormatterWithTraceBack, getLogger
-from ..helpers import splitcommaspace
+from ..helpers import splitwords
 from ..server.datetemplate import DatePatternRegex


@@ -56,13 +56,15 @@ class HelpersTest(unittest.TestCase):
 			# might be fragile due to ' vs "
 			self.assertEqual(args, "('Very bad', None)")

-	def testsplitcommaspace(self):
-		self.assertEqual(splitcommaspace(None), [])
-		self.assertEqual(splitcommaspace(''), [])
-		self.assertEqual(splitcommaspace('  '), [])
-		self.assertEqual(splitcommaspace('1'), ['1'])
-		self.assertEqual(splitcommaspace(' 1 2 '), ['1', '2'])
-		self.assertEqual(splitcommaspace(' 1, 2 , '), ['1', '2'])
+	def testsplitwords(self):
+		self.assertEqual(splitwords(None), [])
+		self.assertEqual(splitwords(''), [])
+		self.assertEqual(splitwords('  '), [])
+		self.assertEqual(splitwords('1'), ['1'])
+		self.assertEqual(splitwords(' 1 2 '), ['1', '2'])
+		self.assertEqual(splitwords(' 1, 2 , '), ['1', '2'])
+		self.assertEqual(splitwords(' 1\n  2'), ['1', '2'])
+		self.assertEqual(splitwords(' 1\n  2, 3'), ['1', '2', '3'])


 class SetupTest(unittest.TestCase):
@@ -35,6 +35,7 @@ from ..server.filter import Filter
 from ..client.filterreader import FilterReader
 from .utils import setUpMyTime, tearDownMyTime, CONFIG_DIR

+TEST_CONFIG_DIR = os.path.join(os.path.dirname(__file__), "config")
 TEST_FILES_DIR = os.path.join(os.path.dirname(__file__), "files")


@@ -60,11 +61,11 @@ class FilterSamplesRegex(unittest.TestCase):
 			"Expected more FilterSampleRegexs tests")


-def testSampleRegexsFactory(name):
+def testSampleRegexsFactory(name, basedir):
 	def testFilter(self):

 		# Check filter exists
-		filterConf = FilterReader(name, "jail", {}, basedir=CONFIG_DIR)
+		filterConf = FilterReader(name, "jail", {}, basedir=basedir)
 		self.assertEqual(filterConf.getFile(), name)
 		self.assertEqual(filterConf.getJailName(), "jail")
 		filterConf.read()
@@ -147,11 +148,15 @@ def testSampleRegexsFactory(name):

 	return testFilter

-for filter_ in filter(lambda x: not x.endswith('common.conf') and x.endswith('.conf'),
-					  os.listdir(os.path.join(CONFIG_DIR, "filter.d"))):
-	filterName = filter_.rpartition(".")[0]
-	if not filterName.startswith('.'):
-		setattr(
-			FilterSamplesRegex,
-			"testSampleRegexs%s" % filterName.upper(),
-			testSampleRegexsFactory(filterName))
+for basedir_, filter_ in (
+	(CONFIG_DIR, lambda x: not x.endswith('common.conf') and x.endswith('.conf')),
+	(TEST_CONFIG_DIR, lambda x: x.startswith('zzz-') and x.endswith('.conf')),
+):
+	for filter_ in filter(filter_,
+						  os.listdir(os.path.join(basedir_, "filter.d"))):
+		filterName = filter_.rpartition(".")[0]
+		if not filterName.startswith('.'):
+			setattr(
+				FilterSamplesRegex,
+				"testSampleRegexs%s" % filterName.upper(),
+				testSampleRegexsFactory(filterName, basedir_))
@@ -24,4 +24,4 @@ __author__ = "Cyril Jaquier, Yaroslav Halchenko, Steven Hiscocks, Daniel Black"
 __copyright__ = "Copyright (c) 2004 Cyril Jaquier, 2005-2016 Yaroslav Halchenko, 2013-2014 Steven Hiscocks, Daniel Black"
 __license__ = "GPL-v2+"

-version = "0.9.4.dev0"
+version = "0.9.5"
@@ -34,15 +34,15 @@ start() {
 	# remove stalled sock file after system crash
 	# bug 347477
 	rm -f /var/run/fail2ban/fail2ban.sock || return 1
-	start-stop-daemon --start --exec ${FAIL2BAN} start \
-	--pidfile /var/run/fail2ban/fail2ban.pid
+	start-stop-daemon --start --pidfile /var/run/fail2ban/fail2ban.pid \
+	-- ${FAIL2BAN} start
 	eend $? "Failed to start fail2ban"
 }

 stop() {
 	ebegin "Stopping fail2ban"
-	start-stop-daemon --stop --exec ${FAIL2BAN} stop \
-	--pidfile /var/run/fail2ban/fail2ban.pid
+	start-stop-daemon --stop --pidfile /var/run/fail2ban/fail2ban.pid \
+	-- ${FAIL2BAN} stop
 	eend $? "Failed to stop fail2ban"
 }

@@ -1,12 +1,12 @@
-.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.2.
-.TH FAIL2BAN-CLIENT "1" "March 2016" "fail2ban-client v0.9.4" "User Commands"
+.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.3.
+.TH FAIL2BAN-CLIENT "1" "July 2016" "fail2ban-client v0.9.5" "User Commands"
 .SH NAME
 fail2ban-client \- configure and control the server
 .SH SYNOPSIS
 .B fail2ban-client
 [\fI\,OPTIONS\/\fR] \fI\,<COMMAND>\/\fR
 .SH DESCRIPTION
-Fail2Ban v0.9.4 reads log file that contains password failure report
+Fail2Ban v0.9.5 reads log file that contains password failure report
 and bans the corresponding IP addresses using firewall rules.
 .SH OPTIONS
 .TP
@@ -1,5 +1,5 @@
-.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.2.
-.TH FAIL2BAN-REGEX "1" "March 2016" "fail2ban-regex 0.9.4" "User Commands"
+.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.3.
+.TH FAIL2BAN-REGEX "1" "July 2016" "fail2ban-regex 0.9.5" "User Commands"
 .SH NAME
 fail2ban-regex \- test Fail2ban "failregex" option
 .SH SYNOPSIS
@@ -48,6 +48,9 @@ set custom pattern used to match date/times
 \fB\-e\fR ENCODING, \fB\-\-encoding\fR=\fI\,ENCODING\/\fR
 File encoding. Default: system locale
 .TP
+\fB\-r\fR, \fB\-\-raw\fR
+Raw hosts, don't resolve dns
+.TP
 \fB\-L\fR MAXLINES, \fB\-\-maxlines\fR=\fI\,MAXLINES\/\fR
 maxlines for multi\-line regex
 .TP
@@ -1,12 +1,12 @@
-.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.2.
-.TH FAIL2BAN-SERVER "1" "March 2016" "fail2ban-server v0.9.4" "User Commands"
+.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.3.
+.TH FAIL2BAN-SERVER "1" "July 2016" "fail2ban-server v0.9.5" "User Commands"
 .SH NAME
 fail2ban-server \- start the server
 .SH SYNOPSIS
 .B fail2ban-server
 [\fI\,OPTIONS\/\fR]
 .SH DESCRIPTION
-Fail2Ban v0.9.4 reads log file that contains password failure report
+Fail2Ban v0.9.5 reads log file that contains password failure report
 and bans the corresponding IP addresses using firewall rules.
 .PP
 Only use this command for debugging purpose. Start the server with
@@ -1,5 +1,5 @@
-.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.2.
-.TH FAIL2BAN-TESTCASES "1" "March 2016" "fail2ban-testcases 0.9.4" "User Commands"
+.\" DO NOT MODIFY THIS FILE!  It was generated by help2man 1.47.3.
+.TH FAIL2BAN-TESTCASES "1" "July 2016" "fail2ban-testcases 0.9.5" "User Commands"
 .SH NAME
 fail2ban-testcases \- run Fail2Ban unit-tests
 .SH SYNOPSIS
Author	SHA1	Message	Date
Yaroslav Halchenko	dca5ff44c2	Merge branch 'bf-common-zzz' * bf-common-zzz: BF: do not rely on long relative path to upstairs config - symlink common.conf	2016-07-14 22:25:07 -04:00
Yaroslav Halchenko	79616b76bf	Added missing files to MANIFEST	2016-07-14 22:19:10 -04:00
Yaroslav Halchenko	687ea8d333	BF: do not rely on long relative path to upstairs config - symlink common.conf	2016-07-14 22:15:22 -04:00
Yaroslav Halchenko	5714ac201b	DOC: preparations for 0.9.5 release	2016-07-14 21:35:49 -04:00
Yaroslav Halchenko	0f88e8a658	Added missing files to MANIFEST	2016-07-14 21:31:18 -04:00
Yaroslav Halchenko	72c4743666	Merge pull request #1481 from yarikoptic/doc-changelog Doc changelog - reformatted for markdown and a bit less indentation	2016-07-14 21:23:09 -04:00
Yaroslav Halchenko	21056c995d	Merge remote-tracking branch 'origin/master' into doc-changelog * origin/master: another variant of regex add trailing anchor to failregex add PR id to ChangeLog improved failregex according to @sebres recomendations * add `__prefix_line` to regex * fix time in log file add info to log file added sample log lines for slapd adding openldap slapd filter	2016-07-14 21:21:35 -04:00
Yaroslav Halchenko	28a0605f69	Merge pull request #1478 from gips0n/master adding openldap slapd filter	2016-07-14 08:30:42 -04:00
Andrii Melnyk	7433b353ee	another variant of regex	2016-07-14 10:19:21 +03:00
Andrii Melnyk	7c5828dd2a	add trailing anchor to failregex	2016-07-13 21:09:42 +03:00
Yaroslav Halchenko	33ed71b3de	DOC: Reformatted ChangeLog into legit Markdown (Closes #962 )	2016-07-10 19:53:54 -04:00
Yaroslav Halchenko	ec9c4a27f3	DOC: tuned up ChangeLog entries for 0.9.5 unified capitalized beginning of each entry no trailing spaces or dots etc	2016-07-10 18:40:20 -04:00
Andrii Melnyk	2c5a489bc7	add PR id to ChangeLog	2016-07-08 13:55:58 +03:00
Andrii Melnyk	48c094f612	improved failregex according to @sebres recomendations	2016-07-08 13:45:10 +03:00
sebres	de5d911fd5	Merge branch 'asterisk_pjsip_2'	2016-07-08 11:47:29 +02:00
sebres	f5f204ca7c	Improved changes of gh-1458: `[^']` after callid was wrong, changed to `[^\)]`; regexp anchored at the end; almost the same regex grouped to one; Closes #1458	2016-07-08 11:45:25 +02:00
nturcksin	72a157b8f2	Improve PJSIP log support for asterisk 13+ with different callID (Squash gh-1458) Change the asterisk pjsip filter to don't take the callId part Add optional part between "Request" and "from" Listed all log message from asterisk	2016-07-08 11:45:22 +02:00
Andrii Melnyk	dcb69b0242	* add `__prefix_line` to regex * fix time in log file	2016-07-08 05:29:51 +03:00
Andrii Melnyk	c335663395	add info to log file	2016-07-08 05:12:25 +03:00
Andrii Melnyk	c9ab669851	added sample log lines for slapd	2016-07-08 04:56:29 +03:00
Andrii Melnyk	b2e3affaa0	adding openldap slapd filter	2016-07-08 04:50:57 +03:00
Serg G. Brester	af8b650a37	badip timeout option introduced, set to 30 seconds in our test cases (#1463 ) cherry-picked from 0.10 (little bit modified in test_badips.py, because no --fast option in test cases)	2016-06-13 12:56:53 +02:00
Yaroslav Halchenko	11f7cf5ad8	DOC: changelog for recent exim filters tune up	2016-06-07 21:38:39 -04:00
Yaroslav Halchenko	636a93f58b	Merge pull request #1438 from yarikoptic/bf-exim exim filters -- make wider use of host_info helper str susbstitution + fix for #1430	2016-06-07 21:35:52 -04:00
Ludovic Gasc	f85fb45b29	Asterisk pjsip (#1456 ) * Improve PJSIP log support for Asterisk 13+ * Update changelog: filter.d/asterisk.conf - fix security log support for PJSIP and Asterisk 13+ * Change pjsip regexp with sebres observation, thanks to @nturcksin	2016-06-07 11:40:35 +02:00
Yaroslav Halchenko	ced6c8307b	BF: finalize that sample log line for exim4 was intended in `743a531eb5` to be an entry without a port after the [host]	2016-06-02 20:57:27 -04:00
sebres	b85347477f	Amend merge for pull request #1454	2016-05-31 17:47:12 +02:00
sebres	b3bb8b20bf	amend for new option of `usedns=raw` - forgotten validation fix inside setUseDns	2016-05-31 17:45:44 +02:00
Serg G. Brester	f23ccbda35	Merge pull request #1454 from sebres/gh-67-ban-wo-host [gh-67] no-host ban	2016-05-31 16:07:06 +02:00
Yaroslav Halchenko	6434661480	RF: for consistency use (?:XXX)? instead of (?:\|XXX)	2016-05-30 12:12:53 -04:00
Yaroslav Halchenko	48a8324662	ENH: use non-capturing regex groups in exim-common and exim filters	2016-05-30 11:02:12 -04:00
sebres	8ec4e1189e	use raw host (don't use textToIp) if usedns exactly `raw`, because `usedns = no` should ignore no ip failures	2016-05-30 15:34:21 +02:00
sebres	c33e25bab6	fail2ban-regex: extended with option '--raw' (switch to raw host, prevent resolving of ip)	2016-05-30 14:08:37 +02:00
sebres	b7787f4af4	use raw host (don't use textToIp) if usedns not yes or warn	2016-05-30 14:08:33 +02:00
Yaroslav Halchenko	e01cd8ab03	Merge pull request #1444 from yarikoptic/enh-courier-username ENH: courier-smtp -- allow for trailing username (no spaces) in the logline	2016-05-26 19:26:35 -04:00
sebres	858c5c0d00	Merge branch 'gh-1417'	2016-05-26 11:14:09 +02:00
sebres	a80043ce80	amend for gh-1419: tags substitution bug - wrong recognition of cyclic recursion, new test cases covered this	2016-05-26 11:13:33 +02:00
sebres	9df7973e95	Merge pull request #1439 from yarikoptic/enh-ignoreipsplit: ENH: splitcommaspace -> splitwords allow to split ignoreip entries with new lines	2016-05-23 15:35:41 +02:00
sebres	156065e70d	splitwords: prevent to split to empty values by multiple separator characters together	2016-05-23 15:33:45 +02:00
Yaroslav Halchenko	9bb869b8d4	ENH: courier-smtp -- allow for trailing username (no spaces) in the logline Closes #1440	2016-05-21 22:17:09 -04:00
Yaroslav Halchenko	01d0506ea0	ENH: splitcommaspace -> splitwords allow to split ignoreip entries with new lines Closes #1432	2016-05-21 10:55:27 -04:00
Yaroslav Halchenko	8b8cf2a660	ENH: exim filters -- make more use of %(host_info)s which in turn made more flexible	2016-05-21 10:29:09 -04:00
Yaroslav Halchenko	743a531eb5	BF: make :port and I=[ip]:port optional for a "AUTH command used when not advertised" Closes #1430	2016-05-21 10:29:01 -04:00
Serg G. Brester	b56f4c533e	Merge pull request #1436 from sebres/speedup-db extremely speedup of all database operations	2016-05-21 13:39:48 +02:00
sebres	1718c8dbe9	pypy: switch journal mode after upgrade (save it during the upgrade), to prevent errors like "database table is locked"	2016-05-20 15:12:32 +02:00
sebres	baafac36a4	ChangeLog entry	2016-05-20 14:51:13 +02:00
sebres	db9e724038	extremely speedup of all database operations: - (synchronous = OFF) write data through OS without syncing - (journal_mode = MEMORY) use memory for the transaction logging	2016-05-20 12:06:04 +02:00
sebres	34ae0b916e	Merge pull request #1421 from sebres/fix-1405 filter.d/common.conf: fixes unexpected extra regex-space in generic `__prefix_line` (gh-1405)	2016-05-20 11:20:34 +02:00
sebres	caecb2080b	Merge pull request #1435 from sebres/fix-1434 fixed --pidfile bug, introduced in gh-1322	2016-05-20 11:02:42 +02:00
sebres	932708de9e	fixed --pidfile bug, introduced in gh-1322: gentoo-initd fixed --pidfile bug: `--pidfile` is option of start-stop-daemon, not argument of fail2ban (see gh-1434) closes gh-1434	2016-05-20 11:01:00 +02:00
sebres	52377984cd	back to mandatory space, ungrouping of sub parameters in `__prefix_line` + small code review;	2016-05-19 17:57:48 +02:00
sebres	25af11215b	test case for generic common moved to `./fail2ban/tests/config/filter.d/zzz-generic-example.conf` to prevent shipping it with fail2ban installations	2016-05-17 20:08:46 +02:00
sebres	cb4f9be8b2	the date brackets removed from filters using `__prefix_line`, because `__prefix_line` already contains the date ambit;	2016-05-17 11:55:02 +02:00
sebres	de813acf51	extends generic `__prefix_line` with optional brackets for the date ambit (gh-1421), added new parameter `__date_ambit` + test case added;	2016-05-17 11:54:43 +02:00
sebres	3e49522b7a	fixes unexpected extra regex-space in generic `__prefix_line` (gh-1405, misleadingly committed in `d2a9537568`); all optional spaces normalized in generic include `common.conf` + test cases are extended (using new example pseudo-filter and test log `zzz-generic-example`);	2016-05-13 20:26:37 +02:00
Yaroslav Halchenko	d6eae28eb5	Merge pull request #1419 from sebres/gh-1417 fixes gh-1417 Grave tags substitution bug. [part. cherry-picked from 0.10]	2016-05-13 10:28:29 -04:00
sebres	cce63926ce	ChangeLog entry added	2016-05-13 16:11:38 +02:00
sebres	a4b8f6e49e	[part. cherry-picked from 0.10] invalid recursion check in substituteRecursiveTags: for example action `bsd-ipfw` produced ValueError('properties contain self referencing definitions and cannot be resolved...') test cases extended for exactly this case; closes gh-1417	2016-05-13 14:12:17 +02:00
sebres	bdc2d07946	fix suhosin_log in common paths - log files should be separated using "\n": prevents to throw an error "File option must be 'head' or 'tail'", if jail suhosin will be enabled.	2016-05-11 18:49:04 +02:00