Another ModSecurity Development Release

Last week I released the second ModSecurity development release, 2.5.0-dev2, in preparation for the next version of ModSecurity. Some may notice that this version is now 2.5.x where as the first development release was 2.2.x. Ivan and I decided that because of the large feature sets going into this next release we would bump the version to 2.5.x to signify a halfway point to 3.0.0. This release is primarily a performance enhancement release, so I want to introduce two of the main new features in more detail.

In this release, I have introduced a phrase matching operator (@pm) to match against a list of phrases. The new operator uses the Aho-Corasick algorithm and is many times faster than the default regular expression operator (@rx) for lists of OR'd phrases. For example, if you want to accept only GET, POST and HEAD requests the following rules are equivalent, but the second is faster (even more so as the list grows):

SecRule REQUEST_METHOD "!^(?:GET|POST|HEAD)$" t:none,deny
SecRule REQUEST_METHOD "!@pm GET POST HEAD" t:none,deny

The new @pm operator should be used for static lists of phrases (black/white lists, spam keywords, etc). However, for large lists, this new operator may cause the rule to become difficult to read and maintain. If your lists are large, you can use an alternate form (@pmFromFile) that accepts a list of files and place the phrases into a file or multiple files (one per line) instead of inline. In this form, the phrase file(s) will be read on startup. To allow for easy inclusion of third party phrase lists, if the filename is given with a relative path, then ModSecurity will look for it starting in the same directory as the file that contains the rule specifying the file. For example:

SecRule REQUEST_METHOD "!@pmFromFile allowed_methods.txt" t:none,deny

### And in allowed_methods.txt in the same directory:
GET
POST
HEAD

Another performance enhancement (that is still being tuned) is transformations are now cached for each transaction. With previous versions of ModSecurity, the transformations for each rule were applied in the order specified to the original value. This was done for every variable in every rule. Starting with ModSecurity 2.5.0-dev2, transformations will only be performed once for each transaction. If more than one rule uses the same transformed value, then the cached value is used instead of reapplying the transformations.

As always, the source code is available in the download section of the ModSecurity Website. Below is an outline of the new features and changes in this release so far. Please see the documentation included in the release for full details and usage examples. Please direct any comments to the ModSecurity User Support mailing list.

Major Changes Since Version 2.1.1

2.5.0-dev2

• Added new phrase matching operators, @pm and @pmFromFile. These use an alternate set based matching engine (Aho-Corasick) to perform faster phrase type matches such as black/white lists, spam keywords, etc.
• Cache transformations per-request/phase so they are not repeated.
• Added @within string comparison operator with support for macro expansion. This was discussed as @contained on the user list, but this seemed too similar to @contains. This operator looks for a variable value within the operator value (opposite of @contains which looks for the operator value in the variable value).
• Fixed issue with requests that use internal requests. These had the potential to be intercepted incorrectly when other Apache httpd modules that used internal requests were used with mod_security.
• Fixed decoding full-width unicode in t:urlDecodeUni.
• Added matching rule filename and line number to audit log so that it is much easier to figure out which rule caused an alert to be generated.
• Enhanced PDF protection allows a choice of forced downloads (of PDF files) or use of token redirection.
• Various other bug fixes and smaller changes. See the CHANGES file and/or the ModSecurity Reference Manual in the release for more details.

2.2.0-dev1

• Experimental support for content injection. You can use @prepend value and @append value to inject content into the output.
• PDF XSS protection added.
• Geographical lookups by IP address/hostname were added. With this you can access the country, city, postal code, etc. -- within a new GEO collection -- from any existing ModSecurity variable by using the new @geoLookup operator.
• String comparison operators -- @streq, @beginsWith and @endsWith -- to allow an easier-to-use non-regular expression operator. Values will have macros interpreted prior to a match, so that you can do "@streq %{REMOTE_ADDR}", etc.
• A length transformation -- t:length -- transforms a value into a numeric character length.
• Whitespace trimming transformations are now available. Use t:trimLeft, t:trimRight or t:trim to trim whitespace from the left, right or both, respectively.
• Experimental variables -- RESPONSE_CONTENT_LENGTH, RESPONSE_CONTENT_TYPE, and RESPONSE_CONTENT_ENCODING -- were added.
• Added the filename and line number of the rule to the debug log on its execution. I will add it to the audit log in the next development release.
• Removed support for CGI style HTTP_HEADER_NAME variables for request headers. The 2.x REQUEST_HEADERS:Header-Name should now be used.
• Removed support for %0 - %9 TX variables that expanded to numbered captures. Use %{TX.0} - %{TX.9} instead.
• Various other bug fixes and smaller changes. See the CHANGES file in the release for more details.