Implementing AppSensor Detection Points in ModSecurity

This is a follow-up to a previous blog post entitled "Real-time Application Profiling" that implements extended profiling logic using the ModSecurity Lua API.

AppSensor Detection Points

SpiderLabs Research Team is happy to announce that we have just updated the OWASP ModSecurity Core Rule Set SVN repository with new application profiling rules that implement the OWASP AppSensor's Request Exception Detection Points:

2.1 RequestException

• 2.1.1 RE1: Unexpected HTTP Command
• 2.1.2 RE2: Attempt to Invoke Unsupported HTTP Method
• 2.1.3 RE3: GET When Expecting POST
• 2.1.4 RE4: POST When Expecting GET
• 2.1.5 RE5: Additional/Duplicated Data in Request
• 2.1.6 RE6: Data Missing from Request
• 2.1.7 RE7: Unexpected Quantity of Characters in Parameter
• 2.1.8 RE8: Unexpected Type of Characters in Parameter

Implementing AppSensor Detection Points in ModSecurity

We have just released a ruleset to the OWASP CRS SVN repository called modsecurity_crs_40_appsensor_detection_point_2.1_request_exception.conf. This ruleset is currently available in the /experimental directory of the CRS and we are hoping that the community will help us with testing and feedback.

This version of the rules has the ability to profile and enforce the following on a per-resource basis:

• Request Method(s)
• Number of Parameters (min/max range)
• Parameter Names
• Parameter Lengths (min/max range)
• Parameter Types
  • Flag (e.g. - /path/to/foo.php?param)
  • Digits (e.g. - /path/to/foo.php?param=1234)
  • Alpha (e.g. - /path/to/foo.php?param=abcd)
  • AlphaNumeric (e.g. - /path/to/foo.php?param=abcd1234)
  • Email (e.g. - /path/to/foo.php?param=foo@bar.com)
  • Path (e.g. - /path/to/foo.php?param=/dir/somefile.txt)
  • URL (e.g. - /path/to/foo.php?param=http://somehost/dir/file.txt)
  • SafeText (e.g. - /path/to/foo.php?param=some_data-12)

Profiling Methodology

Let's take a look at the profiling methodology of the new rules.

Resource-based Persistent Collections

The key capability that we are leveraging to achieve our profiling rules is that we need to have per-resource persistent storage so that we can correlate data both across requests and clients. We do this by using initializing a Resource persistent storage mechanism in phase:1 and we use the macro expanded REQUEST_HEADERS:Host and REQUEST_FILENAME variables as the collection key.

## --[ Step 1: Initiate the Resource Collection ]--## We are using the REQUEST_FILENAME as the key and then set 2 variables -## [resource.pattern_threshold]# Set the resource.pattern_threshold as the minimum number of times that a match should occur # in order to include it into the profile## [resource.confidence_counter_threshold] # Set the resource.confidence_counter_threshold as the minimum number of "clean" transactions# to profile/inspect before enforcement of the profile begins.#SecAction "phase:1,id:'981082',t:none,nolog,pass,initcol:resource=%{request_headers.host}_%{request_filename},setvar:resource.min_pattern_threshold=5,setvar:resource.min_traffic_threshold=10"

The two setvar actions are used to determine the number of transactions we want to profile and how many times our individual checks must match before we add them to the enforcement list.

Post-Process Profiling

In order to mimize the impact (latency) of the profiling rules, we chose to implement these checks in phase:5 (logging). Before we decide whether or not to profile the transaction, we want to do a few security checks to ensure that we are only looking at "clean" transactions.

## --[ Begin Profiling Phase ]--#SecMarker BEGIN_RE_PROFILE_ANALYSISSecAction "phase:5,id:'981098',t:none,nolog,pass,ctl:ruleEngine=DetectionOnly"SecRule RESPONSE_STATUS "^404$" "phase:5,id:'981099',t:none,nolog,pass,setvar:!resource.KEY,skipAfter:END_RE_PROFILE_ANALYSIS"SecRule RESPONSE_STATUS "^(5|4)" "phase:5,id:'981100',t:none,nolog,pass,skipAfter:END_RE_PROFILE_ANALYSIS"SecRule TX:ANOMALY_SCORE "!@eq 0" "phase:5,id:'981101',t:none,nolog,pass,skipAfter:END_RE_PROFILE_ANALYSIS"SecRule &RESOURCE:ENFORCE_RE_PROFILE "@eq 1" "phase:2,id:'981102',t:none,nolog,pass,skipAfter:END_RE_PROFILE_ANALYSIS"SecRuleScript /usr/local/apache/conf/crs/lua/appsensor_request_exception_profile.lua "phase:5,nolog,pass"SecMarker END_RE_PROFILE_ANALYSIS

There are basically four different scenarios where we don't want to profile and these are handled by the first 4 SecRules:

• We use the ctl action to change the ruleEngine to DetectionOnly as we don't want the operator matching to stop the rule processing. This enables us to profile all ARGS.
• If the HTTP response code is a 404, then the resource doesn't exist. In this case, not only do we skip the profiling, but we also remove the resource key so we delete the persistent storage.
• If the HTTP response code is a level 4xx or 5xx, then the application says something is wrong with the transaction so we won't profile it either.
• If the CRS anomaly score is anything other than 0, then we won't profile as this means that one of the negative security rules have triggered.
• Finally, if we have moved into Enforcement mode, then we will stop profiling.

So, if all of these pre-qualifier checks are passed, we then move to profiling the request attributes. This is triggered by the SecRuleScript line that executes the appsensor_request_exception_profile.lua script.

Example Resource Collection Data

To test out the profiling, I sent the following requests to my local webserver using the Apache "ab" tool -

$ ab -n 50 -c 10 "http://localhost/cgi-bin/printenv?param1=123456&param2=abcd"$ ab -n 50 -c 10 "http://localhost/cgi-bin/printenv?param1=1234567890&param2=abcdefgh&param3=/somefile.txt&param4=foo@bar.com"

These requests will cause the Lua script to profile the parameter characteristics where there are a range of parameters. Once the number of transactions profiled has reached the confidence counter threshold (in this case 100 transactions), we then add the enforcement variable to the resource collection from within the Lua script which then moves the rules from profiling to enforcement mode. After the profiling phase has been completed, the data saved in the Resource collection will look something like this:

Retrieved collection (name "resource", key "localhost_/cgi-bin/printenv").Delta applied for resource.UPDATE_COUNTER 100->101 (1): 100 + (1) = 101 [101,3]Wrote variable: name "__expire_KEY", value "1314988412".Wrote variable: name "KEY", value "localhost_/cgi-bin/printenv".Wrote variable: name "TIMEOUT", value "3600".Wrote variable: name "__key", value "localhost_/cgi-bin/printenv".Wrote variable: name "__name", value "resource".Wrote variable: name "CREATE_TIME", value "1314984744".Wrote variable: name "UPDATE_COUNTER", value "101".Wrote variable: name "min_pattern_threshold", value "50".Wrote variable: name "min_traffic_threshold", value "100".Wrote variable: name "traffic_counter", value "100".Wrote variable: name "NumOfArgs_counter_2", value "50".Wrote variable: name "ARGS:param1_length_6_counter", value "50".Wrote variable: name "ARGS:param2_length_4_counter", value "50".Wrote variable: name "LAST_UPDATE_TIME", value "1314984812".Wrote variable: name "enforce_request_methods", value "GET".Wrote variable: name "enforce_num_of_args", value "2, 4".Wrote variable: name "enforce_args_names", value "param1, param2, param3, param4".Wrote variable: name "enforce_ARGS:param1_length", value "6, 10".Wrote variable: name "enforce_ARGS:param2_length", value "4, 8".Wrote variable: name "enforce_charclass_digits", value "ARGS:param1".Wrote variable: name "enforce_charclass_alphas", value "ARGS:param2".Wrote variable: name "MinNumOfArgs", value "2".Wrote variable: name "MaxNumOfArgs", value "4".Wrote variable: name "ARGS:param1_length_min", value "6".Wrote variable: name "ARGS:param1_length_max", value "10".Wrote variable: name "ARGS:param2_length_min", value "4".Wrote variable: name "ARGS:param2_length_max", value "8".Wrote variable: name "enforce_ARGS:param3_length", value "13".Wrote variable: name "ARGS:param3_length_min", value "13".Wrote variable: name "ARGS:param3_length_max", value "13".Wrote variable: name "enforce_ARGS:param4_length", value "11".Wrote variable: name "ARGS:param4_length_min", value "11".Wrote variable: name "ARGS:param4_length_max", value "11".Wrote variable: name "enforce_charclass_path", value "ARGS:param3".Wrote variable: name "enforce_charclass_email", value "ARGS:param4".Wrote variable: name "enforce_re_profile", value "1".Persisted collection (name "resource", key "localhost_/cgi-bin/printenv").

As you can see, we now have a number of request attributes that we can now enforce.

Enforcement Examples

We can now send a series of requests to test out each aspect of the learned profile for the resource:

Request Method Enforcement

Test by sending a different request method such as PUT -

$ curl -X PUT http://localhost/cgi-bin/printenv

This results in the following alert message:

[Wed Aug 31 16:42:31 2011] [error] [client 127.0.0.1] ModSecurity: Warning. Operator EQ matched 1 at TX:request_method_violation. [file "/usr/local/apache/conf/crs/activated_rules/modsecurity_crs_40_appsensor_detection_point_2.1_request_exception.conf"] [line "37"] [id "981088"] [msg "Invalid Request Method for Resource."] [data "Current Request Method: PUT and Allowed Request Method(s): GET"] [tag "POLICY/METHOD_NOT_ALLOWED"] [tag "OWASP_AppSensor/RE1"] [tag "https://www.owasp.org/index.php/AppSensor_DetectionPoints#RE1:_Unexpected_HTTP_Command"] [hostname "localhost"] [uri "/cgi-bin/printenv"] [unique_id "Tl6ct8CoAWYAAKsUIH4AAAAC"]

Note that in the alert message we have the tag data mapped to the corresponding AppSensor Detection Point - RE1: Unexpected HTTP Command.

Number of Parameters

By using the Lua script, we are able to learn the valid parameter number ranges:

Wrote variable: name "MinNumOfArgs", value "2".Wrote variable: name "MaxNumOfArgs", value "4".

We can and test by sending only 1 parameter -

$ curl "http://localhost/cgi-bin/printenv?param1=123456"

This results in the following alert message:

[Wed Aug 31 16:48:09 2011] [error] [client 127.0.0.1] ModSecurity: Warning. Operator EQ matched 1 at TX:MIN_NUM_ARGS_VIOLATION. [file "/usr/local/apache/conf/crs/activated_rules/modsecurity_crs_40_appsensor_detection_point_2.1_request_exception.conf"] [line "45"] [id "981089"] [msg "Invalid Number of Parameters - Missing Parameter(s)"] [data "Min Number of ARGS: 2 and Number of ARGS Submitted: 1"] [tag "POLICY/PARAMETER_VIOLATION"] [tag "OWASP_AppSensor/RE5"] [tag "https://www.owasp.org/index.php/AppSensor_DetectionPoints#RE5:_Additional.2FDuplicated_Data_in_Request"] [hostname "localhost"] [uri "/cgi-bin/printenv"] [unique_id "Tl6eCcCoAWYAAKsVIIEAAAAD"]

Parameter Names

Test by changing a parameter name -

$ curl "http://localhost/cgi-bin/printenv?param1=123456&new_param=foo"

This results in the following alert message:

[Wed Aug 31 16:50:29 2011] [error] [client 127.0.0.1] ModSecurity: Warning. Pattern match ".*" at TX:args_names_violation. [file "/usr/local/apache/conf/crs/activated_rules/modsecurity_crs_40_appsensor_detection_point_2.1_request_exception.conf"] [line "49"] [id "981091"] [msg "Invalid Parameter Name(s)."] [data "ARGS_NAMES:new_param"] [tag "POLICY/PARAMETER_VIOLATION"] [tag "OWASP_AppSensor/RE5"] [tag "https://www.owasp.org/index.php/AppSensor_DetectionPoints#RE5:_Additional.2FDuplicated_Data_in_Request"] [hostname "localhost"] [uri "/cgi-bin/printenv"] [unique_id "Tl6elcCoAWYAAKsWILkAAAAE"]

Parameter Length Checks

Similar to the number of parameter check, the Lua script allows us to learn size ranges of each individual parmeter. For example, for the "param1" parameter, we have learned the following:

Wrote variable: name "ARGS:param1_length_min", value "6".Wrote variable: name "ARGS:param1_length_max", value "10".

This means that this parmeter length should be between "6" and "10" characters. We can test by changing the parameter length -

$ curl "http://localhost/cgi-bin/printenv?param1=123456789012345678&param2=abcd"

This results in the following alert message:

[Wed Aug 31 16:52:33 2011] [error] [client 127.0.0.1] ModSecurity: Warning. Pattern match ".*" at TX:ARGS:param1_max_length_violation. [file "/usr/local/apache/conf/crs/activated_rules/modsecurity_crs_40_appsensor_detection_point_2.1_request_exception.conf"] [line "61"] [id "981093"] [msg "Invalid Parameter Length - Value Is Above Normal Range"] [data "Normal Maximum Length for Parameter (ARGS:param1): 10 and Current Length: 18"] [tag "POLICY/PARAMETER_VIOLATION"] [tag "OWASP_AppSensor/RE7"] [hostname "localhost"] [uri "/cgi-bin/printenv"] [unique_id "Tl6fEcCoAWYAAKsYIMcAAAAF"]

Parameter Type Checks

Test by changing the type for param1 from digits to include meta-characters (such as the single tick mark to test for SQL Injection flaws) -

$ curl "http://localhost/cgi-bin/printenv?param1=123456'&param2=abcd"

This results in the following alert message:

[Wed Aug 31 16:55:33 2011] [error] [client 127.0.0.1] ModSecurity: Warning. Pattern match ".*" at TX:ARGS:param1_digits_violation. [file "/usr/local/apache/conf/crs/activated_rules/modsecurity_crs_40_appsensor_detection_point_2.1_request_exception.conf"] [line "74"] [id "981094"] [msg "Invalid Character(s) in Payload - Expecting Digits."] [data "Parameter (ARGS:param1): 123456'"] [tag "OWASP_AppSensor/RE8"] [hostname "localhost"] [uri "/cgi-bin/printenv"] [unique_id "Tl6fxcCoAWYAAKsZIOkAAAAG"]

Current Limitations

While this ruleset is a great first step in the profiling direction, it is not perfect. The main limitation with this ruleset is that here is no auto-relearning. Once the rules have moved from profiling into enforcement, they stay in that mode. This means that if you have a legitimate code push, then you will probably have to remove the resource collections and then the profiling will restart.

Depending on the amount of data you need to profiler per resource, you may run into SDBM persistent storage size limits. By default, you have ~800 bytes of usable storage. If you run into this issue, you will see this error message:

[1] Failed to write to DBM file "/tmp/IP": Invalid argument

If you need more than that, you should download the separate APR and APR-Util packages then edit the "#define PAIRMAX" setting in the /dbm/sdbm/sdbm_private.h file. You should then recompile both Apache and ModSecurity referencing the updated APR/APR-Util packages.

If you want to exclude certain URLs from profiling all together, we have some rules commented out that you can activate. They will do an @pmFromFile check against an external file. This will allow you to add URLs to be exluded to this list file.

Future Directions

We are planning on adding in more AppSensor Detection Points in the near future. The next section we will tackle is the Session Exception category which will profile Cookie data.

Please help us out by testing these rules and providing feedback on the OWASP ModSecurity Core Rule Set mail-list.