Detecting Malice with ModSecurity: Request Method Anomalies

This week's installment of Detecting Malice with ModSecurity will discuss how to detect HTTP Request Method Anomalies. HTTP Proxies excerpt section of Robert "Rsnake" Hansen's book "Detecting Malice" -

One of the most commonly talked about things in HTTP is the request method (often called a verb), which describes what action a HTTP client wishes to take.

The request method, together with the protocol information, determine how the server will interpret a request.

Request Methods

HTTP 1.1 defines several request methods: GET, POST, PUT, DELETE, OPTIONS, CONNECT, HEAD, and TRACE. There are other legal request methods, because HTTP also allows new methods to be added as needed. WebDAV, which is an extension of HTTP, adds quite a few request methods of its own.

Common HTTP Request Methods

The HTTP v1.1 RFC lists the following common Request Methods:

• OPTIONS
• GET
• HEAD
• POST
• PUT
• DELETE
• TRACE
• CONNECT

The most common request methods used by web sites are GET, HEAD and POST. If you are using an application which uses WebDAV (such as Microsoft's Outlook Web Access - OWA) then you may have to deal with additional Request Methods such as:

• PROPFIND
• PROPPATCH
• MKCOL
• BDELETE
• BMOVE
• BCOPY
• POLL
• SUBSCRIBE
• SEARCH

As you might expect, there are a number of security concerns related to the use of these various request methods and you should certainly understand when, where and how to use them. Deciding on which request methods to allow is beyond the scope of this post however we will highlight how a site admin can use ModSecurity to help identify request method anomalies.

Restricting Allowed Request Methods

In the OWASP ModSecurity Core Rule Set (CRS), an administrator has a number of different methods that they can use to allow the proper request methods for their applications.

Site Level Restrictions

In the modsecurity_crs_10_config.conf.example file in the CRS, the following section allows the user to specify which HTTP request methods they want to allow for their entire site:

## -=[ HTTP Policy Settings ]=-## Set the following policy settings here and they will be propagated to the 30 rules# file (modsecurity_crs_30_http_policy.conf) by using macro expansion.  # If you run into false positves, you can adjust the settings here.#SecAction "phase:1,t:none,nolog,pass, \setvar:'tx.allowed_methods=GET HEAD POST OPTIONS', \setvar:'tx.allowed_request_content_type=application/x-www-form-urlencoded multipart/form-data text/xml application/xml application/x-amf', \setvar:'tx.allowed_http_versions=HTTP/0.9 HTTP/1.0 HTTP/1.1', \setvar:'tx.restricted_extensions=.asa/ .asax/ .ascx/ .axd/ .backup/ .bak/ .bat/ .cdx/ .cer/ .cfg/ .cmd/ .com/ .config/ .conf/ .cs/ .csproj/ .csr/ .dat/ .db/ .dbf/ .dll/ .dos/ .htr/ .htw/ .ida/ .idc/ .idq/ .inc/ .ini/ .key/ .licx/ .lnk/ .log/ .mdb/ .old/ .pass/ .pdb/ .pol/ .printer/ .pwd/ .resources/ .resx/ .sql/ .sys/ .vb/ .vbs/ .vbproj/ .vsdisco/ .webinfo/ .xsd/ .xsx/', \setvar:'tx.restricted_headers=/Proxy-Connection/ /Lock-Token/ /Content-Range/ /Translate/ /via/ /if/'"

The bolded section is where the local admin can specify which request methods they want to allow for their entire site. In this case, the default setting is to allow GET, HEAD, POST and OPTIONS.

This configuration data is then used in the following check within the modsecurity_crs_30_http_policy.conf file:

# allow request methods## TODO Most applications only use GET, HEAD, and POST request#      methods. If that is not the case with your environment, you are advised #      to edit the line or uncomment it.#SecRule REQUEST_METHOD "!@within %{tx.allowed_methods}" "phase:2,t:none,block,msg:'Method is not allowed by policy', severity:'2',id:'960032',tag:'POLICY/METHOD_NOT_ALLOWED',tag:'WASCTC/WASC-15',tag:'OWASP_TOP_10/A6',tag:'OWASP_AppSensor/RE1',tag:'PCI/12.1',logdata:'%{matched_var}',setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+%{tx.warning_anomaly_score},setvar:tx.policy_score=+%{tx.warning_anomaly_score},setvar:tx.%{rule.id}-POLICY/METHOD_NOT_ALLOWED-%{matched_var_name}=%{matched_var}" 

This rule will check the REQUEST_METHOD used in the current transaction and will trigger if it is now found within the TX.ALLOWED_METHODS list defined by the admin.

Here is an example request that uses the DELETE method:

DELETE /upload.html HTTP/1.1User-Agent: curl/7.21.1 (x86_64-apple-darwin10.4.0) libcurl/7.21.1 OpenSSL/1.0.0a zlib/1.2.5 libidn/1.19Host: www.example.comAccept: */*

Here is the alert that is generated by the request method validation rule:

[Tue Apr 26 12:26:01 2011] [error] [client ::1] ModSecurity: Warning. Match of "within %{tx.allowed_methods}" against "REQUEST_METHOD" required. [file "/usr/local/apache/conf/crs/base_rules/modsecurity_crs_30_http_policy.conf"] [line "30"] [id "960032"] [msg "Method is not allowed by policy"] [data "DELETE"] [severity "CRITICAL"] [tag "POLICY/METHOD_NOT_ALLOWED"] [tag "WASCTC/WASC-15"] [tag "OWASP_TOP_10/A6"] [tag "OWASP_AppSensor/RE1"] [tag "PCI/12.1"] [hostname "www.example.com"] [uri "/upload.html"] [unique_id "TbbyGcCoqAEAAJBLL7EAAAAB"]

Invalid Request Methods

This type of validation will not only catch legitimate HTTP Request Methods that you don't use within your site, but it will also catch invalid request methods. These types of request may hold binary data and/or be written in lowercase (example - get vs. GET).

Oftentimes these types of request are a request of fingerprinting tools such as HTTPrint. In order understand how HTTPrint works, I suggest that you read this supporting information.

There are many different possibilities for mitigating the effectiveness of these types of fingerprinting scanners. For complete information, I suggest you read the http fingerprinting Appendix section I wrote as part of the WASC Threat Classification document.

HTTPrint a not a typical "banner grabbing" application, as it has more logic to it. It's main fingerprinting technique has to do with the Sematic differences in how web servers/applications respond to various stimuli. Let's take a look at how it looks in the Apache access_log file when httprint is run -

127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "\x16\x03" 501 214127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "GET / HTTP/1.0" 200 44127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "GET / HTTP/1.0" 200 44127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "OPTIONS * HTTP/1.0" 200 -127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "OPTIONS / HTTP/1.0" 200 -127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "GET /antidisestablishmentarianism HTTP/1.0" 404 226127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "PUT / HTTP/1.0" 405 231127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "JUNKMETHOD / HTTP/1.0" 501 222127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "GET / JUNK/1.0" 200 44127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "get / http/1.0" 501 215127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "POST / HTTP/1.0" 200 44127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "GET /cgi-bin/ HTTP/1.0" 403 210127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "GET /scripts/ HTTP/1.0" 404 206127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "GET / HTTP/0.8" 200 44127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "GET / HTTP/0.9" 200 44127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "GET / HTTP/1.1" 200 44127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "GET / HTTP/1.1" 400 226127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "GET / HTTP/1.2" 200 44127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "GET / HTTP/1.2" 400 226127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "GET / HTTP/3.0" 200 44127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "GET /.asmx HTTP/1.1" 404 203127.0.0.1 - - [26/Apr/2011:12:40:32 -0400] "GET /../../ HTTP/1.0" 400 226  

The bolded entries show some examples of httprint sending invalid request methods so that it can see how the web server responds. ModSecurity generated events for each of these requests:

[Tue Apr 26 12:40:32 2011] [error] [client 127.0.0.1] ModSecurity: Warning. Match of "within %{tx.allowed_methods}" against "REQUEST_METHOD" required. [file "/usr/local/apache/conf/crs/base_rules/modsecurity_crs_30_http_policy.conf"] [line "30"] [id "960032"] [msg "Method is not allowed by policy"] [data "\\x16\\x03"] [severity "CRITICAL"] [tag "POLICY/METHOD_NOT_ALLOWED"] [tag "WASCTC/WASC-15"] [tag "OWASP_TOP_10/A6"] [tag "OWASP_AppSensor/RE1"] [tag "PCI/12.1"] [hostname "macbook-pro.local"] [uri "/"] [unique_id "Tbb1gMCoqAEAAJBOMCsAAAAE"][Tue Apr 26 12:40:32 2011] [error] [client 127.0.0.1] ModSecurity: Warning. Match of "within %{tx.allowed_methods}" against "REQUEST_METHOD" required. [file "/usr/local/apache/conf/crs/base_rules/modsecurity_crs_30_http_policy.conf"] [line "30"] [id "960032"] [msg "Method is not allowed by policy"] [data "JUNKMETHOD"] [severity "CRITICAL"] [tag "POLICY/METHOD_NOT_ALLOWED"] [tag "WASCTC/WASC-15"] [tag "OWASP_TOP_10/A6"] [tag "OWASP_AppSensor/RE1"] [tag "PCI/12.1"] [hostname "localhost"] [uri "/"] [unique_id "Tbb1gMCoqAEAAJmIUlIAAAAF"][Tue Apr 26 12:40:32 2011] [error] [client 127.0.0.1] ModSecurity: Warning. Match of "within %{tx.allowed_methods}" against "REQUEST_METHOD" required. [file "/usr/local/apache/conf/crs/base_rules/modsecurity_crs_30_http_policy.conf"] [line "30"] [id "960032"] [msg "Method is not allowed by policy"] [data "get"] [severity "CRITICAL"] [tag "POLICY/METHOD_NOT_ALLOWED"] [tag "WASCTC/WASC-15"] [tag "OWASP_TOP_10/A6"] [tag "OWASP_AppSensor/RE1"] [tag "PCI/12.1"] [hostname "localhost"] [uri "/"] [unique_id "Tbb1gMCoqAEAAJBKL8UAAAAA"]

Per Resource Restrictions

While the rules we have shown thus far certainly help to identifiy overtly anomalous request method usage, there are still issues when unexpected but legitimate request methods are used with resources. For instance, if a resource normally receives POST requests but suddenly clients start sending GET requests, you need a mechanism to identify that anomaly. From Detecting Malice:

On many websites that are vulnerable to cross site request forgeries (CSRF), one feature of many websites is that an attacker can turn a POST request into a GET request, simply by sending the same information in a query string. Take the following example of a simple POST request:

POST /contact.asp HTTP/1.1 Host: www.yoursite.com Content-Type: application/x-www-form-urlencoded Content-Length: 27contact=test&submit=Submit

The GET request equivalent would then be:

GET /contact.asp?contact=test&submit=Submit HTTP/1.1 Host: www.yoursite.com

There is really no reason to see a POST to GET conversion like this to occur in nature. If you were to see this in your logs it would point to a technical person or a user who is being subverted to click on a link. Clearly someone is doing something either subversive or trying to take a short cut by sending the request directly instead of using the POST submission page you built for that purpose.

In order to detect these types of per-resource request method anomalies, we need to use use some application profiling checks. The OWASP AppSensor project specifies the following Detection Point:

RE1: Unexpected HTTP Command

This check has been implemented in the latest version of the OWASP ModSecurity CRS in the experimental_rules/modsecurity_crs_40_appsensor_detection_point_2.1_request_exception.conf file.

First we have a section that learns the expected Request Method for each resource:

## Check REQUEST_METHOD #SecRule REQUEST_METHOD "@within %{resource.request_method_enforce}" "phase:5,t:none,nolog,pass,skipAfter:END_REQUEST_METHOD_PROFILE_CHECK"SecRule REQUEST_METHOD "." "phase:5,t:none,nolog,pass,setvar:resource.request_method_count_%{matched_var}=+1"SecRule RESOURCE:/REQUEST_METHOD_COUNT_/ "@streq %{resource.pattern_threshold}" "chain,phase:5,t:none,nolog,pass"        SecRule MATCHED_VAR_NAME "request_method_count_(.*)$" "capture,setvar:'resource.request_method_enforce=%{resource.request_method_enforce} %{tx.1}',setvar:!resource.request_method_count_%{tx.1}"SecMarker END_REQUEST_METHOD_PROFILE_CHECK

We then have rules that will enforce the expected Request Methods:

## -=[ RE1: Unexpected HTTP Command ]=-## - https://www.owasp.org/index.php/AppSensor_DetectionPoints#RE1:_Unexpected_HTTP_Command#SecRule REQUEST_METHOD "!@within %{resource.request_method_enforce}" "phase:2,t:none,block,msg:'Invalid Request Method for Resource.',logdata:'%{request_method}',setvar:'tx.msg=%{rule.msg}',setvar:tx.anomaly_score=+%{tx.error_anomaly_score},setvar:tx.profiler_score=+%{tx.error_anomaly_score},tag:'POLICY/METHOD_NOT_ALLOWED',tag:'OWASP_AppSensor/RE1',tag:'https://www.owasp.org/index.php/AppSensor_DetectionPoints#RE1:_Unexpected_HTTP_Command'"

Here is how an example alert looks when a GET request is sent to a resource that we have profiled and learned as usually using the POST method:

[Tue Apr 26 13:39:33 2011] [error] [client ::1] ModSecurity: Warning. Match of "within %{resource.request_method_enforce}" against "REQUEST_METHOD" required. [file "/usr/local/apache/conf/crs/base_rules/modsecurity_crs_40_appsensor_detection_point_2.1_request_exception.conf"] [line "44"] [msg "Invalid Request Method for Resource."] [data "GET"] [tag "POLICY/METHOD_NOT_ALLOWED"] [tag "OWASP_AppSensor/RE1"] [tag "https://www.owasp.org/index.php/AppSensor_DetectionPoints#RE1:_Unexpected_HTTP_Command"] [hostname "localhost"] [uri "/cgi-bin/printenv"] [unique_id "TbcDVcCoqAEAALTPDHkAAAAC"]

This level of protection will help to prevent more subtle types of attacks against a resource.