(Updated) ModSecurity Advanced Topic of the Week: Mitigating Slow HTTP DoS Attacks

Update - the latest version of the ModSecurity 2.6 has a new directive called SecWriteStateLimit that helps to defend against Slow POST attacks.

With the recent OWASP AppSec DC presentation on Slow HTTP POST DoS attacks, the issue of web server platform DoS concerns have reached a new high. Notice that I said, web server platform and not web application code. The attack scenario raised by slow HTTP POST attack is related to web server software (Apache, IIS, SunONE, etc...) and can not be directly mitigated by the application code. In the blog post, we will highlight the two main varieties of slow HTTP attacks - slow request headers and slow request bodies. We will then provide some new mitigation options for the Apache web server platform with ModSecurity.

Network DoS vs. Layer-7 DoS

Whereas network level DoS attacks aim to flood your pipe with lower-level OSI traffic (SYN packets, etc...), web application layer DoS attacks can often be achieved with much less traffic. The point here is that the amount of traffic which can often cause an HTTP DoS condition is often much less than what a network level device would identify as anomalous and therefore would not report on it as they would with traditional network level botnet DDoS attacks.

Layer-7 Request Delay Attacks

Ivan Ristic brought up the concept of connection consumption attacks in his 2005 book "Apache Security":

5.4.3. Programming Model Attacks

The brute-force attacks we have discussed are easy to perform but may require a lot of bandwidth, and they are easy to spot. With some programming skills, the attack can be improved to leave no trace in the logs and to require little bandwidth.

The trick is to open a connection to the server but not send a single byte. Opening the connection and waiting requires almost no resources by the attacker, but it permanently ties up one Apache process to wait patiently for a request. Apache will wait until the timeout expires, and then close the connection. As of Apache 1.3.31, request-line timeouts are logged to the access log (with status code 408). Request line timeout messages appear in the error log with the level info. Apache 2 does not log such messages to the error log, but efforts are underway to add the same functionality as is present in the 1.x branch.

Opening just one connection will not disrupt anything, but opening hundreds of connections at the same time will make all available Apache processes busy. When the maximal number of processes is reached, Apache will log the event into the error log ("server reached MaxClients setting, consider raising the MaxClients setting") and start holding new connections in a queue. This type of attack is similar to the SYN flood network attack we discussed earlier. If we continue to open new connections at a high rate, legitimate requests will hardly be served.

If we start opening our connections at an even higher rate, the waiting queue itself will become full (up to 511 connections are queued by default; another value can be configured using the ListenBackLog directive) and will result in new connections being rejected.

Defending against this type of attack is difficult. The only solution is to monitor server performance closely (in real-time) and deny access from the attacker's IP address when attacked.

The issue at hand with these attacks is that the client(s) are opening connections with the web server and sending request data very slowly. For those of you familiar with the old LaBrea Tarpit app for slowing down network based worms, this is somewhat of a reverse approach. Instead of the defender (LaBrea) sending back a TCP Window size of 0 to the attacker (worm) which would force the TCP client to wait for a period of time before resubmitting, in this scenario the attacker is the one forcing the web server to wait. If a web client opens a connection and doesn't send any data to the web server then the web server will default to waiting for the connection's Timeout value to be reached. Wanna guess how long that time interval is in Apache by default? 300 seconds (5 minutes). This means that if a client can simply open a connection and not send anything, that Apache child process thread will sit idle, waiting for data, for 5 minutes. Ouch... So the next logical question to ask from the attacker's perspective is - What is the upper limit on the number of concurrent connections for Apache? This depends on your configs but the main ServerLimit directive has a hard coded value of 20000 (most sites run much less). This limit makes it very feasible for a much smaller number of DDoS clients to take down a site vs. the extremely large number required for network-based pipe flooding.

There are two types of attack to cover when a malicious client never sends a complete Request as specified by the HTTP RFC:

Request       = Request-Line                      ; Section 5.1
                        *(( general-header        ; Section 4.5
                         | request-header         ; Section 5.3
                         | entity-header ) CRLF)  ; Section 7.1
CRLF
                        [ message-body ]          ; Section 4.3

Notice that the end of the Request is marked by the server when there is a blank line (CRLF). What happens if a client doesn't send these final CRLFs?

Layer-7 Request Delay Attack 1: Slow Headers (A.K.A: Slowloris Attack)

Rsnake wrote the Slowloris tool to show what happens when a client does not send a complete set of Request headers. If you look at the Slowloris script code, you can see that it will send an HTTP request similar to the following:

GET / HTTP/1.1 CRLF
Host: www.example.com CRLF
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.503l3; .NET CLR 3.  0.4506.2152; .NET CLR 3.5.30729; MSOffice 12) CRLF
Content-Length: 42 CRLF

The request is missing the final CRLF that tells the destination web server that the request has completed so the web server dutifully waits for more request data until it reaches it's Timeout setting. Now Slowloris can keep it in a perpetual waiting mode by sending new requests just before the web server's timeout setting is reached.

Mitigating Slow Request Header Attacks with ModSecurity - SecReadStateLimit

Unfortunately for ModSecurity, it was not able to identify or mitigate a slowloris-type of attack due to the fact that its first Apache hook was in the POST-READ-REQUEST phase.

The Slowloris-type requests never complete and thus don't move into the ModSecurity phase:1 processing phase. When the connection reaches the Timeout threshold, Apache will issue a 400 Bad Request status code and log a message to the error_log file. So how could ModSecurity help?

In ModSecurity v2.5.13, we have introduced a new defensive mitigation capability that aims to help prevent the effects of a Slowloris-type of attack. The new directive is called SecReadStateLimit and its purpose is to hook into the Connection-level Filter of Apache and restrict the number of Apache threads that are in a SERVER_BUSY_STATE for each IP address. The rationale for this approach comes from the fact that, with HTTP/1.1 pipelining capability, there really is no legitimate scenario where a single client will be initiating that many simultaneous connections.

If the user specifies the new directive with a positive integer (such as SecReadStateLimit 5) then ModSecurity will issue a 400 Bad Request status code and terminate all connections in increments equal to your threshold. For instance, if you had 5 set as the limit and the attacker had 50 open connections, then you would receive 10 different error messages in the Apache error_log file such as this:

[Mon Nov 22 17:44:46 2010] [warn] ModSecurity: Access denied with code 400. 
Too many connections [6] of 5 allowed in READ state from 211.144.112.20 - 
Possible DoS Consumption Attack [Rejected]
[Mon Nov 22 17:44:47 2010] [warn] ModSecurity: Access denied with code 400. 
Too many connections [6] of 5 allowed in READ state from 211.144.112.20 - 
Possible DoS Consumption Attack [Rejected]
[Mon Nov 22 17:44:47 2010] [warn] ModSecurity: Access denied with code 400. 
Too many connections [6] of 5 allowed in READ state from 211.144.112.20 - 
Possible DoS Consumption Attack [Rejected]
[Mon Nov 22 17:44:48 2010] [warn] ModSecurity: Access denied with code 400. 
Too many connections [6] of 5 allowed in READ state from 211.144.112.20 - 
Possible DoS Consumption Attack [Rejected]

This new capability will allow Apache to flush out these slow/idle connections more quickly which will allow legitimate clients to have access to the web site.

Layer-7 Request Delay Attack 2: Slow Request Bodies (A.K.A: r-u-dead-yet/RUDY Attack)

The other type of slow HTTP attack that was covered in the OWASP AppSec DC presentation by Wong Onn Chee and Tom Brennan (@brennantom) is when a client completes the request headers phase however it sends the request body (post payload) very slowly (e.g. - 1 byte/110sec). When you consider that, by default, Apache will accept a request body of up to 2GB in size, you can can see how effective this attack can be.

Since this attack scenario concept was released, two different tools have been released to test the issue: OWASP HTTP POST Tool and r-u-dead-yet (RUDY).

OWASP HTTP POST Tool - developed by Tom Brennan. Here is a screenshot of the GUI:

This is a great tool to use for testing these types of attacks as you can test both the slow headers and slow request body.

r-u-dead-yet/RUDY - developed by Raviv Raz. If you plan to run the tool, I suggest you update the rudeadyet.conf file, which means that you need to already know of a valid POST form destination. I suggest this since the interactive mode, which will spider a target form seems to have some parsing issues. Here is an example of running RUDY in interactive mode:

$ ./r-u-dead-yet.py http://localhost/upload.html
Found 1 forms to submit. Please select number of form to use:

1 ) http://localhost/upload.html/cgi-bin/fup.cgi

> 1

Found 1 parameters to attack. Please select number of parameter to use:

1 ) note

> 1

Number of connections to spawn: (default=50)

> 

[!] Attacking: http://localhost/upload.html/cgi-bin/fup.cgi
[!] With parameter: note 

Notice that the automatically parsed URL is not correct. The correct URL is - http://localhost/cgi-bin/fup.cgi.

Mitigating Slow Request Body Attacks with ModSecurity - SecWriteStateLimit

In ModSecurity v2.6.0, we also introduced the SecWriteStateLimit directive to help mitigate the request body delay attacks. This directive works in the same manner as SecReadStateLimit, except that it is monitoring threads that are in the SERVER_BUSY_WRITE state.

Mitigating Slow Request Attacks with Mod_Reqtimeout and ModSecurity

Caution: there are several reports that mod_reqtimeout does not issue a 408 status code under various conditions. This means that this implementation described below may not work reliably.

While the new ModSecurity SecReadStateLimit and SecWriteStateLimit directives do help to mitigate these request delaying DoS attacks, they do not utilize any type of request timer checking. In order to identify and respond to this type of slow request body attack, we need to use features of both Apache and ModSecurity. Firstly, newer versions of Apache (2.2 branch) have an experimental module called mod_reqtimeout.c and which introduces a new directive called RequestReadTimeout. This new directive allows you to specify different thresholds for receiving request data. For the purposes of identifying RUDY types of attacks, we can add the following directive to our Apache configs:

RequestReadTimeout header=30, body=30

This places a threshold of 30 seconds to completely receive the request body data. If the data is not received by that time, Apache will issue a 408 Request-Timeout status code. With this directive in place, we then add in some new ModSecurity rules that do the following:

• Identify when Apache triggers the 408 status codes
• Track how many times this happens and keep the data in IP-based persistent storage so we can correlate across requests
• If this event happens more than 5 times in 60 seconds, subsequent requests for that IP address will be dropped by ModSecurity for a period of 5 minutes.

Here are the example ModSecurity rules:

SecRule RESPONSE_STATUS "@streq 408" "phase:5,t:none,nolog,pass, \
setvar:ip.slow_dos_counter=+1,expirevar:ip.slow_dos_counter=60"
SecRule IP:SLOW_DOS_COUNTER "@gt 5" "phase:1,t:none,log,drop, \
msg:'Client Connection Dropped due to high # of slow DoS alerts'"

With these new configurations in place, when a tool such as RUDY is run against the site, you will receive messages similar to the following in the Apache error_log file:

[Tue Nov 23 11:52:27 2010] [error] [client 127.0.0.1] ModSecurity: Access denied with connection close (phase 1). 
Operator GT matched 5 at IP:slow_dos_counter. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_15_customrules.conf"] [line "6"] 
[msg "Client Connection Dropped due to high # of slow DoS alerts"] [hostname "localhost"] [uri "/upload.html/cgi-bin/fup.cgi"] [unique_id "TOvxS8CoAWUAAAOcG9oAAAAQ"] 

As an added bonus, once ModSecurity starts initiating its "drop" action on new connections from the attacker's IP address, this will cause RUDY to abort with similar messages:

Process Client-49:Traceback (most recent call last):

File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/process.py", line 231, in _bootstrap self.run()

File "./r-u-dead-yet.py", line 170, in run c.send("A")

File "./r-u-dead-yet.py", line 156, in send self.s.send(data)error: [Errno 32] Broken pipe

These new ModSecurity rules will be available soon within the OWASP ModSecurity Core Rule Set (CRS) Project.