ModSecurity Advanced Topic of the Week: Mitigation of 'Slow Read' Denial of Service Attack
Slow-Read DoS Attack Background
Another tweak in the ongoing "Slow" DoS attacks has emerged this week. The previous Slow DoS attacks focuses on methods of slowing down the rate of request data sent to the target web server. This time, the attack centers on a method of slowing down the rate at which the client (attacker) is able to consume the response data sent back by the web server - hence the name "Slow Read" DoS.
It is rather serendipitous as I actually had the same attack scenario in mind for some time but never got around to develop working exploit code. So a hat tip to Sergey Shekyan as we have seen in the security space that theoretical exploits fall on deaf ears and that you need working proof of concept code to get people to listen.
For those of you familiar with the old LaBrea Tarpit app for slowing down network based worms, this is somewhat of a reverse approach. Instead of the defender (LaBrea) sending back a TCP Window size of 0 to the attacker (worm) which would force the TCP client to wait for a period of time before resubmitting, in this scenario the attacker is the one forcing the web server to wait. After sending in the request, the client responds with TCP window sizes that are much smaller than normal. This forces the web server to queue the response a break it up into smaller chunks that the client will accept.
Slow-Read DoS Attack Example
You can download the slowhttptest tool from the project site and then follow the steps on this page to test out a Slow-Read attack. Here is an example test command that I modified to more closely mimic the LaBrea concept as it uses TCP window sizes of only 1 or 2:
./slowhttptest -c 1000 -X -g -o slow_read_stats -r 200 -w 1 -y 2 -n 5 -z 32 -k 3 -u http://localhost/target_file -p 3
When running this command against an Apache server, the tool shows this data:
$ ./slowhttptest -c 1000 -X -g -o slow_read_stats -r 200 -w 512 -y 1024 -n 5 -z 32 -k 3 -u http://localhost/target_file -p 3 Fri Jan 6 10:07:09 2012:set open files limit to 1010Fri Jan 6 10:07:09 2012:Using:test type: SLOW READnumber of connections: 1000URL: http://localhost/target_fileverb: GETreceive window range: 512 - 1024pipeline factor: 3read rate from receive buffer: 32 bytes / 5 secconnections per seconds: 200probe connection timeout: 3 secondstest duration: 240 secondsFri Jan 6 10:07:09 2012:slow HTTP test status on 0th second:initializing: 0pending: 1connected: 0error: 0closed: 0service available: YESFri Jan 6 10:07:14 2012:slow HTTP test status on 5th second:initializing: 0pending: 541connected: 383error: 0closed: 0service available: NOFri Jan 6 10:07:19 2012:slow HTTP test status on 10th second:initializing: 0pending: 617connected: 383error: 0closed: 0service available: NO
You can see that after a 5 seconds, the server is no longer able to service new requests as all of the existing threads are stuck in a WRITE state.
The following tcpdump data shows where the tool changes the TCP window size to 1:
10:56:06.872895 IP (tos 0x0, ttl 64, id 5652, offset 0, flags [DF], proto TCP (6), length 64, bad cksum 0 (->a081)!) 192.168.1.105.http > 192.168.1.105.57011: Flags [S.], cksum 0x8455 (incorrect -> 0x6be3), seq 1528097122, ack 3885912325, win 65535, options [mss 16344,nop,wscale 1,nop,nop,TS val 317572454 ecr 317572454,sackOK,eol], length 0 0x0000: 4500 0040 1614 4000 4006 0000 c0a8 0169 E..@..@.@......i 0x0010: c0a8 0169 0050 deb3 5b14 e962 e79e 5105 ...i.P..[..b..Q. 0x0020: b012 ffff 8455 0000 0204 3fd8 0103 0301 .....U....?..... 0x0030: 0101 080a 12ed c566 12ed c566 0402 0000 .......f...f....10:56:06.872902 IP (tos 0x0, ttl 64, id 32564, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->376d)!) 192.168.1.105.57010 > 192.168.1.105.http: Flags [.], cksum 0x8449 (incorrect -> 0x7a07), ack 1, win 65328, options [nop,nop,TS val 317572454 ecr 317572454], length 0 0x0000: 4500 0034 7f34 4000 4006 0000 c0a8 0169 E..4.4@.@......i 0x0010: c0a8 0169 deb2 0050 6a91 b7b5 5fc3 67ad ...i...Pj..._.g. 0x0020: 8010 ff30 8449 0000 0101 080a 12ed c566 ...0.I.........f 0x0030: 12ed c566 ...f10:56:06.872906 IP (tos 0x0, ttl 64, id 1603, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->b05e)!) 192.168.1.105.57011 > 192.168.1.105.http: Flags [.], cksum 0x8449 (incorrect -> 0xe6a1), ack 1, win 65328, options [nop,nop,TS val 317572454 ecr 317572454], length 0 0x0000: 4500 0034 0643 4000 4006 0000 c0a8 0169 E..4.C@.@......i 0x0010: c0a8 0169 deb3 0050 e79e 5105 5b14 e963 ...i...P..Q.[..c 0x0020: 8010 ff30 8449 0000 0101 080a 12ed c566 ...0.I.........f 0x0030: 12ed c566 ...f10:56:06.877862 IP (tos 0x0, ttl 64, id 42608, offset 0, flags [DF], proto TCP (6), length 64, bad cksum 0 (->1025)!) 192.168.1.105.57012 > 192.168.1.105.http: Flags [S], cksum 0x8455 (incorrect -> 0x4db9), seq 1017984763, win 1, options [mss 16344,nop,wscale 0,nop,nop,TS val 317572454 ecr 0,sackOK,eol], length 0 0x0000: 4500 0040 a670 4000 4006 0000 c0a8 0169 E..@.p@.@......i 0x0010: c0a8 0169 deb4 0050 3cad 36fb 0000 0000 ...i...P<.6..... 0x0020: b002 0001 8455 0000 0204 3fd8 0103 0300 .....U....?..... 0x0030: 0101 080a 12ed c566 0000 0000 0402 0000 .......f........
For further visual evidence, here is a screen shot of the Server-Status page right when the attack is stopped:
Notice that all of the threads are in a "W" Sending Reply state.
Mitigating Slow-Read DoS Attacks with ModSecurity
ModSecurity v2.6 introduced a new directive called SecWriteStateLimit, which will place a limit on the concurrent number of threads (per IP address) in a SERVER_BUSY_WRITE state. This was originally created to help mitigate the Slow Request Body DoS attacks as Apache moves these threads into this state when it is reading request body payloads. Well, as it turns out, when Apache is sending response body data back to a client, it too moves the threads into the SERVER_BUSY_WRITE state. This means that we can use the ModSecurity SecWriteStateLimit to set an upper threshold of concurrent threads (per IP address) in this state. If this threshold is met, then new threads over this limit will be terminated. The end result is that an attacker will not be able to tie up all available threads and other clients will be able to access the web server.
For testing purposes, the following ModSecurity directive was added to the configuration:
SecWriteStateLimit 100
The same slowhttptest test was then run and the Apache Server-Status page looked like this during the attack:
Notice that there is now a limit of ~100 concurrent connections (per IP address) in the W state and that the other threads are open and able to process new requests.
During the attack, the Apache error_log showed these error message from ModSecurity's SecLimitWriteState directive:
[Fri Jan 06 10:39:38 2012] [warn] ModSecurity: Access denied with code 400. Too many threads [101] of 100 allowed in WRITE state from 192.168.1.105 - Possible DoS Consumption Attack [Rejected][Fri Jan 06 10:39:38 2012] [warn] ModSecurity: Access denied with code 400. Too many threads [101] of 100 allowed in WRITE state from 192.168.1.105 - Possible DoS Consumption Attack [Rejected][Fri Jan 06 10:39:38 2012] [warn] ModSecurity: Access denied with code 400. Too many threads [101] of 100 allowed in WRITE state from 192.168.1.105 - Possible DoS Consumption Attack [Rejected][Fri Jan 06 10:39:38 2012] [warn] ModSecurity: Access denied with code 400. Too many threads [101] of 100 allowed in WRITE state from 192.168.1.105 - Possible DoS Consumption Attack [Rejected][Fri Jan 06 10:39:38 2012] [warn] ModSecurity: Access denied with code 400. Too many threads [101] of 100 allowed in WRITE state from 192.168.1.105 - Possible DoS Consumption Attack [Rejected][Fri Jan 06 10:39:38 2012] [warn] ModSecurity: Access denied with code 400. Too many threads [101] of 100 allowed in WRITE state from 192.168.1.105 - Possible DoS Consumption Attack [Rejected][Fri Jan 06 10:39:38 2012] [warn] ModSecurity: Access denied with code 400. Too many threads [101] of 100 allowed in WRITE state from 192.168.1.105 - Possible DoS Consumption Attack [Rejected][Fri Jan 06 10:39:38 2012] [warn] ModSecurity: Access denied with code 400. Too many threads [101] of 100 allowed in WRITE state from 192.168.1.105 - Possible DoS Consumption Attack [Rejected][Fri Jan 06 10:39:38 2012] [warn] ModSecurity: Access denied with code 400. Too many threads [101] of 100 allowed in WRITE state from 192.168.1.105 - Possible DoS Consumption Attack [Rejected]