Web Application Defender's Cookbook: CCDC Blue Team Cheatsheet
Trustwave is a corporate sponsor of the National Collegiate Cyber Defense Competition (CCDC) where the SpiderLabs team members actively participate on the Red Teams and simulate attackers. While we have been highly active in the "attack" portion of CCDC over the years, we haven't done much to help the competitors themselves (the Blue Team). The purpose of this blog post is to help the CCDC defenders with the challenge of fending off the web attackers during the competition!
I recently authored a book from Wiley entitled " Web Application Defender's Cookbook: Battling Hackers and Protecting Users " where I outline various "recipes" to help web defenders protect their web applications. These are essentially the same steps that CCDC competitors must follow in order to protect their web applications from attack. The main difference in context is that CCDC competitors typically have a very condensed time window to quickly lock down their environments before the Red Teams are unleashed. It is for this reason that I will distill down the essential tasks that CCDC competitors should do to get the most value for protecting their websites. Think of this as the "cheatsheet" or CliffNotes version of my book :)
Preparing the Battlespace - Application Fortification
In CCDC events, competitors inherit networks with server and applications that they must defend from attacks. They must quickly get a handle on their assets and implement tools, scripts, etc... to help them defend.
Install ModSecurity
ModSecurity is a cross-platform (Apache, IIS and Nginx), open source web application firewall (WAF) module. It is a Trustwave SpiderLabs project and is the most widely deployed WAF in existence. ModSecurity can be manually installed from source code by downloading the archive from our site however for CCDC you will most likely not want to fool around with configuration/compilation time. Fortunately, ModSecurity is already installed in most OS repos. This means that you should be able to easily install it by using your package managers with commands like:
- Ubuntu
$ sudo apt-get install libapache-mod-security
- CentOS
$ yum install mod_security
Once installed, you then need to setup some basic configurations.
Recommended Base Configuration
ModSecurity comes with a recommended base config file that sets reasonable settings for items such as whether you want to allow block or not, where to send log files, etc...
Set Blocking Mode
The default setting is to run in "DetectionOnly" mode which means ModSecurity will generate events but it won't take any disruptive actions. While this is reasonable for a normal production environment, I would recommend that CCDC competitor change this to On to allow blocking. Do keep in mind, however, that when the scoring engine visits your site you do not want any false positive blocks or you will lose points!
Enable HTTP Audit Logging
When dealing with web applications, one of the first issue that you must deal with is that of Visibility. Let's be honest - default web server logging sucks. You have limited visibility into HTTP(S) transactional data which makes it very hard to identify attacks, vulnerabilities and also leaked or stolen data exiting the application. Proper web application instrumentation is paramount. ModSecurity has an Audit Engine (SecAuditEngine directive) that allows the user to configure how much data they want to log. For CCDC competitors, I would absolutely recommend setting this to On. This will create a transactional log file for every single request/response with all inbound and outbound data. This is invaluable when doing quick IR reviews to confirm attack vectors, success and data leakages.
Install the OWASP ModSecurity Core Rule Set (CRS)
The OWASP ModSecurity CRS is a project that Trustwave SpiderLabs Research leads. It's purpose is to provide essential, drop-in protections for issues such as SQL Injection, Cross-site Scripting and data leakage prevention. It uses a generic, negative security model approach to identify malicious activity. It is by no means perfect, but does significantly raise the bar needed for attackers to evade the filters. It is during these times that CCDC competitors should be reviewing alerts and audit log data to verify attack vectors and if there are any vulnerabilities that need to be addressed. The CRS archive can be downloaded from the SpiderLabs GitHub Repo. It is recommended that you review the following blog post to decide whether you want to run the CRS in traditional vs. anomaly scoring mode.
Hint - you may want to activate some of the optional rules that help to prevent sensitive data leakage such as Credit Card numbers :)
Dynamic Virtual Patching
If time permits, another quick method of securing web applications is to do Virtual Patching. In this pre-deployment scenario, CCDC competitors are doing Proactive Virtual Patching by identifying vulnerabilities themselves by running a dynamic vulnerability scanner against the site and then taking the XML report data and auto-converting this intelligence into custom virtual patches for ModSecurity. The Red Team is most certainly going to be using similar tools against your sites. You should too! Read the following two blog posts:
- Automated Virtual Patching with Arachni Scanner
- Automated Virtual Patching with OWASP Zed Attack Proxy
Setting HoneyTraps
Again, if time permits, you can also configure ModSecurity to set out HoneyTraps for the Red Teams. These are valuable as they can quickly identify malicious users. Reference blog posts:
Asymmetric Warfare
As Theodin proclaimed in the Two Towers: And so it begins... Once the CCDC competition begins, and the enormous Orc Army (Red Team) starts charging the walls, you will need to keep a keen eye on your alert logs.
Monitoring Logs/Alerts
You can do this by manually reviewing the Apache error_log file. Two methods of log review you should consider during competition -
- Review repeated alerts for specific resources. This usually indicates Red Teams have narrowed in their focus on a resource and are attempting filter evasions.
- Any Outbound error/leakage alerts. This indicates that an inbound request was not block (perhaps an evasion) and the application is complaining. This is a time bomb waiting to go off. You should react quickly to take action against the attacker and/or create a virtual patch.
Alert Anlaysis
As an example of the types of items CCDC competitors should keep an eye out for are any events of severity ALERT (1) or EMERGENCY (0) as these incidate application errors or data leakages. Here is an example alert:
[Fri Feb 17 16:26:37 2012] [error] [client 192.168.1.103] ModSecurity: Warning. Operator GE matched 1 at TX. [file "/usr/ local/apache/conf/crs/base_rules/modsecurity_crs_60_correlation. conf"] [line "29"] [id "981202"] [msg "Correlated Attack Attempt Identified: (Total Score: 21, SQLi=5, XSS=) Inbound Attack (981242-Detects classic SQL injection probings 1/2 Inbound Anomaly Score: 13) + Outbound Application Error (ASP/JSP source code leakage - Outbound Anomaly Score: 8)"] [severity "ALERT"] [hostname "192.168.1.103"] [uri "/zapwave/active/inject/inject- sql-form-basic.jsp"] [unique_id "Tz7GDcCoAWcAAOsFI@cAAAAA"]
This was generated when an attacker was sending SQL Injection attacks to a Java app server and it generated this example stack dump page:
If you see these types of alerts, you must take immediate action to secure that resource.
Verifying Data Leakages
Here is another example of conducting event analysis using the ModSecurity audit log file. Let's say you identified the following entries in your Apache access_log file:
109.70.36.102 - - [15/Feb/2012:09:08:16 -0500] "POST /wordpress//xmlrpc.php HTTP/1.1" 500 163 "-" "Wordpress Hash Grabber v2.0libwww-perl/6.02" 109.70.36.102 - - [15/Feb/2012:09:08:17 -0500] "POST /wordpress//xmlrpc.php HTTP/1.1" 200 613 "-" "Wordpress Hash Grabber v2.0libwww-perl/6.02"
Remember how I said that default web server log files suck? Yeah, here is why. You can see that the User-Agent data is suspicious as it indicates a malicious program for extracting password hash data, however the rest of the information is incomplete. First of all, the HTTP Request Method is POST which means that there was a request body that was passing parameters to the application. This data, unfortunately, is not logged. The other item of concern is that HTTP Response Status Codes. In the first request the result was a 500 Error while in the second one it was a 200 OK. Is this good or bad? Did the application error in the first request leak information? You can not answer this question as the default logging does not capture the outbound HTTP Responses. This is where the ModSecurity audit log file will be like gold to CCDC competitors! Here is an example audit log file entry for the second request above:
--26b60826-A-- [15/Feb/2012:09:08:17 --0500] Tzu8UcCoqAEAAR4rI1cAAAAA 109.70.36.102 58538 192.168.1.111 80 --357b3215-B-- POST /wordpress//xmlrpc.php HTTP/1.1 TE: deflate,gzip;q=0.3 Connection: TE, close Host: localhost User-Agent: Wordpress Hash Grabber v2.0libwww-perl/6.02 Content-Length: 738--357b3215-C-- <?xml version="1.0"?><methodCall><methodName>mt.setPostCategories </methodName> <params> <param><value><string>3 union all select user_pass from wp_users where id=3</string></value></param> </param> </param> <member><param><value><string>admin</string></value> <param><value><string>admin</string></value><param><value> <array> <name>categoryId</name><data><value> <struct> <value><string>1</string></value> </member> <name>categoryName</name> </string></value> </member> </name> <value><boolean>0</boolean></value> </member> </struct></value> </data></array></value> </param> </params> </methodCall>--357b3215-F-- HTTP/1.1 200 OK X-Powered-By: PHP/5.3.2-1ubuntu4.5 Content-Length: 649 Vary: Accept-Encoding Content-Type: text/xml Connection: close--357b3215-E-- <div id='error'><p class='wpdberror'><strong>WordPress database error :</strong> [You have an error in your SQL syntax; check the manualthat corresponds to your MySQL server version for the right syntax to use near 'union all select user_pass from wp_users whereid=3' at line 3]<br /> <code>DELETE FROM wp_post2cat WHERE category_id = 349508cb0ff9325066aa6c490c33d98bAND post_id = 3 union all select user_ pass from wp_users where id=3</code></p> </div><?xml version="1.0"?><methodResponse> <params><param> <value><boolean>1</boolean> </value></param> </params></methodResponse>--357b3215-H-- Apache-Handler: proxy-server Stopwatch: 1329314896780667 97446 (- - -) Stopwatch2: 1329314896780667 97446; combined=278, p1=9, p2=229, p3=10, p4=11, p5=18, sr=0, sw=1, l=0, gc=0 Response-Body-Transformed: Dechunked Producer: ModSecurity for Apache/2.7.0-dev1 (http://www. modsecurity.org/). Server: Apache/2.2.17 (Unix) mod_ssl/2.2.12 OpenSSL/0.9.8r DAV/2--357b3215-Z--
The two most important data elements are highlighted -
- Under Section C (Request Body) - you can see the full XML payload, including where the attack tool used SQL Injection to dump the user_pass hash data from the wp_users table.
- Under Section E (Response Body) - you can see the full HTML text sent to the attacker. Unfortunately, the highlighted section shows that the SQL Injection attack was successful at extracting the password hash for the user with ID 3. At this point, you can bet that the CCDC Red Team will start password cracking tasks to try and identify this user's password. It is for this reason that you should change this users password immediately and then proceed to the next section to create a virtual patch to plug this SQL Injection vuln on the WordPress xmlrpc.php page.
Response Actions
Reactive Virtual Patching
As opposed to the Proactive Virtual Patching mentioned previously, this scenario is Reactive and is based upon identifying vulnerabilities as part of live attack attempts. While you are conducting log/alert analysis, you will most likely identify a hot-spot that the attackers are focusing on. In this scenario, you should supplement the generic, blacklist protections provided by the OWASP ModSecurity CRS by create a positive security model ruleset for the targeted resource. Reference the OWASP Virtual Patching CheatSheet section on creating positive security model patches.
Blacklisting Source IP Addresses
ModSecurity has the capability to temporarily blacklist IP addresses. Do use this with care however to ensure that you do not block network segments or the scoring engine clients.
Slowing Down Automated Scanners
ModSecurity can also slow down responses by using the pause action. This can be useful to help slow down scanners and to give CCDC defenders more time to respond while the attacker is still in the enumeration phase.
Submitting Incident Reports
The data provided by ModSecurity's alerts and full audit logs will be critical for submitting your incident reports during the competition. This information will allow you to efficiently outline which systems/resources were attacked and what the outcome was.
Conclusion
If you are a CCDC competitor, I hope you found this blog post useful. The tactics outline here, if properly implemented, will greatly increase your chances of successfully defending your systems during the competition. We would also love to hear any feedback on your experiences using ModSecurity during the competitions. Thanks and Good Luck!