Creating the ModSecurity v3 IDS connector as part of Google Summer of Code

A note from the Trustwave Spiderlabs ModSecurity team:

The following blog was written at the culmination of the Google Summer Of Code (GSOC) program by Akhil Koul. The ModSecurity team mentored Akhil to help enhance the open source ModSecurity project that Trustwave also supports. This project, including the additions Akhil added, is a community project and we always welcome community members to help in whatever way they can. We'd like to thank Akhil for his hard work and dedication and wish him continued success.

From trying my hands at Google Summer of Code (GSoC) for the first time to getting selected (yay :D) and later spending 3+ hours daily trying to figure out how to complete the milestones set, this summer has been a very fruitful experience for me. Now as the final evaluation period of GSoC is just a few days away, I am excited to share my experiences working on the ModSecurity v3 IDS Connector.

Initial days:

Being an Information Security enthusiast, I was always looking for an opportunity to contribute to both the Information Security and the Open Source worlds. GSoC gave me an excellent platform to get started with these goals. While going through the list of accepted organizations, I came across ModSecurity and started going through the organization's project ideas. The project titled ModSecurity IDS Connector caught my eye and after long discussions with the mentors over the IRC channel, I decided I wanted to work on this project.

Motivation behind the project:

I got interested in ModSecurity when I found out about their need for completing their v3 (aka libmodsecurity) implementation. The fact that ModSecurity was being used by millions of users around the world was a critical motivator for me to keep learning and fuelled my desire to make a positive impact on the user experience of so many people. The main objective of the project was to build a libmodsecurity connector for Snort which would enable Snort to send HTTP data to libmodsecurity for processing, after extracting the data from an HTTP buffer.

Getting Started:

Starting out, I wanted to get the feel of ModSecurity and its working before diving into the coding part. I started by installing and compiling ModSecurity and all its dependencies and submitting the compilation recipe for Debian based systems. The example connector gave me a taste of the various libmodsecurity functions and their working. The documentation was also pretty exhaustive and covered essential aspects of the connector creation aspect. Subsequently, I shifted my focus to understanding Snort and its working. To my surprise, a newer version of Snort (Snort++) was under active development. It took me quite a while to understand Snort and it's inner-working due to its large and ever-changing code base. After spending around a month reading and testing, I felt that I was ready for the coding part.

Project Description and Approach:

An IDS/IPS, like Snort, works at layer 3/4 of OSI Model. An IDS might be very good at examining things like TCP/IP packet info, and doing some DPI but often high-level protocols like HTTP are tricky for an IDS to examine because packets can be obfuscated or out of order. On the other hand, a WAF works at layer 7 of OSI model. They are designed specifically to look at layer 7 HTTP traffic and don't care about the underlying transport mechanism. The purpose of this project was to enable an IDS like Snort to utilise the HTTP processing capabilities of ModSecurity by extracting and sending HTTP data from Snort to ModSecurity. The libmodsecurity connector would then handle collecting relevant data from Snort and sending that to the ModSecurity core for processing. Check out this post for a better understanding of differences between an IDS/IPS and a WAF.

Project Specifics and Execution:

Snort++ is organized into a collection of built-in and plugin modules. There are several plugin types:

• Codec - to decode and encode packets
• Inspector - allow Snort functionality to be extended by dropping modular plugins
• Logger - for handling events

Fig 1: Snort3X Packet Processing

One important feature of Snort++ which proved useful in this project was the publish-subscribe patterns. By using inspection events, it is possible for an inspector to supply data that other inspectors can process. The normalized data is not actually published. Instead, access to the data is published, and that means that subscribers can access the raw or normalized version(s) as needed.

A basic example of this is provided by the extra data_log plugin. Being a passive inspector, it does nothing until it receives the data it subscribed for. By adding the following to your snort.lua configuration, you will get a simple URI logger:

data_log = { key = 'http_raw_uri' }

This feature allowed me to extract TCP and URI data from Snort. The http_inspect pre-processor publishes an HTTP URI to which a data_log plugin subscribes. The data_log plugin then writes the extracted data along with TCP data (srcIP, srcPort, desIP, desPort) to a data.log file on the system.

A simple example of data written in the data.log file using the sample http.cap present in the project repository is shown below:

1466442542, 10.0.2.15, 1033, 72.21.215.82, 80, http_raw_uri, /zcheck.html
1466442542, 10.0.2.15, 1035, 72.21.215.82, 80, http_raw_uri, /zcheck.html
1466442542, 10.0.2.15, 1036, 50.17.253.86, 80, http_raw_uri, /as/udu
1466442542, 10.0.2.15, 1038, 54.231.2.68, 80, http_raw_uri, /zero.html

Once all the data was extracted and written into the data.log file, I began working on the connector part. By reading data from the data.log file, I was able to extract the relevant fields to be sent to the libmodsecurity functions. The libmodsecurity functions then automatically took care of the processing part by calling the ModSecurity core and passing the extracted data as arguments.

Problems faced:

I faced several problems throughout the project. They are listed as follows:

1. Snort version confusion:
   Initially I spent a lot of time on Snort v2 as it the stable release version currently is use. However, on the advice of Snort developers, I shifted to Snort++ (v3) which has improved features and is much more flexible. This took up a lot of my time as shifting from one code base to other was time-consuming. Snort++ had a humongous and ever-changing code base which made it difficult to get a grasp on its workings.

2. All HTTP data not published:
   As of now, the http_inspect inspector only publishes the HTTP URI which can be subscribed to by the data_log plugin. Other HTTP data like Http version, method, and request and response body data is currently not supported.

3. New http_inspect inspector in development:
   The http_inspect inspector currently used will be depreciated in the near future and will be replaced by a newer version. However, I am unsure of the timeframe for that work to be completed.

Future Scope:

The future scope of the project includes:

1. Extracting the remaining HTTP fields from Snort and sending them to libmodsecurity for processing
2. Replacing the old http_inspect plugin with the new one.
3. Bug fixes and added functionality (maybe trying to call libmodsecurity functions from the inspector itself)

Links:

• List of commits
• GSoC Project
• About Me

Acknowledgments:

I would like to thank Chaim Sanders and Robert Paprocki for guiding me throughout the project and for being wonderful mentors.

To know more about the ModSecurity project, take a look at the official GitHub page.

In case of any queries and feature requests, please don't hesitate in dropping a mail to: Akhil Koul[akhil.koul8@gmail.com].