ModSecurity version 3: Fuzzing as part of the QA
The stability of any given project is often tracked by its maturity, which is generally measured by how old the code is. Even though this may be true a lot of the time, here at Trustwave SpiderLabs we wanted to test and assure ModSecurity v3's stability, despite its youthfulness.
We aim to achieve this level of stability by testing the code extensively: Regression tests, Unit tests, and Fuzzer test. In this case, we are using Fuzzer as an instrument to understand how resilient ModSecurity v3 is while receiving a broad variety of inputs.
We have decided to individually test each operator and transformation. In fact, we wanted to not only create a simple test, but a solid testing foundation to be further extended by the community.
In addition to the operators and transformation tests, we are developing a third category of tests focusing on the rules parser, which we will cover in a future blog post. Before jumping in to the explanation of the tests themselves, let's first discuss the American Fuzzy Lop (AFL).
The American Fuzzy Lop
One of the most popular implementations of Fuzzer, is the American Fuzzy Lop or AFL. AFL uses genetic algorithms to increase the code coverage. It maintains a very good reputation and is noted as helping the detection of bugs in significant open source software, such as OpenSSL, Firefox, BIND, and Qt. The full list is available here: AFL's bug-o-rama trophy case.
As a first step to test ModSecurity with AFL you will need to compile the project using clang. Depending on the target software this may be an issue, due to the fact that clang is sometimes pickier than the popular option, gcc. You will also need to add the option --enable-afl-fuzz during the configuration portion of compilation. The Image 1. Illustrates the configuration summary while the option --enable-afl-fuzz was supplied.
Image 1. Summary of ModSecurity version 3 configuration options. (a) Indicates that the afl fuzzer project will be compiled.
After compilation with clang, it is necessary to have the gateway for the AFL. This gateway is a simple implementation where we tell AFL which methods from ModSecurity to call for testing. Within ModSecurity v3 our gateway drives the test towards our operators and transformations and they are available to any user by compiling ModSecurity with a fuzzer implementation.
By having the AFL support integrated on our build system, we hope to have ModSecurity widely tested. We also hope that the ModSecurity community will contribute to improve this fuzzer implementation even more.
The fuzzer implementation of ModSecurity is available under the directory test/ and it is part of the official ModSecurity repository. In order to run the fuzzer, just execute the following from the test/fuzzer directory:
AFL_SKIP_CPUFREQ=1 afl-fuzz -m none -i testcases/ -o syncdir/ -M fuzzer2 -- ./afl_fuzzer
You should see a screen like the one in the Image 2.
Image 2. Screenshot of an afl session.
Now that we had a brief explanation on AFL and how it is integrated with ModSecurity, let's take a look at how the operators and the transformations are being tested. If you want to learn more about AFL please refer their website at: http://lcamtuf.coredump.cx/afl/
Testing the operators
ModSecurity v3 has a pluggable architecture. The level of interdependency from the core to the operators is very minimal. This is ease the implementation of new operators without demanding deep understanding of ModSecurity code. The same philosophy goes for transformations.
Another benefit of this modular architecture is the fact that we can execute an operator without even touching the ModSecurity's core. This is exactly what we want in terms of testing the operators. Looking at an operator structure, like the one illustrated in Image 3, you can see that we have two main variables: the parameter and the input. Those are, of course, related to the way that the rules are written. In Image 4 you can see an example of a rule using the operator @streq.
Image 3. Operator skeleton. (a) Input string coming from the request. (b) m_param is informed while loading the rules.
Image 4. Operator usage in the SecLang. (a) ModSecurity variable: variable to be replaced with the argument test from the request. (b) String to be compared with (a). (c) Lowercase transformation to be applied in (b) before the comparison with (a).
Notice that on Image 4 (a) came from the rules while (b) is something that came from the request (in the format of a ModSecurity variable). In this scenario we basically have two different parameters to fuzz: (a) and (b).
All the ModSecurity operators have the exact same methods to receive parameters. The init function is used to load the parameter while the evaluate function is the one responsible to call the operator and to inform the input.
In Image 5 we have an example of our fuzzer implementation calling the operator @streq. The very same test is performed with all the operators that we currently support.
Image 5. Code snippet from our fuzzer implementation. (a) Operator instance. (b) Operator evaluation method.
Testing the transformations
Similar to the operators, the transformations can also be executed without touching ModSecurity core. The major difference between the transformations and the operators is the fact that transformations only expect a single input.
Image 6 contains the skeleton of a transformation. All the transformations use the same method signature. Currently we are testing all the supported transformations.
Image 6. The skeleton of a transformation. (a) Value to be transformed. (b) Reference to the transaction (context).
What next
As mentioned earlier, a fuzzer for the parser is currently under development. The parser is a little bit more complex as it may demand the input to be somewhat structured. Another important component to be tested is the API itself. Currently we are testing individual parts of ModSecurity: operators and transformations. Having test cases to support ModSecurity entirely will be very welcomed.