ModSecurity Advanced Topic of the Week: Passive Vulnerability Scanning Part 1 - OSVDB Checks

One of the most under-appreciated capabilities of web application firewalls (WAFs) is traffic monitoring and analysis. Due to the fact that WAFs have access to the full inbound request and outbound response payloads, they are able to glean valuable insight into vulnerabilities and configuration issues such as missing HttpOnly or Secure cookie flags, etc... The main benefit to using a WAF to supplement Dynamic Application Security Testing (DAST) is that the identification happens in real-time by monitoring real clients interacting with the web application, hence the name Passive Vulnerability Scanning (PVS).

PVS Concepts

PVS is much different than then standard WAF "Blocking Inbound Attacks" mentality. First of all, you often need to correlate both inbound and outbound data in order to accurately identify an application flaw or vulnerable resource. Second, you don't want to take any disruptive actions against the live transaction. Remember, these are not "active attack" focused rules where a malicious client is attacking your site. The alerts that are being triggered are used to notify the administrator that there are some potential weaknesses in the application that should be addressed, such as upgrading the software. The WAF is being used solely as the identification mechanism in this scenario.

OSVDB Integration

Ok, so a WAF has access to the live transactional stream, but how does it identify known vulnerabilities in public software? There are a number of different approaches and resources that could be used for application defect identification and we will highlight some in future blog posts. For this post, however, we will focus on using data from the Open Source Vulnerability Database (OSVDB) project which describes itself as:

an independent and open source database created by and for the security community. The goal of the project is to provide accurate, detailed, current and unbiased technical information on security vulnerabilities.

If you have an OSVDB account (sign-up is free), you can download the entire OSVDB data to a local file. Once you download the CSV archive and unpack it, you will have a file called vulnerabilities.txt. This file holds all of the vulnerability details for each issue. The format is one entry per line. Here is an example entry for a recent AWBS SQL Injection vulnerability:

1145002,70616,"Advanced Webhost Billing System (AWBS) cart.php oid Parameter SQL Injection","2011-01-22 14:30:25","2011-01-24 22:31:12","1970-01-01 00:00:00","2011-01-16 00:00:00","2011-01-16 00:00:00",
"Advanced Webhost Billing System (AWBS) 2.9.2 cart.php oid Parameter SQL Injection","Advanced Webhost Billing System (AWBS) contains a flaw that may allow an attacker to carry out an SQL injection attack. The issue is due to the 'cart.php' not properly sanitizing user-supplied input to the 'oid' parameter. This may allow an attacker to inject or manipulate SQL queries in the back-end database, allowing for the manipulation or disclosure of arbitrary data.","",
"Currently, there are no known workarounds or upgrades to correct this issue. However, Total Online Solutions, Inc. has reportedly released a patch to address this vulnerability. Contact the vendor for more information.",
"https://[target]/cart?ca=add_other&oid=1'%20AND%20SLEEP(100)='","2011-01-19 00:00:00"

The format of the entry is:

• id: integer
• osvdb_id: integer
• title: string
• disclosure_date: timestamp
• discovery_date: timestamp
• osvdb_create_date: timestamp
• last_modified_date: timestamp
• exploit_publish_date: timestamp
• solution_date: timestamp
• description: text
• solution: text
• t_description: text
• manual_notes: text
• short_description: text

The "manual notes" section of the vuln entry is highlighted as this data provides a proof of concept exploit code that identifies the attack vector location. This is the data that we can use to conduct our PVS analysis ruleset.

Lua API

Now that we have a local vulnerability database, the next question is - how do we use it? ModSecurity allows for the conditional execution of external scripts via the exec action.

Description: Executes an external script/binary supplied as parameter. As of v2.5.0, if the parameter supplied to exec is a Lua script (detected by the .lua extension) the script will be processed internally. This means you will get direct access to the internal request context from the script.

SpiderLabs has created a new experimental rules file in the SVN repository of the OWASP ModSecurity CRS called modsecurity_crs_56_pvs_checks.conf. This file includes some example PVS SecRules that run in phase:5 (logging) and use the exec action to conditionally execute a Lua script:

SecRule &RESOURCE:OSVDB_CHECK "@eq 0" "chain,phase:5,t:none,nolog,pass"
        SecRule RESPONSE_STATUS "@streq 200" "exec:/usr/local/apache/conf/modsec_current/base_rules/osvdb.lua"
SecRule TX:OSVDB_MSG "!^$" "phase:5,t:none,log,pass,msg:'Passive Vulnerabilty Check with OSVDB - %{matched_var}'"

Keep in mind that these rules run after the request/response have already been completed in the post-processing phase. The first SecRule simply checks to see if we have already checked this URI resource against the OSVDB data. If not, then it proceeds to the 2nd SecRule in the chain which checks the returned HTTP Status Code. If the code is 200, then this means that the requested resource does in fact exist locally and therefore we will proceed with the OSVDB lua script execution. Here is the example PoC osvdb.lua script contents:

#!/opt/local/bin/lua

local request_filename = m.getvar("REQUEST_FILENAME", "none")
local args = {};
args = m.getvars("ARGS_NAMES", "none")


function main ()
for line in io.lines("/usr/local/apache/conf/modsec_current/base_rules/vulnerabilities.txt") do
    if line:find(request_filename) then
        if string.find(line, "^%d+\,") then
                for k,v in pairs(args) do
                local arg_name = v["value"] .. "=";
                        if string.find(line, arg_name) then
                        m.setvar("resource.osvdb_check", "1")
                        m.setvar("resource.osvdb_vulnerable", "1")
                        m.setvar("tx.osvdb_msg", line)
                        return(line)
                        end
                end
        end
    end
end
        m.setvar("resource.osvdb_check", "1")
        return nil
end

The script does the following:

1. Identifies the requested filename in the current transaction and saves it in a local variable
2. Extracts out the parameter names and saves it into an array
3. Runs a check to see if the requested filename appears in any OSVDB entries (this would match in the manual notes sections highlighted previously). The bolded line in the lua script shows where you will need to update the path to the local vulnerabilities.txt file on your system.
4. If there is a match, the parameter names are then checked to see if there also a match. This step is to help with accuracy as there are many different files with the same name.
5. If a match is found, then two variables are saved to the Resource persistent collection file. One variable says that we have conducted an OSVDB check so that further requests to this filename will not trigger the Lua script execution. The other variable flags the current resource as being vulnerable. This data is useful later on for correlation when decided to block inbound attacks.
6. The OSVDB message data is then placed into a TX variable for inspection by a following SecRule.
7. If no match is found with the current requested filename/parameter names, then we will want to flag this resource as checked by OSVDB.

Example PVS Process Flow

With these rules in place, if your site were running the AWBS application and a client accessed the "cart" page with a normal request such as:

http://www.example.com/cart?ca=add_other&oid=58

The following PVS alert would be generated:

[Wed Feb 23 08:48:28 2011] [error] [client ::1] ModSecurity: Warning. Match of "rx ^$" against "TX:osvdb_msg" required. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_15_customrules.conf"] [line "3"] 
[msg "Passive Vulnerabilty Check with OSVDB - 1145002,70616,\\"Advanced Webhost Billing System (AWBS) cart.php oid Parameter SQL Injection\\",\\"2011-01-22 14:30:25\\",\\"2011-01-24 22:31:12\\",\\"1970-01-01 00:00:00\\",\\"2011-01-16 00:00:00\\",\\"2011-01-16 00:00:00\\",\\"Advanced Webhost Billing System (AWBS) 2.9.2 cart.php oid Parameter SQL Injection\\",\\
"Advanced Webhost Billing System (AWBS) contains a flaw that may allow an attacker to carry out an SQL injection attack. The issue is due to the 'cart.php' not properly sanitizing user-supplied input to the 'oid' parameter. This may allow an attacker to inject or manipulate SQL queries in the back-end database, allowing for the manipulation or disclosure of arbitrary data.\\",\\"\\",\\"Currently, there are no known workarounds or upgrades to correct this issue. However, Total Online Solutions, Inc. has reported [hostname "localhost"] 
[uri "/cart"] [unique_id "TWUQKsCoAWQAABbfNx0AAAAA"]

After completing this OSVDB check against the resource, the following data is saved in the resource persistent collection:

Re-retrieving collection prior to store: resource
Wrote variable: name "__expire_KEY", value "1298472508".
Wrote variable: name "KEY", value "/cart".
Wrote variable: name "TIMEOUT", value "3600".
Wrote variable: name "__key", value "/cart".
Wrote variable: name "__name", value "resource".
Wrote variable: name "CREATE_TIME", value "1298468906".
Wrote variable: name "UPDATE_COUNTER", value "1".
Wrote variable: name "osvdb_check", value "1".
Wrote variable: name "osvdb_vulnerable", value "1".
Wrote variable: name "LAST_UPDATE_TIME", value "1298468908".
Persisted collection (name "resource", key "/cart").

With the "osvdb_check" variable saved in a per/resource collection, we will not re-run the osvdb.lua script again until the variable expires (you can change this manually with expirevar). This allows for periodic alerting.

Additionally, with the "osvdb_vulnerable" variable set for this resource, we can also now factor this intelligence into the OWASP ModSecurity Core Rule Set (CRS) anomaly scoring system. We have added in the following rules to the v2.1.3 SVN version of the modsecurity_crs_49_inbound_blocking.conf file:

# Alert and Block based on Anomaly Score and OSVDB Check
#
SecRule TX:ANOMALY_SCORE "@gt 0" \
    "chain,phase:2,t:none,deny,log,msg:'Inbound Attack Targeting OSVDB Flagged Resource.',setvar:tx.inbound_tx_msg=%{tx.msg},setvar:tx.inbound_anomaly_score=%{tx.anomaly_score}"
        SecRule RESOURCE:OSVDB_VULNERABLE "@eq 1" chain
                SecRule TX:ANOMALY_SCORE_BLOCKING "@streq on"

This rule will not only check the overall anomaly score of the transaction, but it will also check the resource collection to see if it has been flagged as vulnerable by osvdb. If so, then you can get more aggressive in your blocking. If a client were to actually send an attack payload to this resource, such as with the SQLi example in the OSVDB manual testing data, then the following alerts would be triggered:

[Wed Feb 23 09:24:53 2011] [error] [client ::1] ModSecurity: Warning. Pattern match "\\b(\\d+) ?(?:=|<>|<=>|<|>|!=) ?\\1\\b|[\\'"\\`\\\xc2\xb4\\\xe2\x80\x99\\\xe2\x80\x98](\\d+)[\\'"\\`\\\xc2\xb4\\\xe2\x80\x99\\\xe2\x80\x98] ?(?:=|<>|<=>|<|>|!=) ?[\\'"\\`\\\xc2\xb4\\\xe2\x80\x99\\\xe2\x80\x98]\\2\\b|[\\'"\\`\\\xc2\xb4\\\xe2\x80\x98](\\w+)[\\'"\\`\\\xc2\xb4\\\xe2\x80\x99\\\xe2\x80\x98] ?(?:=|<>|<=>|<|>|!=) ?[\\'"\\`\\\xc2\xb4\\\xe2\x80\x99\\\xe2\x80\x98]\\3\\b|([\\'"\\;\\`\\\xc2\xb4\\\xe2\x80\x99\\\xe2\x80\x98]*)?\\s+(and|or)\\s+([\\s\\'"\\` ..." at ARGS:oid. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "425"] [id "950901"] [rev "2.1.1"] 
[msg "SQL Injection Attack"] [data "' and sleep"] [severity "CRITICAL"] [hostname "localhost"] [uri "/cart"] [unique_id "TWUYtcCoAWQAACYRJnIAAAAA"]
[Wed Feb 23 09:24:53 2011] [error] [client ::1] ModSecurity: Access denied with code 403 (phase 2). String match "on" at TX:anomaly_score_blocking. [file "/usr/local/apache/conf/modsec_current/base_rules/modsecurity_crs_49_inbound_blocking.conf"] [line "12"] 
[msg "Inbound Attack Targeting OSVDB Flagged Resource."] [hostname "localhost"] [uri "/cart"] [unique_id "TWUYtcCoAWQAACYRJnIAAAAA"]

The first alert identifies the SQL Injection attack and then the final alert correlates the anomaly score with the fact that OSVDB has flagged this resource as vulnerable, so blocking is initiated.

Conclusion

This is but the first step showing how web application firewalls can be used to conduct passive vulnerability scanning operations. We will be showing more examples in the future that implement different concepts of monitoring real-time traffic to identify vulnerabilities. Stay tuned!