Detecting Malicious Behavior by Unmasking WebSockets

WebSockets allow a single TCP connection to have full duplexing communications.  This type of connection reduces the overhead of HTTP polling, where the client would have to constantly request information from the server in order to get updates.  One of the features of WebSocket connection is the ability for the client to mask its payload when sending messages to the server.  Since this feature can also be used to hide malicious traffic, we’ll use Suricata’s Lua engine to unmask and inspect WebSocket payloads.

You’d think the masking was done to hide the clear text data in HTTP but it was actually put in place to protect against cache poisoning.  According to RFC 6455, an attacker could send what looked like a GET request to a server under their control and respond so that it looks like a typical response to that request.  There is the potential to poison the caches for a particular user and anyone else using the same one.

This masking can create a barrier to detecting malicious activity directed at servers using web sockets.  Luckily enough, all the information is present in the frame for us to decode the payload and take a peek inside.

Wireshark will automatically decode these payloads for you.  All you need to do is select the “Unmasked data” tab at the bottom.  The first two bytes in the payload set various flags and indicate the size.  The “Mask” flag is the one that needs to be checked in order for there to be masking used.  The 4 bytes right after the payload length are used to XOR the rest of the data.

Figure 1: masked Websockets data

Figure 2: unmasked Websockets data

Figure 3: Websockets decoded by Wireshark

The formula for both encoding and decoding the data is the same.  According to the RFC, the formula is as follows:

j = i MOD 4
transformed-octet-i = original-octet-i XOR masking-key-octet-j

If the above doesn’t quite make sense then here is some pseudo code that is similar to the Lua script we’ll see later. There will be some differences because of the way Lua handles array positions.

for i=0, i<len(payload), i++
            key = arr_of_keys[i % 4]
            decoded_data = payload[i] ^ key

CVE-2018-1270 is a good example of needing to inspect masked WebSocket traffic.  In this example, the exploit is using java.lang.Runtime to execute a Netcat command.  The images below show both the masked and unmasked traffic.  Since the traffic is masked when the IDPS sees it, a normal signature isn’t going to be useful without some kind of decoding first.

Figure 4: CVE-2018-1270 exploit masked by WebSockets

Figure 5: CVE-2018-1270 exploit unmasked

The signature below checks to see if this is a masked WebSocket request and, if so, calls the webscoket_decode.lua script.  You could add another signature to this and check for the `Upgrade: websocket` header in the server response and combine them with flowbits.  This could reduce the number of times the signature would call the Lua script and be more certain of the type of traffic being passed to the script.  We’ll keep it simple and use just one signature.

alert tcp any any -> any 8080 (msg:"Pivotal Spring Framework Remote Code Execution (CVE-2018-1270)"; 
flow:established,to_server; content:"|81|"; depth:1; byte_test:1,&,0x80,0; luajit:scripts/websocket_decode.lua; 
reference:url,exploit-db.com/exploits/13918 sid:20181270; rev:1;)

The init() function of the lua script grabs whichever buffer is needed.  In this case the payload is what we’ll need to unmask the data.  After that we move into the match() function and check to make sure there’s data in the buffer.  If there is then we can extract a 4 byte key using substring from the payload. Remember that all arrays in Lua begin with 1 not 0.

function init (args)
    local needs = {}
    needs["payload"] = tostring(true)
    return needs
end

function match(args)
    local a = tostring(args["payload"])
    if #a > 0 then
        local key = string.sub(a,3,6)
        local decoded_payload = decode_payload(a, key)
        if string.find(decoded_payload, 'java.lang.Runtime.*exec') then
            return 1
        end
        return 0
    end
return 0
end

The payload and key then gets passed to the decode_payload() function. In this function we start decoding right after the 4 byte key, so we set i in the for loop to 7.  In order to make sure we are using the correct byte in the key array this is typically done by key[i%4] which would give us a result between 0-3.  To make sure this aligns properly we just subtract 7 from i.  After that we XOR each byte with the appropriate key and store the results in an array.  Once the array is complete we need to convert it back to a string to make things easier to search with.

function decode_payload(a, key)
    local tohex = bit.tohex
    local unmask = {}
    for i=7,#a,1
    do
        xor_key = string.byte(key, mod(i-7,4))
        unmask[i-6] = string.char(bit.bxor(string.byte(a,i),xor_key))
    end
    local unmasked_string = arr_to_str(unmask)
    return unmasked_string
end

Once we have a string we can search the data using any keyword we’d like.  For instance, I can use `string.find()` and use regex to look for `java.lang.Runtime` and the `exec` command to trigger the alert.

        if string.find(decoded_payload, 'java.lang.Runtime.*exec') then
            return 1
        end
        return 0

While Suricata’s signatures have many options and can detect many things on their own, there are some circumstances where more advanced logic is needed.  Masked WebSockets are a great example of that and is one of many uses of Suricata’s Lua scripting engine.