In addition to acquiring a full memory image of a subject Linux system, it is also valuable for the investigator to gather the contents of process memory associated with suspicious processes, as this will greatly decrease the amount of data that needs to be parsed. Further, the investigator may be able to implement additional tools to examine process memory, such as strings, that may not be practical for full memory contents analysis. Generally, process memory should be collected only after a full physical memory dump is completed, as many of the tools used to assess the status of running processes, and in turn, dumping the process memory of a suspect processes, will impact the physical memory.
As with other live response techniques on a Linux system, to minimize interaction with the subject system during your investigation, consider using trusted (ideally statically linked) binaries from external media such as a CD or thumb drive, as discussed in Chapter 2.
For the purpose of the following case scenario, we will be collecting the results of our tool output to our trusted toolkit thumb drive; however, the results can just as easily be transferred over a netcat listener to a forensic collection system.
Table 3.6
Offsets of Select Fields Within the Task_Struct Object for the DFR-Value Offset
Next 0x00
Prev 0x04
PID 0x30
Case Scenario
“It’s a SYN!”
Scott, the manager of a local Internet Café and Copy shop, calls you because his net- work is very slow and is affecting business. He knows you work relatively close to his shop and asks if you can stop by to take a look at his network to see what the problem is. Upon your arrival, you conduct a few basic queries on the shop’s main server, including the netstat –anp command, as shown in Figure 3.22. You learn that the server is sending numerous SYN requests to a foreign address in a seemingly auto- mated fashion. Further, the netstat output reveals that the process assigned PID 6194 is responsible for the network activity.
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:2208 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:9050 0.0.0.0:* LISTEN - tcp 0 0 127.0.0.1:2207 0.0.0.0:* LISTEN - tcp 0 1 192.168.110.130:59828 xxx.211.23.57:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:55459 xxx.211.22.9:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:48247 xxx.211.22.108:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:45880 xxx.211.23.98:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:60501 xxx.211.23.62:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:43620 xxx.211.22.121:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:57994 xxx.211.23.49:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:48230 xxx.211.22.105:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:44901 xxx.211.22.122:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:57109 xxx.211.23.11:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:45024 xxx.211.23.123:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:57398 xxx.211.22.52:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:42019 xxx.211.22.112:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:46834 xxx.211.23.99:80 SYN_SENT 6194/gol
tcp 0 1 192.168.110.130:59511 xxx.211.22.63:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:48709 xxx.211.22.104:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:34513 xxx.211.23.81:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:48526 xxx.211.22.100:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:40372 xxx.211.22.68:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:46767 xxx.211.22.111:80 SYN_SENT 6194/gol tcp 0 1 192.168.110.130:51766 xxx.211.22.18:80 SYN_SENT 6194/gol Before we actually dump the memory contents of our suspicious process, we’ll first want to gain some context about the process through our live response data collection methods detailed in Chapter 2. In particular, we’ll want to:
Determine system activity in relation to the process with top Gather information about the process with ps
Identify process activity with lsof
Gather information from the /proc directory relating to the process Gather process memory mappings with pmap
After gathering this information about the suspicious process, we can choose from a variety of methods to dump the memory associated with the process to our live response external media for further examination.
Process Activity on the System
Using the top command, we can obtain real-time CPU usage and system activity information. Of particular interest in our investigation will be the identification of any unusual processes that are consuming system resources. Tasks and processes listed in the top output are in descending order by virtue of the CPU consumption. By default, the top output refreshes every 5 seconds. Examining the top output on the subject system we see an unusual process named “gol,” assigned PID 6194, that is consuming more system resources relative to other tasks in the top output.
scott@xxxxxxx:/media/thumbdrive/Linux-IR$ ./top
top - 17:45:43 up 27 min, 4 users, load average: 1.27, 0.79, 0.72 Tasks: 119 total, 4 running, 115 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.0%us, 7.6%sy, 0.0%ni, 0.0%id, 88.0%wa, 1.3%hi, 1.0%si, 0.0%st Mem: 657824k total, 559744k used, 98080k free, 49124k buffers Swap: 409616k total, 0k used, 409616k free, 267308k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4651 root 15 0 43504 15m 6520 S 3.0 2.4 0:40.82 Xorg 6194 Scott 15 0 812 508 460 S 2.0 0.1 0:10.75 gol 7204 root 26 10 1872 736 520 R 2.0 0.1 0:01.75 updatedb 7244 root 18 0 3916 2416 1340 R 2.0 0.4 0:00.06 lsb_release 6144 scott 15 0 77628 17m 10m S 0.7 2.7 0:02.58 gnome-terminal ■ ■ ■ ■ ■
2260 root 10 -5 0 0 0 S 0.3 0.0 0:00.39 kjournald 5452 scott 15 0 15932 2304 1372 S 0.3 0.4 0:01.80 gnome-screensav 6233 scott 15 0 2316 1176 880 R 0.3 0.2 0:01.54 top 1 root 18 0 2912 1844 524 S 0.0 0.3 0:00.81 init 2 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 3 root 34 19 0 0 0 R 0.0 0.0 0:00.03 ksoftirqd/0 4 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0 5 root 10 -5 0 0 0 S 0.0 0.0 0:00.02 events/0 6 root 11 -5 0 0 0 S 0.0 0.0 0:00.02 khelper 7 root 12 -5 0 0 0 S 0.0 0.0 0:00.00 kthread 30 root 10 -5 0 0 0 S 0.0 0.0 0:00.03 kblockd/0 31 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
Gather Information About the Process with ps
Now that we’ve identified a potentially rogue process, we can gain further information about the process by using the ps command. To display detailed information about all running processes, we’ll query the subject system with ps –aux. To discover instances of our suspicious process by name (not PID), we can also parse the output with grep. Through this process, we learn that the process “gol” has three different associated PIDs: 6192, 6193, and 6194. scott@xxxxxxx:/media/thumbdrive/Linux-IR$ ./ps -aux | grep gol Warning: bad ps syntax, perhaps a bogus ‘-’? See http://procps.sf.net/faq.html scott 6192 0.0 0.0 620 148 pts/0 S 17:31 0:00 ./gol scott 6193 0.0 0.0 620 68 pts/0 S 17:31 0:00 ./gol scott 6194 1.2 0.0 812 508 pts/0 S 17:31 0:12 ./gol scott 7397 0.0 0.1 2884 752 pts/1 R+ 17:47 0:00 grep gol scott@xxxxxxx:/media/thumbdrive/Linux-IR$ ./ps -ef | grep gol scott 6192 1 0 17:31 pts/0 00:00:00 ./gol scott 6193 6192 0 17:31 pts/0 00:00:00 ./gol scott 6194 6192 1 17:31 pts/0 00:00:13 ./gol scott 7421 6217 0 17:48 pts/1 00:00:00 grep golIdentifying Process Activity with lsof
As discussed in Chapter 2, we can identify files and network sockets opened by running processes using the lsof (“list open files”) utility. This will provide us valuable insight into the system and network activity relating to our suspect process. Since we know the suspicious PIDS associated with gol, we can query each PID with lsof. scott@xxxxxxx:/media/thumbdrive/Linux-IR$ ./lsof -p 6192 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME gol 6192 scott cwd DIR 8,1 4096 932227 /tmp/eyt gol 6192 scott rtd DIR 8,1 4096 2 /gol 6192 scott txt REG 8,1 400492 932228 /tmp/eyt/gol
gol 6192 scott mem REG 0,0 0 [heap] (stat: No such file
or directory)
gol 6192 scott 3u sock 0,5 18827 can’t identify protocol
gol 6192 scott 4u IPv4 18828 UDP *:27015
scott@xxxxxxx:/media/thumbdrive/Linux-IR$ ./lsof -p 6193 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
gol 6193 scott cwd DIR 8,1 4096 932227 /tmp/eyt
gol 6193 scott rtd DIR 8,1 4096 2 /
gol 6193 scott txt REG 8,1 400492 932228 /tmp/eyt/gol
gol 6193 scott mem REG 0,0 0 [heap] (stat: No such file or
directory)
gol 6193 scott 3u sock 0,5 18827 can’t identify protocol
gol 6193 scott 4u IPv4 18828 UDP *:27015
scott@xxxxxxx:/media/thumbdrive/Linux-IR$ ./lsof -p 6194 COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME gol 6194 scott cwd DIR 8,1 4096 932227 /tmp/eyt gol 6194 scott rtd DIR 8,1 4096 2 /
gol 6194 scott txt REG 8,1 400492 932228 /tmp/eyt/gol
gol 6194 scott mem REG 0,0 0 [heap] (stat: No such file or directory)
gol 6194 scott 0u IPv4 298684 TCP xxxxxxx.local:37342->xxx .234.77.19:www (SYN_SENT) gol 6194 scott 1u IPv4 298185 TCP xxxxxxx.local:54145->xxx
.234.75.29:www (SYN_SENT) gol 6194 scott 2u IPv4 298186 TCP xxxxxxx.local:51957->xxx
.234.75.30:www (SYN_SENT) gol 6194 scott 3u sock 0,5 18827 can’t identify protocol gol 6194 scott 4u IPv4 18828 UDP *:27015
gol 6194 scott 5u IPv4 298187 TCP xxxxxxx.local:35663->xxx .234.75.31:www (SYN_SENT) gol 6194 scott 6u IPv4 298188 TCP xxxxxxx.local:48974->xxx
.234.75.32:www (SYN_SENT) gol 6194 scott 7u IPv4 298189 TCP xxxxxxx.local:60421->xxx
.234.75.33:www (SYN_SENT) gol 6194 scott 8u IPv4 298190 TCP xxxxxxx.local:51866->xxx
.234.75.34:www (SYN_SENT) gol 6194 scott 9u IPv4 298191 TCP xxxxxxx.local:46478->xxx
.234.75.35:www (SYN_SENT) gol 6194 scott 10u IPv4 298192 TCP xxxxxxx.local:44929->xxx
.234.75.36:www (SYN_SENT) gol 6194 scott 11u IPv4 298193 TCP xxxxxxx.local:52356->xxx
gol 6194 scott 12u IPv4 298194 TCP xxxxxxx.local:38429->xxx .234.75.38:www (SYN_SENT) gol 6194 scott 13u IPv4 298195 TCP xxxxxxx.local:33105->xxx
.234.75.39:www (SYN_SENT) We learn that the executable program “gol” resides in an anomalous location on the system, the “/tmp/eyt” directory. Further, the lsof output reveals that PIDs 6192 and 6193 are not actively attempting network connectivity, whereas PID 6194 is the process that is generating numerous SYN packet requests from Scott’s network. To confirm our findings, we’ll use lsof with the –i flag, which shows both User Datagram Protocol UDP and Transmission Control Protocol (TCP) network connections.
scott@xxxxxxx:/media/thumbdrive/Linux-IR$ ./lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
gol 6192 scott 4u IPv4 18828 UDP *:27015 gol 6193 scott 4u IPv4 18828 UDP *:27015
gol 6194 scott 0u IPv4 310801 TCP xxxxxxx.local:51670->xxx .234.118.148:www (SYN_SENT) gol 6194 scott 1u IPv4 310302 TCP xxxxxxx.local:35435->xxx
.234.116.158:www (SYN_SENT) gol 6194 scott 2u IPv4 310303 TCP xxxxxxx.local:45055->xxx
.234.116.159:www (SYN_SENT) gol 6194 scott 4u IPv4 18828 UDP *:27015
gol 6194 scott 5u IPv4 310304 TCP xxxxxxx.local:55432->xxx .234.116.160:www (SYN_SENT) gol 6194 scott 6u IPv4 310305 TCP xxxxxxx.local:56676->xxx
.234.116.161:www (SYN_SENT) gol 6194 scott 7u IPv4 310306 TCP xxxxxxx.local:36092->xxx
.234.116.162:www (SYN_SENT)