Analyzing and Detecting
Malicious Code and
Intruders
In This Chapter
✓ Analyzing abnormal system processes
✓ Detecting unusual or hidden files
✓ Locating rootkits and backdoors
✓ Detecting and preventing network sniffers
THE INTERNET HAS ALTERED THE WAY BUSINESS IS CONDUCTED.Nearly everything is accessible with the click of a mouse. While individuals and organizations use the Internet daily for e-mail messaging, e-commerce, or instant messaging, rarely do we consider the dangers posed by hack- ers, viruses, worms, and Trojan horses as we click away from Web site to Web site. We place trust in our computers and assume that they offer protection for our sensitive information. The largely unregulated Internet harbors numerous threats. While hackers, crackers, and law-abiding citi- zens all live together in the cyber world, the average Internet user is usually unaware of the pres- ence of malevolent individuals. Malicious code in the form of viruses, worms, and Trojan horses traverses the Internet, exploiting flaws that exist in the operating systems and applications of many an innocent user.
Because of these cyber threats, individuals and organizations need to change the way they operate and manage their networks. Security training frequently is focused on basic security issues and not on responding to Internet threats. During typical user awareness training the specifics of e-mail message attachments (and their ability to hide malicious code or masquerade as legitimate documents or graphics files) are not addressed sufficiently. We cannot blame com- puter users for virus proliferation if they are thrust into a computerized environment without first having been exposed to some incident response basics. This chapter focuses on detecting, analyzing, and responding to threats posed by intruders and malicious coders.
System Processes
Simply put, a processis an executing program. Oftentimes a thread is confused with a process. Similar to a process, a threadis the unit of execution to which the operating system assigns pro- cessing time (a time slice), and it consists only of data flow and control. Threads provide a useful programming technique for dividing work into separate pieces. Thread execution is monitored and scheduled solely by the operating system, and every process is started with the execution of a single thread, usually called the primary thread. Even where there are multiple threads, they still use only the address space of a single process.
Every program running on a computer uses at least one process, consisting of the memory address (space) allocated to the process by the computer to run the program and the ability of the computer’s OS to monitor the program throughout the execution process. In modern-day 32-bit multitasking operating systems, processes are managed for the most part as isolated entities so that if one process crashes, the others are generally not affected. The resources they use (memory, disk, I/O, and CPU time) are virtual in nature, meaning that every process has its own set of virtual resources, untouched by other processes. Even when several programs are running at the same time, each process has its own address space and flow of control. Thus, a process is a place to work and a way to keep track of what a program is doing.
Detecting Abnormal System Processes
Monitoring system processes can be both complicated and time-consuming. The ability to identify suspicious or abnormal processes first requires a thorough understanding of the types of processes one normally would expect to be executing on a system at a given time as well as how they should behave. However, due to the enormous number of processes running simultaneously and their constantly changing nature, it is nearly impossible for a single individual to monitor all of them continually. To make the job of monitoring system resources easier, some organizations divide system monitoring among several different personnel. Each individual is assigned a partic- ular system resource to monitor.
The value of information that can be gathered from a periodic snapshot of currently executing processes is limited. Organizations may need to utilize a range of information gathering and mon- itoring mechanisms to assist in collecting and analyzing data associated with processes, and to alert incident response personnel to any suspicious activity.
In general, monitors should look for the following signs:
✓ Unusual resource utilization or process behavior
✓ Missing processes
✓ Added processes
✓ Processes that have unusual user identification associated with them (such as an ID belonging to someone not employed by an organization)
Abnormal system processes can be caused by
✓ Malicious code (viruses, Internet worms, and Trojan horse applications)
✓ Spyware (software that transmits information back to a third party withoutnotifying the user)
As mentioned in Chapter 1, log files should be checked for connections from unusual locations or for any unusual activity. All versions of Windows NT have a built-in Event Viewer that allows you to check for unusual logon entries, failures of services, or abnormal processes. Data collected from log files can help in the analysis of the process behavior.
These include the following:
✓ The process names and startup times
✓ The status of the process (for example, time duration, resources consumed, and so on)
✓ Which user executed the process
✓ The amount of system resources used (for example, CPU, memory, disk, and time) by specific processes over time
✓ System and user processes and services executing at any given time
✓ The method by which each process is normally started (for example by the system administrator, other users, other programs, or spawned from other processes) and what authorization and privileges have been assigned to those processes
✓ Hardware devices used by specific processes
✓ Files currently opened by specific processes
When reviewing operating system or network logs, look for the following:
✓ Processes consuming excessive resources (for example, memory, disk, or CPU time)
✓ Processes starting or running at unexpected times
✓ Unusual processes not the result of normal authorized activities (for example, packet sniffing, password cracking, and so on)
✓ Processes that prematurely terminate
✓ Previously inactive user accounts that suddenly begin to spawn processes and consume computer or network resources
✓ Unexpected or previously disabled processes, which may indicate that a hacker or intruder has installed his own version of a process or service
✓ A workstation or terminal that starts exhibiting abnormal input/output behavior
✓ Multiple processes with similar names (for example, when a computer virus runs Explorer.exe using a capital letter to disguise itself rather than the actual process, which is called explorer.exe by the operating system)
✓ An unusually large number of running processes
Using the Windows Task Manager to View
Running Processes
The Windows Task Managerprovides information about programs and processes running on your computer. It also displays the most commonly used performance measures for any running processes. While the Task Manager is useful for monitoring key indicators of your computer’s per- formance, it also permits you to quickly see the status of the programs and processes that are running and even terminate programs (when they freeze or stop responding). You may assess the activity of running processes using numerous parameters and viewing graphs and data on CPU and memory usage (see Figures 5-1, 5-2, and 5-3). The name of the Task Manager program is taskmgr.exe, and there are several manners in which to access the program. One way is to type
taskmgrin the Run box on the Start menu or at the command (DOS) prompt to bring up the pro- gram. Another convenient access method is to create a shortcut link to the program directly on the Windows desktop. Finally, you may type Ctrl+Alt+Del (a.k.a. “the three finger salute”) at any time while the operating system is running.
To create a desktop shortcut to the taskmgr.exe program, you first need the location of this file in the Windows OS. The default location is C:\winnt\system32for Windows NT and 2000 and c:\windows\system32for Windows XP.
Figure 5-1:The Windows XP Task Manager window
showing all currently running processes
Figure 5-2:The Select Columns box in Windows XP Task Manager
Figure 5-3:Viewing CPU performance with
Windows XP Task Manager
Default Processes in Windows NT, 2000, and XP
In order to detect errant or unauthorized processes using the Windows Task Manager, it is helpful to understand which processes run by default during normal system operations. There are a num- ber of default processes that are automatically run by the Windows operating system (they vary depending upon which version of Windows is being used). The following is an alphabetical listing of some of the default processes that are commonly run under Windows NT/2000/XP, along with a brief explanation of their functions:
✓ Csrss.exe.Csrss or Client/Server Run-time Subsystem is an essential subsystem that must remain running at all times. Csrss provides text window support, shutdown, and hard-error handling to the Windows NT environment subsystems.
✓ Explorer.exe. This is the Graphical User Interface in which we see the familiar taskbar and desktop environment. Explorer lets users open documents and applications from various icons and Windows cascading menus.
✓ Lsass.exe. This process helps handle security administration on the local computer, including user access and permissions. This process is responsible for authenticating users for the Winlogon service and is shared by the Netlogon service.
✓ Mstask.exe. This is the task scheduler service. It is responsible for running tasks at times that are predetermined by the user.
✓ Services.exe. This is the Windows Services Control Manager, which is responsible for starting and stopping system services and works with other Windows machines on the network to maintain a current list of available resources.
✓ Smss.exe. Session Manager Subsystem is responsible for starting the user session. Smss is initiated by the system thread and is responsible for a range of actions, including launching the Winlogon and Win32 (Csrss.exe) processes and setting system variables.
✓ Spoolsv.exe. This is the Windows spooler service and is responsible for the management of spooled print and fax jobs.
✓ Svchost.exe. A generic process, which acts as a host for other processes running from DLLs; therefore, don’t be surprised to see more than one entry for this process.
✓ System. This permits system kernel-mode threads to run as the System process.
✓ System Idle Process. This process is a single thread running on each processor. Its sole task is accounting for processor time when the system isn’t processing other threads.
Process-Monitoring Programs
In the nonvirtual world, when a complex task must be carried out, tools are available to make the job easier. The same principal applies in the virtual world of computers. As mentioned earlier in this chapter, monitoring system processes can be time-consuming and complex. Most Unix oper- ating systems ship with a command-line tool called ps. The pscommand allows you to list what processes are being executed by the machine on which the command was entered. Administrators
can use it along with the-efoption (for example, ps -ef) to get a full listing of all processes on the Unix system. While Windows NT, 2000, and XP come with Task Manager, they don’t offer a sig- nificant level of detail about individual processes. Luckily, there are several programs available that can make the job of monitoring system processes less onerous. At the Sysinternals Web site (www.sysinternals.com) there is a downloadable suite of advanced freeware utilities called PsToolscoded by Mark Russinovich. It can assist in the monitoring and gathering of detailed infor- mation about system processes under the Windows operating system. The PsTools suite includes the following tools, which can be downloaded individually or as a package:
✓ PsExecexecutes processes remotely.
✓ PsFileshows files opened from a remote location.
✓ PsGetSiddisplays the SID (security identifier) of a computer or a user.
✓ PsInfolists information about a system.
✓ PsKillterminates processes by name or process ID.
✓ PsListlists detailed information about processes.
✓ PsLoggedOnshows who’s logged on locally and via resource sharing.
✓ PsLogListdumps event log records.
✓ PsServiceviews and controls services.
✓ PsShutdownshuts down and optionally reboots a computer.
✓ PsSuspendsuspends processes.
✓ PsUptimeshows how long a system has been running since its last reboot.
Since all of the PsTools are command-line tools, they must be run from a command (DOS) prompt. You will need to add the folder that they are stored in to your system’s path in order to run them from any directory other than the directory or folder in which they were placed. By plac- ing these tools in your Winnt directory (Windows NT/2000) or your Windows directory (Windows XP), they will automatically be included in your system’s Path statement. If you wish to place these tools in their own directory (for example, C:\PsTools) yet still have Windows locate them automatically, perform the following steps.
Under Windows NT 4.0, do the following:
1. Right-click the My Computer icon.
2. Select Properties from the context menu.
3. Double-click the System icon.
4. Click the Environment tab.
5. Select Path from the list of system variables.
6. Edit the Path statement by adding the directory in which PsTools is stored.
Under Windows 2000, XP, do the following:
1. Right-click the My Computer icon. (In Windows XP, the My Computer Icon may be located in the Start menu.)
2. Select Properties from the context menu.
3. Select the Advanced tab.
4. Click the Environment Variables button.
5. From the System Variables window, highlight Path entry and then select Edit.
6. Add (append) the location of the PsTools directory to the list of directories in the Path by using a semicolon directly after the last statement in the Path, then adding the location of the PsTools Directory (see Figure 5-4).
Figure 5-4:Adding the location of the
PsTools directory to your directories list via Windows XP Pro Environmental Variables editing utility
Another useful tool from the Sysinternal Web site is Process Explorer (see Figure 5-5) by Mark Russinovich. With this powerful point-and-click, Windows-based, freeware utility, you can find out who owns each process, and, for each of these processes, what files, Registry keys, and other objects are open. In addition, Process Explorer shows which DLLs have loaded and which handles opened with each process. This makes it a powerful tool for understanding the internal behavior of applications, as well as for tracking down handle leaks and DLL version mismatches.
Figure 5-5:Process Explorer 5.25
Unusual or Hidden Files
It is sometimes difficult to determine if a system has been compromised. It is important therefore to search periodically for any unusual or hidden files that may have bypassed intrusion detection and antivirus protection. Hidden files can be used to conceal hacker tools, malicious code, and sensitive information (for example, password-cracking programs, password files from other sys- tems, and so on). A number of recently developed malicious programs have exploited the default behavior of Windows operating systems to hide file extensions. This behavior can be used to trick users into executing code by making a file appear to be something it is not. Multiple e-mail-borne viruses are known to exploit this vulnerability.
Viewing Hidden Files in Windows
Windows operating systems contain an option to “Hide file extensions for known file types.” The option is enabled by default, but a user may choose to disable this option in order to have file extensions displayed by Windows.
To show hidden files, folders, and filename extensions under Windows, perform the following steps.
For Windows NT, do the following:
1. Click the Start button, select Settings, then Control Panel.
2. From the View menu, select Options.
3. Click the View tab.
4. From the View tab, select “Show all files.”
5. Deselect the “Hide file extensions for known file types” option.
6. Click OK to complete the changes. For Windows 2000, do the following:
1. Click the Start button, select Settings, then Control Panel.
2. From the Tools menu, select Folder Options.
3. Click the View tab.
4. Under “Hidden files and folders,” select “Show hidden files and folders.”
5. Deselect the “Hide file extensions for known file types” option.
6. Deselect “Hide protected operating system files.” (Note: Windows 2000 will display a dialog box asking for confirmation. Be sure to read and understand the information contained in the dialog, and then click Yes.)
7. Click OK to complete the changes. For Windows XP, do the following:
1. Click the Start button, and select Control Panel.
2. From the Tools menu, select Folder Options.
3. Click the View tab.
4. Under “Hidden files and folders,” select “Show hidden files and folders.”
5. Deselect the “Hide file extensions for known file types” option.
6. Deselect “Hide protected operating system files.” (Note: Windows XP will display a dialog box asking for confirmation. Be sure to read and understand the information contained in the dialog, and then click Yes.)
In all versions of Windows you can view hidden files at the command prompt by typing
dir /ah.
Viewing Hidden Files under Unix/Linux
Even when using Unix or Linux, it is important to search the system for unusual or hidden files. Under Unix and Linux systems, these are files that start with a period and normally are not shown by the lscommand. (The lscommand lists all of the files and subdirectories you have in a given directory.) These files are often used by hackers to hide tools and password-cracking programs. A common technique used with Unix/Linux systems is to put a hidden directory or file in a user’s account with an unusual name such as .. (dot-dot-space) or ..^G(dot-dot-control-G). Luckily, the built-in Find program in Unix can help seek out these types of concealed files. Here are two examples:
find / -name “.. “ -print -xdev find / -name “.*” -print -xdev
Another favorite hacker ploy is to exploit SUID root programs. (SUID root refers to Set User ID root.) SUID root allows the program to carry out functions that only system administrators with full root privileges would be permitted to perform. Programs that run as root have complete access to a Unix system, and SUID programs run as root regardless of who is executing them. Programs that run low-level networking routines and control functions such as graphical display, changing passwords, and logging in are all examples of programs that require a user with full root privileges to execute them. Intruders often leave behind SUID copies of /bin/shor /bin/timeto allow them to gain root access later. Find all SUID programs on your system and keep track of