There are different approaches to the investigation of the binary code. The techniques of blind search do not suggest anything except methodically checking different
combinations of input data (as a rule, these might be strings of different lengths used mainly for detecting overflowing buffers). Purposeful analysis requires fundamental knowledge of the system, a nontrivial way of thinking, and experience of designing
"industrial" software products. The hacker must exactly know before-hand what must be found in the result of the search, the pet errors of the developers, the most probable locations where bugs are concentrated, and the specific features and limitations of different programming languages. Disassembling skills alone are not enough (I assume that you know how to disassemble).
A brute-force attack doesn't always produce a positive result, and lots of security holes may remain undetected. On the other hand, studying a disassembled listing also is not a guarantee of success. You might spend years but never find a worthy bug. This depends on your luck. Therefore, before starting the disassembling, make sure that everything in your power has been done already. At the least, it is necessary to carefully investigate the input fields by supplying long strings. At the most, it is necessary to try typical conceptual vulnerabilities. In particular, if the firewall being attacked freely passes strongly-fragmented TCP packets, then there is no need to disassemble it. It
immediately becomes clear that to detect such a hole in the binary code, it is necessary to clearly understand the working mechanism of the firewall and assume its existence beforehand. Because this is so, it will be much easier to form a fragmented packet on your own and look at how the firewall would react to it. Forged packets are a different matter. When sending them to the target computer, the hacker must know, which fields are checked and which are not. Here it is impossible to do without disassembling. It is necessary to locate the code responsible for header processing and analyze the criteria, according to which the packets are discarded. After all, the disassembler is only an instrument, and it cannot play the role of a generator of ideas. Aimless disassembling is a road that leads nowhere.
To a considerable extent, any program is made up of libraries, and analyzing these doesn't make any sense. All libraries were analyzed in detail long ago, and no radically new holes can be found there. In addition, the overwhelming majority of libraries are distributed with the source code, which makes disassembling them even more
senseless. In most cases, the library code is located after the main code of the
program, and separating it is an easy job. However, it is much more difficult to identify the name of the library functions; without knowing them, you'll become stuck in the disassembled listings as though they were a bog. Fortunately, the vast majority of
standard libraries are automatically recognized by IDA. The signatures of exotic libraries from third-party developers can be manually added at any moment, thanks to IDA, which provides such a possibility (for more details, see Hacker Disassembling Uncovered by Kris Kaspersky and standard documentation supplied as part of the IDA product).
IDA makes the decision about loading a specific signature database on the basis of analysis of the start-up code; thus, "foreign" libraries might remain unrecognized. The same happens when loading memory dumps with damaged or missing start-up code or an incorrectly set entry point (which is the chronic ailment of all dumpers). Therefore, if most of the program functions remain unrecognized (Fig. 8.1), try to manually load the signature database. To achieve this, select the FLIRT Signature file command from the File → Load file menu. After that, a long list of all libraries known to IDA will appear (Fig. 8.2). Which one should you choose? If you are a beginner with disassembling and cannot identify the required library manually, you can use the trial method, loading one signature after another and trying to achieve maximum extension of the blue filling (Fig.
8.3).
Figure 8.1: The IDA Pro navigator. The domination of blue means that most library functions remain unrecognized, because the disassembler could not determine the type of compiler. In this case, appropriate signatures must be loaded manually
Figure 8.2: The list of signatures known to IDA
Figure 8.3: The domination of blue filling means that now everything is OK
When viewing the list of recognized and imported functions, it is necessary to choose the most dangerous ones. First, these are functions that accept the pointer to the allocated buffer and return data of an unpredictable size (such as sprintf or gets). The
functions with an explicit limitation on the maximum allowed buffer length (fgets, GetwindowText, GetFullPathName) are considerably less dangerous, although even for such functions no one can provide any guarantee that they would behave legally. The complete list of potentially dangerous functions is too long. Therefore, it won't be provided here. However, a brief listing of the most dangerous functions is provided in Table 8.1.
Table 8.1: Potentially dangerous functions of the standard C library
Function Overflow probability
gets Extreme
strcpy/strcat Extreme
memmove/memcpy High
The programmers often allocate buffers of a considerably smaller size and this
precaution doesn't work. Consider the example shown in Listing 8.2. If the user enters a string 100 or more bytes long, then buffer overflow is inevitable and no length limitations would help.
Listing 8.2: Program vulnerable to overflow despite protection measures
#define MAX_BUF_SIZE 100
#define MAX_STR_SIZE 1024
char *x; x = malloc(MAX_BUF_SIZE); fgets(x, MAX_STR_SIZE, f) ;
As was already mentioned, a complete list of potentially dangerous functions will not be provided here. It is much better to learn how to act according to specific circumstances.
Load the program being studied into a disassembler (the preferred one is IDA Pro), press <Shift>+<F3>, and click the L column with the mouse ("L" stands for library), thus separating library functions from all other ones. (This function is missing in the console version of the application.) Then take a weighty volume of the user manual or start MSDN and view the prototypes for all functions listed here (something like char*,
void*, LPTSTR, etc.). If this buffer accepts the data returned by the function, then it is worth trying to check how it would react to overflow.
Press <Enter> to go to the beginning of the function, then choose the following menu options: View → Open Subview → Cross Reference. The window containing cross-references will open. Each cross-reference leads to the point, at which the function being investigated was called. Depending on the specific features of the compiler and the programming style of the developer of the investigated application, the call might be direct (like call our_func), or indirect (something like mov ecx, pclass/mov ebx, [ecx + 4]/call ebx/.../pclass dd xxx/dd offset our_func). In the latter case, cross-references to our_func will lead to dd offset our_func, and in this case it will be difficult to locate the point, at which it was actually called. As a rule, hackers in such cases start the debugger and set a breakpoint to our_func, after which they write the EIP of all locations, from which it is called (by the way, this process
becomes considerably faster in the latest IDA versions because of the availability of the integrated debugger).
Finally, the hacker comes close to the environment of the calling code. If the argument determining the size of the buffer being accepted is a direct value (something like push 400h, as in Listing 8.3), this is a good sign, indicating that the hole, most probably, is somewhere near. If this is not the case, do not fall into despair. Instead, scroll the cursor slightly upward and view where this size is initialized. It probably represents a constant passed via a long chain of variables or even arguments of the parent functions.
Listing 8.3: Direct value of the maximum buffer length passed to the function .text:00401017 PUSH 400h
.text:0040101C MOV ECX, [EBP + var_8]
.text:0040101F PUSH ECX .text:00401020 CALL _fgets
The course of actions is shown in Fig. 8.4. Take a library function, the prototype of which allows potential overflow capability (step 1). Then, using cross-references, go to the environment near the point of its call (step 2). View the limitation of the maximum allowed length of the return data and compare it with the size of the allocated buffer (step 3). Based on the result of this comparison, it is possible to draw a conclusion about the possibility (or impossibility) of the overflow.
Figure 8.4: Drawing a conclusion about the possibility (or impossibility) of overflow
Now find the code that allocates memory for the buffer. As a rule, this task is carried out by the malloc and new functions. If the argument determining the size of memory to be allocated also is a constant and this constant is smaller than the maximum allowed
length of the return data, then the hole is detected. After that, the hacker can proceed
with analysis of the possible methods of influencing the overflowing buffer using the input data fields.
The rules of designing safe code state that before allocating a buffer, it is necessary to determine the exact size of data that must be placed there. In other words, in a
correctly-written program the call to malloc or new must always be preceded by
strlen, GetwindowTextLength, or something similar. Otherwise, the program will have a potential vulnerability. The presence of a preventive check of the size alone
doesn't guarantee program stability. This is because the requested size is not always determined correctly, especially if data from several sources are loaded into the buffer.
Dealing with local variables is more complicated in this respect, because their size has to be explicitly specified at compile time, when the length of return data is not yet known.
No wonder that overflowing buffers are most frequently encountered among local variables.
Local variables are stored in stack frames, also known as automatic memory. Each function is allocated its own personal frame, into which all local variables belonging to that function are loaded. Most frequently, the frame is formed by the sub esp, xxx machine commands; more rarely, it is formed by the add esp, -xxx command, where xxx is the frame size in bytes. Current versions of IDA Pro by default interpret all direct values and unsigned numbers. Therefore, the conversion of xxx into -xxx must be carried out manually by pressing the <-> key.
Unfortunately, in general it is impossible to divide a monolithic frame into individual local variables. This is because the compiler destroys the initial information, thus making analysis ambiguous. However, for the goals here, the capabilities of the automatic analyzer built into IDA Pro are more than enough. Assume that local buffers most
frequently have the byte * type and their size is at least 5 bytes (although, as statistics have shown, most frequently overflow errors are encountered in 4-byte buffers, which in the course of superficial analysis can be easily confused with DWORD).
As an example, consider the stack frame in Listing 8.4, parsed by the automatic IDA Pro analyzer, and try to detect local buffers there.
Listing 8.4: Local variables automatically recovered by IDA
.text:00401012 sub_401012 proc near ; CODE XREF: start + AF↓p
.text:00401012 var_10 = dword ptr -l0h .text:00401012 var_C = dword ptr -0Ch .text:00401012 var_8 = dword ptr -8 .text:00401012 var_4 = dword ptr -4 .text:00401012
The var_38 variable has the DWORD type and occupies 4 bytes (the size of the variable can be determined by subtracting the address of the current variable from the address of the next variable: (-34h) - (-38h) = 4h. It doesn't look like a buffer.
The var_34 variable has the BYTE type and occupies l0h bytes, which is typical for local buffers. The same is true for the var_20 variable. The var_24 variable, although it has the BYTE type, takes only 4 bytes. Therefore, it might represent either a compact local buffer or a simple scalar variable (note that the latter is encountered much more frequently). Until all explicit buffers are investigated for the possibility of overflow, it doesn't make sense to consider such "candidates for the buffer role."
Viewing the disassembled function code, find all references to the detected buffer and analyze the possibilities of its overflow. Consider, for example, Listing 8.5.
Listing 8.5: Passing the pointer to a local buffer text:0040100B PUSH 300
text:0040100D LEA EAX, [EBP + var_34]
text:00401010 PUSH EAX text=00401011 CALL _fgets text:00401016 ADD ESP, 0Ch
It becomes immediately clear that the var_34 variable is used for storing the input string (which means that this is a buffer) with the maximum allowed length of 300h
bytes. At the same time, the local variable has a length of l0h bytes. It is even possible that var_34, var_24, and var_20 are "parts" of a single buffer; however, this
wouldn't change anything because their total size is considerably smaller than 300h.
If, despite all efforts, overflowing buffers could not be found among local variables, it is possible to try searching among the ruins of the dynamic memory, tracing all cross-references to functions such as new and malloc, and analyzing the vicinities of the points of their calls.
Nevertheless, do not rejoice prematurely after locating an overflowing buffer in one of deeply-nested functions. It is possible that the detected buffer is not related to the flow of input user data or (even more disappointingly) that one of the parent functions limits
the maximum allowed input length, in which case no overflow will occur. Users of the graphical version of IDA can use the CALL GRAPH instrument for viewing the calls tree that displays the interrelations between child and parent functions and allows you (at least, in theory) to trace the route of the data supplied by the user along the entire program (Fig. 8.5). Unfortunately, the lack of any navigation capabilities (even the simplest searching capabilities are missing) depreciates all potential benefits of CALL GRAPH. It is simply unrealistic to navigate in the diagrams that it builds. However, who can prevent hackers from developing adequate visualization tools on their own?
Until adequate GUI tools appear, hackers have to use the debugger. To begin with, fill all available fields of the user input, set a breakpoint to the call to the function that reads them (the recv function, for example), set breakpoints directly to the buffer that
receives the input data, and wait for subsequent attempts to access them. Most frequently, the data are not processed immediately after reception. On the contrary, they travel through the chain of intermediate buffers, each of which might be vulnerable to overflow errors. To keep the situation under control, it is necessary to set breakpoints at every intermediate buffer and trace their release (after release of the local buffer, the memory area belonging to it can be used by anyone else, causing false popups of the debugger, which makes the hacker nervous and produces no results except a waste of the time). But there are only four breakpoints. How is it possible to trace access to dozens of local buffers using only four breakpoints?
Here is the way out. The Softlce version for Windows 9x supports setting breakpoints to entire regions, and the number of such breakpoints is practically unlimited. Unfortunately, the Softlce version for Windows NT lacks this feature, and hackers must emulate it using intricate manipulations over the page attributes. By switching the page to the
NO_ACCESS state, it is possible to trace all attempts at accessing it (including the buffer being studied). If the buffer size is significantly smaller than the page size (which is 4 KB), then each time the debugger pops up it will be necessary to find out, which variable was accessed. If desired, this process can be fully or partially automated, because there are lots of features for Softlce supporting scripting languages.
Figure 8.5: Hierarchy of function calls in graphical representation
This is the way, in which it is necessary to search for holes. Maximum routine work,
minimum creativity. Nevertheless, it is much cooler to work with the debugger than watch The Matrix again or upgrade your computer to play Doom 3.