Linux’s programming environment provides support for interactive debugging. The gcc compiler can generate symbol information for debugging, and several debuggers are available. We will begin by demonstrating how to add debugging information to an executable and then take a brief tour of two popular
debugging environments for Linux.
Compiling for Debugging
In order for a debugger to analyze an executable’s behavior in a way that is useful to humans, it needs to determine the exact locations of the program’s
variables and function entry points. This requires a bit of help from the
compiler; applications must be specifically compiled for debugging, or symbolic debuggers will be useless. To compile a program with the necessary debugging support (and in particular, support for the gdb debugger), use the-ggdbflag:
$ gcc -ggdb foo.c -o foo
It is a good idea to disable optimization when debugging (that is, do not use the
-On compiler option). Although gcc and gdb allow you to debug optimized executables, the results might be a bit surprising (since optimization, by definition, changes the internals of a program).
Although programmers sometimes use the-fomit-frame-pointer compiler option in the hope of improving performance, this option is incompatible with debugging in most cases. (It causes the compiler to omit the instructions that usually keep track of an important piece of position information.) Compiling an executable for debugging will increase its size and most likely decrease its performance; executables intended for public release should not be compiled for debugging.
gdb
The GNU debugger, known as gdb, is the primary debugger for Linux. It allows you to single-step programs, inspect variables while programs are running, and analyze corefiles (memory dump files, usually named core, generated
automatically when applications crash, affectionately dubbed “core pies”). gdb is an extremely powerful tool, but its interface is likely to throw beginners for a loop.
gdb is a text-based interactive debugger. Once a program is loaded into the debugger, gdb accepts commands to direct the program’s operation. There are lots of commands, but there is also a nice online help facility. Simply typehelp
A Trivial Example
The following program is supposed to print the numbers from 0 to 9. However, it has a bug. There is an extra semicolon after thefor loop, which causes the
printfstatement to be separated from the loop. This is a fairly common error, simple to fix but often hard to locate. gdb is great for pinpointing this type of error, since it lets you see exactly what’s happening in the program.
#include <stdio.h> int main()
{
int i;
for (i = 0; i < 10; i++);
printf("Counter is now %i\n",i); return 0;
}
First, we compile the program and test it:
$ gcc -ggdb buggy.c -o buggy $ ./buggy
Counter is now 10
Yikes! That shouldn’t have happened—we’ll use gdb to figure out what’s going on. To load a program into gdb, pass the name of the program on the command line:
$ gdb buggy GNU gdb 4.18
Copyright 1998 Free Software Foundation, Inc. license notice removed
This GDB was configured as "i386-redhat-linux"...}
gdb is now ready to accept commands. We will set a breakpoint (a position at which gdb suspends the process for inspection) and then start the program:
(gdb) b main
Breakpoint 1 at 0x80483d6: file buggy.c, line 6. (gdb) r
Starting program: /home/overcode/book/test/buggy Breakpoint 1, main () at buggy.c:6
6 for (i = 0; i < 10; i++);
Thebcommand (short for breakpoint) sets a breakpoint at the specified function name. We also could have specified a line number or an actual memory address. In this case, gdb reports that breakpoint #1 has been successfully added for line 6 of the source file buggy.c. Thercommand starts the program’s execution. Since we have set a breakpoint on the functionmain, gdb immediately suspends the program and prints the current line (which happens to be thefor loop with the error). We will now use the n(next) command to single-step the program:
(gdb) n
7 printf("Counter is now %i\n",i);
The ncommand runs the program until it reaches a different line of code, so this is the expected result. After executing this line of code, the program should continue through the loop. Let’s see what happens:
(gdb) n
Counter is now 10
8 return 0;
That’s not good—the program moved on to line 8, meaning that the loop is no longer running. It is now fairly obvious that line 7 is not part of the loop. We can take a quick look at the source code with thel(list) command:
(gdb) l
3 int main()
4 {
5 int i;
6 for (i = 0; i < 10; i++);
7 printf("Counter is now %i\n",i);
8 return 0;
Hopefully, at this point the programmer would notice the extra semicolon and fix the problem. (One would hope that the programmer would have found it before the gdb session, but we all make dumb mistakes.) gdb doesn’t eliminate the need to study the source code for errors—it just helps you focus on the right areas.
Accessing Data
gdb can report the value of any variable that is accessible from the current scope in the program. It can also modify variables while the program is running. To view a variable, use the p(codeprint) command. p foo would report the current value of foo(if foois visible from the current location in the program). There is also aprintfcommand, which behaves much like its C namesake. To modify a variable, use theset var varname=value command.
Programmers frequently need to track variables as they change throughout the program. With gdb, you can define a list of variables to display each time the program is suspended. The displaycommand adds variables to this list, and theundisplay command removes them.
gdb’s watchpoints are useful for tracing variable corruption. A watchpoint is a hardware trap placed on a memory location. If that memory location is read from or written to, gdb will catch the access and pause the program for inspection. Since watchpoints are independent of the semantics of a particular programming language, they can be used to trace memory corruption from misplaced pointers. There are three types of watchpoints: write-only,read-only, and access. Write-only watchpoints detect modifications but not reads, read-only watchpoints detect reads but not modifications, and access watchpoints detect any type of access to the given memory address. Thewatch,rwatch, and
awatchcommands correspond to these types of watchpoints. These three commands take a symbol name as an argument. Use the enableand disable
commands to toggle watchpoints. info breakpointsprints a list of all breakpoints and watchpoints.
Viewing the Stack
It is often useful to examine the call stack. Programs often crash because of invalid data passed to the C library (notably the free function), and a normal
gdb crash report will list the name and memory address only of the function where the crash actually occurred. This is essentially useless in a typical program that makes hundreds of calls to these functions; the bewildered
programmer would have no idea where the erroneous library call took place. For instance, the following is the late-night programmer’s worst nightmare (other than a copy of eggdropfound running in an unknown account):
Program received signal SIGSEGV, Segmentation fault. 0x401371eb in free () from /lib/libc.so.6
This message indicates a crash in the C library itself, resulting from an invalid call tofree. This information is almost useless to us, since most nontrivial C programs make hundreds of calls tofree. Since the segmentation fault occurred in a function outside of our program (and, more importantly, in one that does not contain debugging information), gdb cannot simply tell us the line number of the crash location.
gdb solves this problem with itsbacktrace command. When a program crashes under gdb, backtrace will display the names of all functions that are currently active on the stack. In this particular program,backtrace provides us with the following information:
(gdb) backtrace
#0 0x401371eb in free () from /lib/libc.so.6
#1 0x804b85e in ParseSurf (f=0x8112568, buf=0xbfffd6f0 "SURF 0x10") at ac3dfile.c:252
#2 0x804c71f in ParseObject (scene=0x8112620, f=0x8112568, buf=0xbfffe34c "OBJECT poly") at ac3dfile.c:545
#3 0x804c7c3 in ParseObject (scene=0x8112620, f=0x8112568, buf=0xbffff380 "OBJECT world") at ac3dfile.c:559
#4 0x804cb74 in AC3D_LoadSceneFile (filename=0xbffff957 "crash.ac") at ac3dfile.c:829
#5 0x804d7a2 in main (argc=3, argv=0xbffff7e4) at ac3dembed.c:15
Aha! The invalid free call occurred while the program was executing line 252 of
ac3dfile.c, in the function ParseSurf. We now know exactly where the
erroneous call to free was made, and we can use standard debugging techniques to figure out why this particular line caused a crash. (A hint, in case you find
yourself in this situation: crashes in free are usually due to heap corruption, which can result when a program overruns allocated memory buffers.)
backtrace’s output will be useless if the program has corrupted the stack in some way. If this happens, you’re in for a challenge, but at least you’ll know to look for memory accesses that might cause stack corruption. If you’re feeling particularly adventurous, you can try setting a watchpoint on an address in the stack, but it could easily be triggered by legitimate accesses as well as bugs.
Remote Debugging
Linux is a network-enabled multiuser operating system, and it makes remote debugging extremely easy. Remote debugging (that is, debugging from a different console than the one on which the program is running) is useful when you’re dealing with applications that take over the screen or keyboard (as is frequently the case with games). Anyone who has had to debug a full-screen OpenGL game can attest to the importance of remote debugging.
gdb supports two types of remote debugging. It provides support for debugging over a serial connection, which is useful for kernel debugging but probably overkill for game development. Serial debugging is important when one cannot count on the stability of the operating system itself (and therefore the stability of the debugger). gdb also has the ability to attach to programs that are already running. You start the buggy application (compiled for debugging) normally and launch gdb via a remote login from a second computer. You then attach gdb to the buggy application. Now you can use the debugger without fear of losing control of the console. Note that gdb is running on the same computer as the application; it is just controlled from a remote terminal.
To attach gdb to a running program, first use thefile command with the name of the executable you want to debug:
(gdb) file foo
Reading symbols from foo...done.
gdb is now ready to attach to a running copy of foo. Use the attachcommand with the process ID of the running application:
(gdb) attach 3691
Attaching to program: /home/overcode/test/foo, Pid 3691 Reading symbols from /usr/X11R6/lib/libX11.so.6...done. Reading symbols from /lib/libc.so.6...done.
Reading symbols from /lib/ld-linux.so.2...done. 0x4016754e in __select () from /lib/libc.so.6
The debugger has suspended foo, and you can now use the normal gdb debugging commands, just as if you had startedfoo under gdb directly.
Debugging Multithreaded Applications
Games frequently use multiple threads of execution to smoothly coordinate the various parts of the game engine. Unfortunately, multithreading has always been a thorn in the side of source-level debuggers. gdb can debug multithreaded applications locally,but it cannot attach to more than one thread of an application that is already running. This is because threads under Linux are implemented as separate processes that share an address space, and each thread has a separate process ID. gdb needs to catch threads as they are created in order to debug them.
When gdb suspends a multithreaded application, it suspends all of its threads at once. This allows you to switch between threads and examine the program without the fear that something will change in the background. Keep in mind, however, that single-stepping a multithreaded application may result in more than one line of code being executed in some threads; gdb only directly controls the execution of one of the threads.
Working with threads in gdb is not particularly difficult. The info threads
command prints a list of threads owned by the application, and the thread id
command switches between threads. gdb assigns its own thread IDs to a program’s threads; these are listed in the leftmost column of theinfo threads
display. To apply a gdb command to one or more threads, use thread apply ids, whereids is a list of thread IDs or “all.”
Unfortunately, multithreading causes problems with watchpoints. gdb can reliably detect memory changes only within the current thread; it might fail to detect a change caused by another thread. Watchpoints can still be useful in multithreaded applications, but you will have to determine which thread is causing the change on your own.
Screen shot of ddd
ddd
Many people find the gdb interface hard to live with, and so several front ends have been created. Perhaps the best-known front end is the Data Display Debugger, or ddd. This program adds a nice interface to gdb, perhaps limiting its usefulness to hardcore gdb fans but certainly making life considerably easier for beginners.
ddd requires only a minimal introduction, because it closely mirrors the functionality provided by gdb (and with good reason; itis gdb, inside a GUI wrapper). To begin a debugging session with ddd, chooseOpen Program from the File menu. You may then set breakpoints and control execution with ddd’s toolbar and menus. ddd allows you to attach to running programs after the corresponding executables have been opened. If you need a piece of functionality provided by gdb but not ddd, you can send commands directly to gdb with the console at the bottom of the screen.
Bug Tracking
A very important but often overlooked aspect of debugging is keeping track of the information pertaining to identified bugs. A game development team might easily receive hundreds of bug reports during the course of a game’s development and beta test, and it is essential to organize these reports so that the developers can easily verify and resolve the bugs. Bug-tracking software is every bit as important to a serious game development operation as the debugger itself. The Mozilla project’s Bugzilla has emerged as one of the best and most widely used bug-tracking systems. Bugzilla is a Web-based system written in Perl and designed for use with the popular Apache Web server and MySQL database server. With it, users can report bugs, check to see if a reported bug has been resolved, and browse through other bugs that have been reported. Bugzilla is covered under the Mozilla Public License, and it can be freely used and modified. It is relatively simple to install if MySQL and Apache are already configured. To see Bugzilla in action, visit http://bugzilla.mozilla.org.