DDT can attach to running processes on any machine you have access to, whether they are from MPI or scalar jobs, even if they have different executables and source pathnames. Clicking the Attach to a Running Program button on the Welcome Page will show DDT’s Attach Window:
Figure 19: Attach Window
There are two ways to select the processes you want to attach to: you can either choose from a list of automatically detected MPI jobs (for supported MPI implementations) or manually select from a list of processes.
5.9.1
Automatically Detected MPI Jobs
DDT can automatically detect MPI jobs started on the local host for selected MPI implementations (and other hosts you have access to if an Attach Hosts File is configured—see sectionA.5.1 Systemfor more details).
The list of detected MPI jobs is shown on the Automatically-detected MPI jobs tab of the Attach Window. Click the header for a particular job to see more information about that job. Once you have found the job you want to attach to simply click the Attach button to attach to it.
5.9.2
Attaching To A Subset Of An MPI Job
You may want to attach only to a subset of ranks from your MPI job. You can choose this subset using the Attach to ranks box on the Automatically-detected MPI jobs tab of the Attach Window. You may change the subset later by selecting the File→ Change Attached Processes... menu item.
5.9.3
Manual Process Selection
You can manually select which processes to attach to from a list of processes using the List of all processes tab of the Attach Window. If you want to attach to a process on a remote host see sectionA.4 Connecting to remote programs (remote-exec)first.
Initially the list of processes will be blank while DDT scans the nodes, provided in your node list file, for running processes. When all the nodes have been scanned (or have timed out) the window will appear
as shown above. Use the Filter box to find the processes you want to attach to. On non-Linux platforms you will also need to select the application executable you want to attach to. Ensure that the list shows all the processes you wish to debug in your job, and no extra/unnecessary processes. You may modify the list by selecting and removing unwanted processes, or alternatively selecting the processes you wish to attach to and clicking on Attach to Selected Processes. If no processes are selected, DDT uses the whole visible list.
On Linux you may use DDT to attach to multiple processes running different executables. When you select processes with different executables the application box will change to read Multiple applications selected. DDT will create a process group for each distinct executable
With some supported MPI implementations (e.g. Open MPI) DDT will show MPI processes as children of the mpirun (or equivalent) command (see figure below). Clicking the mpirun command will auto- matically select all the MPI child processes.
Figure 20: Attaching with Open MPI
Some MPI implementations (such as MPICH 1) create forked (child) processes that are used for com- munication, but are not part of your job. To avoid displaying and attaching to these, make sure the Hide Forked Children box is ticked. DDT’s definition of a forked child is a child process that shares the par- ent’s name. Some MPI implementations create your processes as children of each other. If you cannot see all the processes in your job, try clearing this checkbox and selecting specific processes from the list.
Once you click on the Attach to Selected/Listed Processes button, DDT will use remote-exec to attach a debugger to each process you selected and will proceed to debug your application as if you had started it with DDT. When you end the debug session, DDT will detach from the processes rather than terminating them—this will allow you to attach again later if you wish.
DDT will examine the processes it attaches to and will try to discover the MPI_COMM_WORLD rank of each process. If you have attached to two MPI programs, or a non-MPI program, then you may see the following message:
Figure 21: MPI rank error
If there is no rank (for example, if you’ve attached to a non-MPI program) then you can ignore this message and use DDT as normal. If there is, then you can easily tell DDT what the correct rank for each process via the Use as MPI Rank button in the Cross-Process Comparison Window—see section 8.17 Assigning MPI Ranksfor details.
if used in attaching mode. Any input/output will continue to work as it did before DDT attached to the program (e.g. from the terminal or perhaps from a file).
5.9.4
Configuring Attaching to Remote Hosts
To attach to remote hosts in DDT, click the Choose Hosts button in the attach dialog. This will display the list of hosts to be used for attaching.
Figure 22: Choose Hosts Window
From here you can add and remove hosts, as well as unchecking hosts that you wish to temporarily exclude.
You can also import a list of hosts from a file by clicking the Import button.
The hosts list is initially populated from the attach Hosts File, which can be configured from the Options window: File→ Options (Allinea Forge → Preferences on Mac OS X) .
Each remote host is then scanned for processes, and the result displayed in the attach window. If you have trouble connected to remote hosts, please see sectionA.4 Connecting to remote programs (remote- exec).
5.9.5
Using DDT Command-Line Arguments
As an alternative to starting DDT and using the Welcome Page, DDT can instead be instructed to attach to running processes from the command-line.
To do so, you will need to specify a list of hostnames and process identifiers (PIDs). If a hostname is omitted then localhost will be assumed.
The list of hostnames and PIDs can be given on the command-line using the --attach option: mark@holly:∼$ ddt --attach=11057,node5:11352
Another command-line possibility is to specify the list of hostnames and PIDs in a file and use the -- attach-fileoption:
mark@holly:∼$ cat /home/mark/ddt/examples/hello.list node1:11057 node1:11094 node2:11352 node2:11362 node3:12357 mark@holly:∼$ ddt --attach-file=/home/mark/ddt/examples/hello.list In both cases, if just a number is specified for a hostname:PID pair, then localhost: is assumed. These command-line options work for both single- and multi-process attaching.