Given the small area of the detector (200 pm diameter), most light em itted from the source fibre would not find its way to the detector. Hence any simulation of the problem would be very slow. For this reason, the programme was written so that it could be run simultaneously on a num ber of computers, of various types (PC’s, workstations etc) on one com puter network that was accessible to all the computers. This reduced considerably the time needed to perform the simulation, as it was performed on a number of computers in parallel.
This method was developed as a result of problems encountered with an earlier attempt to use M onte Carlo technique to simulate Laser Doppler blood flow m easurem ents, where thirteen different Sun workstations were used143. Unfortunately, with many com puters writing to many different data files, it became difficult to m onitor the progress of the calculation. The random nature o f the M onte Carlo method meant that each computer could be writing data to its files at anytime, making it impossible to safely add the data files from many com puters for analysis, without first stopping the processes, summing the data, then restarting the computation, which was very tedious and error prone. A brief description of the parallel im plem entation of the programme developed to overcome this problem follows.
The description of the problem to be solved was held in one small ASCII file. This contained the measurement geometry, tissue optical properties (pa, fls and g), position o f source and detector, fibre characteristics etc. The Monte Carlo simulation software read this from disk, performed the simulation, then wrote the photon history results, consisting of intensity, position, pathlength and angle with respect to the detection fibre into a 4 dimensional array of memory. After a minimum of 1000 photons were launched, the results were written to a file on a disk accessible to all computers on the network. One copy of the programme ran on each computer available, usually at reduced priority when being executed on Unix based m achines that implement scheduling priorities.
Only one results file was stored on the computer network. Each computer perform ed the M onte Carlo calculations and wrote the photon history results into this one results file. Since two computers editing the one file could easily result in file corruption, the time of the day at which any computer could update the data file was fixed. The first computer to execute the program m e wrote a file named 0 into the directory where the data was stored. The file did not need to contain any data at all, although when the computer was running under the Unix operating system, the m achine’s name and the process identification num ber (PID) were written into the file in ASCII to enable the user to determine on which machines the program m e had been
194
started. The second execution of the programme, on finding the file named 0, wrote a file named 7. A third execution of the programme wrote a file named 2 and so on, up to the 24th execution, which wrote a file named 23. The source code was written such that the program m e that wrote file 0, could only store data to disk between the times o f midnight and 00:55 am (until this time the programme continued to run and accumulate data into the local machines m em ory). The second programme to run (which wrote the file 7) could only store data between the tim es of 01:00 and 01:55, the third stored data between the times o f 02:00 and 02:55 etc. This ensured that each execution of the programme could only edit the data file at a tim e when no other programme was doing this. Obviously, this limited the num ber of separate executions o f the programme to 24, but even this could be extended if necessary, by using smaller tim e steps than one hour. Conversely, if the number of computers available was smaller (say 6) it would probably be preferable to write the data once every six hours, rather than once per day, as this would reduce the time that could elapse without the data file being updated. For safety, no programmes could edit the data file in the last 5 minutes of any hour, since tim e m ust be allowed for writing the file. If for any reason (such as hardware failure) a process was unable to write the data to disk in the 55 minutes allocated for the task, the data was retained (and continued to further accumulate) in the computer’s random access memory (RAM) until the next day, when another attempt would be made to write the data to disk.
o 7 8 15 16 23 24 31 An important feature of the
programme is the way that one data file could be updated by any com puter on the network, which consisted o f both Sun workstations and P C ’s. Initially, the photon data was stored in the random access memory
4-byte integer 12345678,, in 'little-endian' m em ory format (RAM) of each Computer in whatever format
F ig u re 8.6 Diagram showing how a 4-byte
integer (standard integer on 32 bit computers) the computer normally used. This would be would be stored in memory in both big-endian big-endian on a Sun computer, but use little-
and little-endian formats.
endian on a PC , as shown in figure 8.6.
Before writing data to disk, there was a choice of using either binary or text. Binary is much more efficient in its use of disk space, using only 8 bytes to write a double precision floating point number (the data type used for most of the data), compared to 16 bytes if text were chosen. It is also simpler to read and write binary data, since one can also know exactly where the data will be in a disk file - something that is not so easy to achieve when data is written as
12,„ 34, „ 56„ 00
4-byte integer 12345678,, in 'big-endian' m emory format 15 16 23 24 31 78, „ 5 6 ,6 34 „ 1 2 lfi
195
text. Unfortunately, since PCs and Suns store binary data in a different format, storing binary data in a way that both PCs and Suns can access it, is not without its problems. To circum vent this, data written to disk was always stored in the Sun (big-endian) format. Any PC version of the programme stored data internally in RAM in the little-endian format, but reversed the order of the bytes to convert to big-endian format before writing to disk.
Before a programme wrote data to a disk, it first ensured that the data in the data file was for the same conditions - ie geometry, tissue optical properties etc. Both the num ber of launched and detected photons were incremented along with all intensity and pathlength data from the programme.
A paper describing this parallel Monte Carlo program me was published in 1997145.