The performance of the event processing engine can be described in two parts. The first question raised is the event processing speed during sequential simulation. This
provides a guideline to compare the performance of SDES with other discrete event simulators of similar type. The second part is the parallel performance of SDES, which shows how the event processing power grows with the available computing power provides an insight to the scalability of the simulator.
Sequential simulation speed
The first performance parameter for a simulator is the event execution speed. This is an essential parameter, as it shows the event processing effectiveness of a simulator. It can be compared across different systems, as shown at the end of this section. The size of the main event list at any time instance has a huge impact on the event execution speed. In order to demonstrate this effect, circuits with different size are simulated in
169
GA 1 GB 1 GC 1
GA 2 GB 2 GC 2
GA 3 GB 3 GC 3
GA n GB n GC n
Figure 99 Individual clock generation circuit
An array of ring oscillator circuits is employed to explore the event execution speed. Each individual clock generation circuit is a 3-stage ring oscillator (Figure 99). All of them are running at a clock speed of 333MHz. Each oscillator generates 1 event at any discrete time instance. In this test, the number of inverter oscillators involved ranges from 1 to 1000 ring oscillators. The tests measure the overall simulation time for the simulator to process a fixed amount of events across different sizes of circuit. The amount of events simulated ranges from 1 million to 10 million. The simulation time against the number of simulated events is plotted in Figure 100.
Figure 100 Simulation time in wall clock vs. number of simulated events
The time difference for simulating the same number of events is due to the increase in search space when new events are generated, as a newly generated event needs to search and compare with all the existing events that fall in the timestamp. The last event in a 10 ring oscillator circuit requires comparing with 9 other events to check if there are any duplicated value alternations. If these duplicated events are not removed from local event list, after propagating through the communication network, the order of execution for events carrying a same timestamp can be out of order, and hence the wrong result can be produced. The total number of search and compare operation needs to be carried
in a single time step out can be represented as ⁄ , where N is the number events sharing the same timestamp. If the total number of search and compare operations for a 1 ring oscillator circuit is x, the n oscillator circuit needs to perform ⁄ operations.
The linear relationship between the overall simulation time and the total number of simulated events shows the event execution speed is fixed for a circuit. However, the execution speed varies wildly. Based on Figure 100, the event execution speed for the four different circuits are 110K events per second (eps) for 1 ring oscillator, 108K eps for 10 rings, 73K eps for 100 rings, and 18K eps for 1000 rings. This gives a spectrum
171
of speed for SDES to compare with other discrete event simulators. Other work [89], [136], [137] achieves a speed of 2M eps and 500K eps in serial mode respectively. Both techniques are pursuing the time warp technique, which tested to have a better
performance than deadlock avoidance technique in some cases. This shows that there is a 20X – 100X sequential performance gap between the SDES and other
implementations. This is a great room of performance improvement in the future. The focus of this project is to develop a simulation system which is capable of large scalability. The absolute speed is not the main focus at this stage.
Parallel simulation speed
The relationship between the input computing power and the performance speedup defines the performance of a parallel simulation. This is different from the evaluation of sequential speed.
In a parallel environment, simulators have to deal with communication between processors as well as event execution. However, it is the event execution that counts in the speedup measurement. If communication processing time dominates a simulation, the speedup in this simulation is capped due to the limited time contributed to event processing, upon which the speedup measurement is based.
The individual clock generation circuit is developed to reach the computational end of the performance spectrum. If the individual clock generation circuits are ideally distributed among LPs, there is no communication involved during a simulation.
The delay on each inverter is 1 ns, which means the clock rate in each ring oscillator is 333MHz. The simulation end time is set to 20µs, so each of the ring oscillators
generates 20,000 events in the overall simulation. As a result, by simply stacking up the number of ring oscillators, the density of events in the test circuit increases.
If the speedup that is shown by the SDES continues to increase until the number of LPs reaches the number of gates in the system, the SDES can be asserted to have a good scalability. In this test, three sets of ring oscillator arrays are employed, containing 25, 50, and 100 oscillators. They have 75, 150, and 300 gates respectively.
A key parameter needs to be determined before applying these tests: the optimal number of LPs to be employed in each virtual SpiNNaker node. The physical structure of the SpiNNaker hardware system has a two-tiered processing system, the monitor process and the slave processes. The monitor process might become a potential bottleneck of the SDES system.
To establish the value of this parameter, the 100 ring oscillator array is employed, which contains 300 gates. By fixing the number of LPs to around 300 and simulating under different numbers of virtual SpiNNaker nodes, an insight provided to this problem. In addition, this test also reaches the point where the number of LP equals to the number of gates in a CuS.
The only parameter changes during the simulation is the ratio between monitor and slave processes. The number of LPs and monitors is shown in Table 7. The first row shows the number of LPs per node, which is the only parameter changed during the simulation. The second row shows the number of nodes required to reach 300 LPs, as there is only one monitor process per node, this row also presents the number of monitors exists during a simulation. The third row shows the number of LPs which are the effective number of processes engaged in event processing. The total number of processes is the sum of the number of monitors and LPs plus an additional overseer process which controls the initialization and finalization stage of the system.
173 LPs/node 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Total Nodes / No. of Monitors 300 150 100 75 60 50 43 38 33 30 27 25 23 21 20 Total LPs 300 300 300 300 300 300 301 304 297 300 297 300 299 294 300 Total Processes 601 451 401 376 361 351 345 343 331 331 325 326 323 316 321
Table 7 Total process count during simulation
The graph (Figure 101) shows that the simulation with 5 LPs per node has the fastest performance relative to other numbers of LPs per node. In other words, a monitor to slave process ratio of 1:5 is the most effective rate for monitor process to avoid being a bottleneck. The initial improvement of performance is mainly due to the increase in event processing power while the cost of monitor is relatively fixed. However, as the number of LPs on a node increases, the bottleneck effect of monitor process starts to appear, as more and more LPs request attention from the monitor process.
Figure 101 Performance under various numbers of LPs per node
The ring oscillator array can now be simulated using five LPs per node to compare the performance speedup relationship across different circuit sizes. By applying the 34X scaling, the maximum speedup that can be achieved should be much higher than it is on the Iridis platform, as the event processing time is negligible compared to the
communication required to get the event across the system. The result graph is shown in Figure 102.
Figure 102 Scalability test with different ring oscillator array size
As the size of the ring oscillator array increases, the maximum speed up before scaling increases and saturates at a certain level. Even with more than 100 LPs simulating the 100 oscillator array; the maximum speedup is only around 4 times faster than simulating using 1 LP. This is mainly caused by the ever increase in the communication that travels between different LPs. When the communication cost is scaled down to 3% of its
original, the speedup shoots up to over 10 times compared to its original speedup. As a result, communication cost replaces the event processing cost.
However, if the communication cost is scaled down, the effect on the overall simulation time can be easily observed. When simulating a 50 and 100 ring oscillator array circuit, the average speedup achieved after scaling down the communication cost is around 60 times compared to only a single digit speedup without the communication cost scaling. The SDES is capable of achieving a high level of scalability, if the CuS (circuit under simulation) possesses good parallelism.
Apart from the advantages that parallel simulation brings, the worst case scenario must also be explored. At the other end of the spectrum where communication activity dominates a simulation, the performance is hard hit by the lack of parallelism in the
175
simulated circuit. Based on this idea, a circuit that is fully sequential is devised to examine the parallel overhead that is introduced when the simulation is parallelized.
9
Figure 103 Sequential chain of inverters
Figure 103 shows a fully sequential system which consists of 9 inverters with a delay of 1 ns on each inverter. A source which continuously generates events at 333MHz is connected to them. This circuit is simulated under ten LPs, but they are mapped in two different settings. In one simulation setup, the nine gates plus the source are mapped to 10 different nodes, each node has 1 monitor process and 1 slave process therefore only 1 LP is available for event processing. Each holds only one simulation component. In the second simulation setup, all ten components are located in a single node, but in 10 separate LPs on this node, i.e. 10 LPs sharing 1 monitor process only. This can differentiate the boundary setup cost that is involved, as the first setup uses 1 monitor process to serve 1 LP, and the second setup uses only 1 monitor process for all 10 LPs. As all the communication, which consists of P2P packets, in the first CuS requires going through 2 monitor processes, whereas the second one only travels through 1 monitor process. By removing the total communication and total event processing time required, the cost for using different types of boundary can be revealed.
tp is the overall simulation time for running in parallel. ts is the overall simulation time for running the simulation with only one LP. n is the number of LPs in the parallel simulation, and cp is the total communication time taken during the simulation.
Comm. Cost (s) Overall Time (s) Events per LPs Boundary Cost (s)
Sequential 0.000162 0.940687 100000 n/a
Proc. Bound. 1.100189 1.776259 10000 0.000058200
Chip Bound. 1.553753 3.4083 10000 0.000176048
The simulation is based on 10 different tests, and the average result is shown in Table 8. Simulation shows that node boundary is around 3 times more expensive than process boundary.