Performance Testing for the Real-time Requirement

In the previous example, only four rooms were used to demonstrate the features of the real-time fire monitoring system. To this point, the near real-real-time computing performance for the case with a single sub-model (event-detection model) has not been discussed. The theoretical performance is expected to be sufficient when analyzing the computations involved in the event-detection model for a single sensor. The computational demand is very low because there is no solver for any matrix equations in the event detection model: all the computation is comprised of

149

the base floating point operations (addition, subtraction, multiplication, and division) on scalar values with various compound logic loops (while, if, else). This results in a theoretically lightweight solution because the individual calculations associated with a single sensor are limited to the FED calculations and threshold checks resulting in approximately O(100) operations per sensor for each time increment. Thus, scaling from n sensors to 2n sensors is not a major concern from a floating-point operations standpoint.

In terms of the message-passing problem, scaling could become an issue for meeting the real-time requirement. For the four-room example shown, data collection was performed using a nominal frequency of 1.0 Hz and thus each of the four sensors would send their new data to the monitoring system for event detection approximately every 1.0 seconds. If the number of sensors grows, the number of messages sent through LCM increases for every time step. For example, in a 100-sensor model at 1.0 Hz, the system would need to handle 100 messages per second and then perform the subsequent event-detection calculations while trying to maintain the real-time feature.

The computational demand is need to perform more calculations in the same small time window.

The scaling of the system was tested by employing more sensors and timing the data collection and the completion of the event-detection calculations. This test was designed to measure the computational cost of the real-time fire monitoring system and provide insight on the performance of the system. Recall from Fig. 5-3 that the data is transferred through the system using LCM. First, the simulated sensor measurements are made in the initial component: time series data is read from the file and published to main program using a waiting function in the sensor simulator, as discussed previously. This first time point is tsensor, which is any time that the sensor records a new measurement.

150

Second, the sensor encodes the newly measured data and sends it to the main program in Fig. 5-3 using LCM: the time it is received is tmain. Since LCM is an efficient and reliable software library, the transfer of data from sensors to the main computing program is essentially immediate from a computational cost perspective. However, in the real application of this system for an actual fire scenario, this first data transfer represents the delay between a sensor measurement in the field and the arrival of the data for computation at a stable, remote location. A natural time delay would be expected for data transmission. In the computational case presented here, the delay is less than a fraction of a millisecond and thus tsensor is very nearly equal to tmain in these tests. Furthermore, comparing tsensor and tmain is essentially a measurement for how efficient LCM can encode and decode messages passed between two applications, which is something that the LCM developers have already provided.

The third and final stage of the data is in the event detection model after it has been forwarded from the main program. A data packet from the main program is encoded with the new measurements and sent to the event detection model using LCM. The message arrives in the event detection model, it is decoded, and is immediately used in the calculations discussed previously.

The new event detection model results are written to an output file one line at a time for each calculation. After the last of the three hazard checks is made in the event detection model and the output is written to a file, another time sample is taken and given the label of tend for the time at the end of the event detection calculation for the current data set. Since the sensor simulator and the event detection model were both developed using C++, it is possible to measure these timings using the same tool. In particular, the gettimeofday function in conjunction with localtime was used to record the times tsensor and tend for every data message sent by every sensor throughout the duration of the simulation. Using these two time stamps gives a measure of the computational cost

151

required to perform the event-detection calculations on every packet of new data. The format and precision of this time measurement was chosen to give the hour, minute, and second at the current time. However, custom code was used to represent the seconds position to three digits after the decimal. Thus, a sample timing measurement during the monitoring simulation would be in the form of tsensor = 16:55:08.234, where the last position contains the seconds to three decimal places.

In this format, the smallest measurable time increment was 1.0 ms. Then at some later point in time (when the event detection model is done performing its calculation on the current data), the time measurement would be taken again within that program: tend = 16:55:08.237 (for example).

The computing cost associated with each message and for every sensor was calculated as the difference between these two timing measurements: tcost = tend – tsensor (performed for every message in the system). For example, in the four-room example of the previous section, fire data was measured at a frequency of 1.0 Hz for 10 minutes resulting in 601 measurements (including the initial time of t = 0 seconds) per room for the duration of the simulation. Four rooms, each with 601 measurements, results in 2404 data messages sent to the event detection model for computational work to be performed and thus tcost was computed 2404 times for this case.

Since the goal of this analysis was to measure the performance of the system handling data in real time, the actual numerical values for the temperatures and species concentrations were not considered important. The data files used in the four-room example (i.e., four .csv files containing the time series data for each room) were copied several times to provide additional data files for the new cases of 8, 16, 32, 64, and 128 sensors. The same 1.0 Hz frequency was used with a random noise of 25% in the sensor delay times. These new cases provided a significantly higher message-passing volume for the system and the simulation time was cut back from 10 minutes down to five minutes for simplicity. Five minutes corresponded to 301 data messages per room

152

which resulted in 2408, 4816, 9632, 19264, and 38528 total messages and subsequent rounds of event detection model calculations in the simulation for the new cases of 8, 16, 32, 64, and 128 sensors, respectively.

Aggregate statistics for the various cases were computed to analyze the performance. The difference tcost was computed for every message in the system for each of the cases (for example, it was computed 4816 times for the 16-room test). The smallest measurable difference for this calculation was 1.0 ms. For the real-time requirement, a target computing cost of 0.1 seconds per measurement for the 1.0 Hz case would be reasonable and was selected as the target deadline. In other words, if it takes less than 0.1 seconds (100 ms) to perform the event-detection calculations, then the system should perform sufficiently for the real-time scenario. The concern with such a high message volume in the cases of several rooms is that the event detection model must process new data serially and thus bottlenecks could theoretically occur.

Table 5-5. Quantification of the system performance with respect to increasing number of sensors; the average computational cost, maximum observed single cost, and the standard deviation are all provided as

well as the percentage of measurements that required ≤ 1.0 ms

Sensors Messages Avg Cost [ms] Max Cost [ms] Std Dev [ms] Cost ≤ 1 ms

4 2400 0.33 3.0 0.48 99.7%

8 2408 0.29 3.0 0.46 99.8%

16 4816 0.30 3.0 0.47 99.7%

32 9632 0.28 4.0 0.46 99.7%

64 19264 0.27 6.0 0.47 99.4%

128 38528 0.24 10.0 0.45 99.6%

Thus, the aggregate statistics measured in this test serve to ensure that the system is performing efficiently for the purposes of real-time use. Table 5-5 provides the average computing cost for all the data measurements in each of the cases, as well as the maximum single cost observed in the test. The standard deviation and other characterizing features were included for completeness. For example, the percentage of messages requiring less than 1.0 ms (the smallest difference that could be measured) or equal to 1.0 ms using this timing function. Table 5-5 shows

153

that, for 32 sensors, 99.7% messages required less than or equal to 1.0 ms of computational time in the event detection model: 9608 of the 9632 messages. This is an acceptable performance for meeting the real-time requirement of the system. Note that for the four-room case, the full 10 minutes from the previous system-testing example was used (600 seconds of data at 1.0 Hz frequency); the remaining cases used only the first five minutes (301 seconds at 1.0 Hz). The results show that for the number of sensors considered, the average computational cost of the event detection model was in the range of 0.24 ms to 0.33 ms with 99% of the calculations requiring 1.0 ms or less. Figure 5-22 also shows the average cost and the maximum observed cost for each case in addition to the real-time target of 100 ms; the target deadline of 100 ms was sufficiently met in these tests. These timing results were obtained using Ubuntu 14.04 on a standard dual-core personal computing workstation with a 2.53 GHz processor.

Figure 5-22: Computational cost quantified by the average time required for the event detection model to process the data; the maximum observed cost for a single data message is also provided as the upper bound

154

The reason for the high efficiency of the calculations, as seen in the average cost, is due in large part to the robustness of LCM in handling large volumes of message-passing tasks.

Additionally, the actual calculations associated with the event detection model are very lightweight for real-time computing purposes. Specifically, there is no matrix algebra involved, there are no solvers, and file input/output (i.e., reading/writing files) is very limited.