Experiments

For running best-effort experiments, we designed a simple setup where Linux containers running emulated PLCs are directly interconnected with each other. As TimeKeeper and S3F are not involved (and are required for support of serial communication), the emulated PLCs use the IP protocol to carry their Modbus communications. At the point a source constructs a packet, we note the “send time” and have the source delay transmitting the packet for a length of time equal to the modeled transmission delay. The time a packet is received by the target PLC is likewise noted, and the observed packet trans- mission delay was calculated by logging the receive times and send times of each packet.

(a) CDF of packet delay for best-effort experiment for different topologies

(b) CDF of packet delay for TimeKeeper controlled experiment for different topolo- gies

Figure 6.3: Comparison of CDF of per packet delays for best-effort and TimeKeeper controlled experiments

(a) Histogram of standard deviation of cycle times for undilated 63 node topology.

(b) Histogram of standard deviation of cycle times for dilated 63-node topology. Figure 6.4: Comparison of standard deviation of cycle times for dilated and undilated 63-node topologies

Figure 6.5: Comparison of maximum throughput of update messages for best-effort and controlled experiments.

Figure 6.3a shows the plot of the cumulative density function (CDF) of the observed packet delays for different sizes of full binary trees. At each step, the topology size was nearly doubled but throughout the experiment was run on the same Linux server. Each tree node represents a PLC emulation, so as we increase the size of the topology, the computational resources available to a container decreases; Figure 6.3a shows that this has a clear impact on processing time, increasing topology significantly increases the packet delay. The reason for this is that as these containers are scheduled purely by the Linux kernel, when a container is swapped out in the midst of a transmission (i.e., the send time has been recorded and the wait time before delivery has not yet completed) the time it is swapped out will increase with the number of other processes clamoring for CPU attention. The larger the topology, the longer a container will be swapped out. For the topology with 7 nodes, we measured the mean and 90 percentile delays as 25.26 ms and 32 ms (approx-

imately) respectively. On the other hand, for the topology with 63 nodes, the mean delay equaled 191.52 ms whereas the 90 percentile delay exceeded 200 ms. Whatever the true packet delay might be, we know that it should not change with increasing topology size, and in this system it does.

We repeated the same experiment under a TDF of 15 with S3F and Time- Keeper. Recall that Awlsim is a Python program that interprets Step-7 instructions, and furthermore spawns a number of other processes in support of that interpretation. On a PLC the instructions run almost on bare metal. While a TDF of 15 is admittedly arbitrary, it does reflect that there is con- siderable overhead in interpreting Step-7 instructions. Fine-tuning a TDF requires instrumenting the modeled PLC to acquire a solid understanding of its native performance, and a study of the cost of interpreting its instructions within Awlsim.

Figure 6.3b, shows the CDF of the observed packet delays in the dilated experiment. Packet delay is considerably smaller; the cumulative distribu- tion functions is largely insensitive to topology size. For the 7 node topol- ogy, the mean and 90 percentile delays were approximately 11.42 ms and 16 ms respectively whereas the corresponding values for the 63-node topology equaled 15.57 ms and 26 ms respectively. We attribute the slight increase in the packet delay values to TimeKeeper’s timing errors in starting and stopping processes and detecting and adding newly spawned processes to the experiment. It is also important to note that while the specified link delay was 4 ms, the observed mean delays were all greater than 4 ms. This is because each process inside the container is assigned a default fixed virtual time-slice of 1 ms and scheduled in a round robin fashion by TimeKeeper. This can introduce a maximum delay proportional the number of processes running in the container. In our experiments, each container ran a total of 10 processes including the PLC emulator, which can in theory introduce an

additional packet reception delay of up to 10 ms. We understand the source of this inaccuracy; to deal with it more directly will require somewhat exten- sive modifications to the Linux kernel scheduler.

For both sets of experiments, we also compared the standard deviation of each emulated PLC’s cycle execution time. Figure 6.4, shows the histogram (taken over all network nodes) of the standard deviation of cycle execution times in the controlled and best-effort executions of the 63-node topology. Each marked point on the x-axis represents a standard deviation interval starting from the specified point until the next one. The y-axis indicates the number of nodes with standard deviation of cycle execution time within the specified interval. From the figures, it is easy to see that the standard deviation of the per cycle execution time is very low in the controlled ex- periment with the 90 percentile standard deviation below 3 ms whereas it is of the order of hundreds of milliseconds for the best-effort experiment. The best-effort performance can again be attributed to the impact of a container being suspended in the midst of an execution burst, with increasing length of suspension as the number of other containers grows. The fact that the cycle execution times of PLCs in a controlled experiment remain fairly constant at each node further underlines the benefits of the tight coupling between emulation and simulation.

We also measured the maximum number of processed update messages for both classes of experiments. Figure 6.5 plots these measurements for various topology sizes. As expected in the real systems, we observed a near linear increase in the throughput values in the controlled experiment whereas the throughput fell dramatically in the best-effort experiment. The observed variance in the maximum throughput of the controlled experiment was also fairly small, as should be observed in the real system.