TAO OpenDDS4 is an open-source C++ implementation of the OMG’s v1.0 Data Distribu- tion Service (DDS) specification. DDS, as part of the Event Driven Architectures (EDAs), is emerging as new technology to design flexible applications by means of message-driven processing [92, 93]. Its recent use in mission-critical scenarios, e.g., the Air Traffic Con- trol domain (Coflight5 project), makes it useful to perform an in-depth evaluation of DDS
logging capabilities. The testbed shown in Figure 3.7 has been deployed to perform the campaign. A test application, coming with the DDS software distribution, provides the workload to exercise the middleware. The application consists of two processes. The pub- lisher (PUB) process, deployed on node 1, sends DDS messages bounded to a topic. The
4
http://download.ociweb.com/OpenDDS/
5
Table 3.4: TAO Open DDS: breakup of the experiments by fault operator and coverage of the logging mechanism.
Fault # locations Fault # locations MFC 762 MLAC 144 MVIV 140 MLOC 87 MVAV 139 MLPA 1,105 MVAE 317 WVAV 125 MIA 345 WPFV 505 MIFS 304 WAEP 40 MIEB 110 Total experiments 4,123 Number of failures 1,023 Logged failures (coverage), PUB 346 (33.8%) Logged failures (coverage), SUB 297 (29.0%) Logged failures (coverage), joint 492 (48.1%)
subscriber (SUB) process (node 2) subscribes the topic of the publisher and then receives the messages transmitted via the DDS middleware. Furthermore, node 2 hosts the Test Manager program: for each experiment it (1) initializes the test application (2) starts the publisher process, i.e., the workload generator, and (3) once the workload terminates, stops the testbed components and collects experiments data. Table 3.4 reports the 4,123 experi- ments composing the campaign broken down by fault operator: faults have been injected in the library encapsulating the code of the DDS middleware.
3.4.1 Coverage and verbosity
Injected faults caused 1,023 failures out of total 4,123 experiments (356 halt, 597 silent, and 70 content). The coverage of Open DDS is preliminary assessed at the publisher and subscriber side, separately, because the DDS infrastructure does not provide a native support to centralize the event log. Publisher and subscriber are usually deployed on different nodes and it might not be possible to access the logs of both the components of the application.
(a) Publisher. (b) Subscriber.
(c) Joint (i.e., Publisher or Subscriber).
Figure 3.8: TAO Open DDS: coverage by failure type.
The logging mechanisms reports 346 out of 1,023 failures at the publisher side: coverage is around 33.8%. Figure 3.8a shows how the coverage varies with respect to the failure types. Similarly to Apache, many halt failures, i.e., 73% do not leave any trace in the log. Most of them are caused by (i) the DataWriterImpl and PublisherImpl DDS modules (16.4%), due to the bad manipulation of DDS messages during the sending phase, and (ii) Service_Participant DDS module (8.8%). Silent failures are logged in a relevant number of cases (37%). The DDS library is able to log silent failures occurring (i) in the DDS lower transport layer (12.5%), and (ii) in the DataWriterImpl (10%), which are mainly due to the
bad manipulation of the topic of the DDS message. Unlogged silent failures (63%) mainly occur within (i) the lower DDS transport layer (18%), because of the bad manipulation of the send buffer, and (ii) the Service_Participant DDS module (13%). Content failures are mostly unlogged (61%). Corrupted messages are delivered to the subscriber side without any notification in the log of the DDS-based application.
Coverage is around 29% at the subscriber side. Although the percentage is similar to the publisher, the overall logging behavior is different, as described in the following. In this case (as shown in Figure 3.8b), the coverage of halt failures, i.e., 40%, is higher than the publisher: subscriber is able to log failures caused by a bad QoS setup within the Service_Participant DDS module (7%), and a significant percentage of halt failures (18%) that occurred at the publisher side, thus acting as an external failure detector. Unlogged halt failures (60%) are mainly due to problems occurring in the Service_Participant module (12%), which still remains a significant source of unlogged halt failures. A significant percentage (77%) of silent failures is unlogged. Unlogged silent failures mainly occur (i) in the DataReaderImpl module (13.5%) due to problems affecting the topic-subscription phase, and (ii) algorithmic errors during the message delivery occurring in the DDS lower transport layer (11%). Again, content failures are mostly unlogged (80%). A corrupted messaged is delivered to the subscriber due to problems occurring in the DDS transport layer.
Furthermore, the coverage of the overall DDS (i.e., those failures logged by either the publisher or the subscriber), is around 48.1% (Figure 3.8c), thus higher than the publisher and subscriber: the logging mechanism might benefit of a log centralization support, in case of distributed applications.
Table 3.5: TAO Open DDS: verbosity of the logging mechanism.
failure logged number of entries in the log type failures average ±std-dev; (min-MAX)
publisher halt (96) 5 ±3; (1-17) silent (223) 7 ±24; (1-139) content (27) 7 ±3; (6-16) subscriber halt (145) 2,949 ±34,732; (1-409,492) silent (138) 2 ±1; (1-4) content (14) 7 ±2.5; (6-12) average 496
Table 3.5 reports the average number of entries in the log notifying the occurrence of a failure in case of TAO Open DDS. In the average (last row of Table 3.5), the logging mechanism reports each failure with 496 entries in the log. The PUB process is less verbose than the subscriber. The worst case is represented by halt failures logged at the subscriber side: in the average, this type of failure cause 2,949 entries in the traditional log.
3.4.2 Recall and precision
Recall and precision analysis has been conducted for TAO Open DDS at the publisher and subscriber sides: results, reported in Figure 3.9a and 3.9b, highlight a similar trend at both the sides of the DDS application, thus the only publisher is discussed in the following. It can be observed that the maximum achievable recall is 0.34, obtained for the bytes and lines indexes when the classification parameter K is set at the minimum value (Figure 3.9a); however, precision is only around 0.58. The highest precision, i.e., 0.96, is observed for the words index when K = 6, but the recall parameter is just 0.07. As observed for the Apache Web Server, achieving high recall conflicts with a good precision rate (and viceversa).
(a) Publisher. (b) Subscriber
Figure 3.9: TAO Open DDS: recall and precision.