A Fully Standards-Based Approach to Logging High-Throughput
Distributed Real-Time Data
Leveraging the DDS and SQL Standards
Mark A. Hamilton And Edwin de Jong Real-Time Innovations 3975 Freedom Circle Santa Clara, CA 95054 (408) 200-4700 [email protected] [email protected]
Abstract
The ability to capture run-time activity for future analysis and playback is vital to being able to debug, integrate, test, and support complex distributed systems. Unfortunately, implementing a logging capability is often very difficult and expensive because data must be captured from disparate sources, likely consists of many different native data structures, and often has a higher aggregate throughput (burst or steady-state) than can be supported by traditional commercial databases or even hard disks in some cases. As a result, distributed application developers traditionally have had to implement their own custom logging capabilities and tools to go with them.
By integrating widely-used standards and high-performance database technology, a fully standards-based and off-the-shelf approach to logging high-throughput distributed real-time data is now possible. This solution uses the Object Management Group (OMG) Data Distribution Service (DDS) standard to distribute real-time data to one or more instances of a Structured Query Language (SQL) database that provides persistent storage. When used with a memory-optimized SQL database, optionally as a cache to a traditional disk-centric database, throughput rates can be supported and sustained that are an order of magnitude higher than most legacy databases support.
A significant benefit of this standards-based logging approach is that applications and tools already using the DDS or SQL standards can take advantage of this solution with little or no change.
An off-the-shelf implementation of this logging solution is now provided by RTI. RTI Real-Time Connect enables the development of systems that can: efficiently bridge real-time DDS traffic to/from SQL and other enterprise applications, route high-performance DDS traffic between multiple DDS domains, and capture/log DDS message flows persistently. In conjunction with its logging capability, the RTI Real-Time
In this paper, we will focus on applying the capabilities provided by RTI Real-Time Connect in the design of a generic DDS message logging subsystem. In addition, we’ll also address performance issues related to logging with respect to sustainable rates at which real-time messages or data produced by RTI’s DDS solution, RTI Data Distribution Service, can be captured in Oracle TimesTen In-Memory Database and Oracle 10g. We’ll also touch on how TimesTen Cache Connect to Oracle can be employed to handle burst message rates during logging activity.
Design of a Generic DDS Message Logging Subsystem
Being able to persistently log specified messaging activity from various systems and subsystems within a complex distributed system provides a key role in being able to monitor, maintain, and debug system behavior. Once captured, these message logs can be further utilized to support post processing, as well as message replay for further system testing and maintenance within a controlled environment.
The DDS standard is particularly well-suited to distributing real-time data for logging as well as for general distributed application development and system integration. The DDS standard has been widely adopted for real-time systems, including in industrial automation, aerospace, and defense applications. DDS specifies an API designed for enabling real-time data distribution. It uses a publish-subscribe communication model and supports both messaging and data-object centric data models. DDS offers several enhanced capabilities with respect to content-based filtering and transformation, per dataflow connectivity monitoring, redundancy, replication, delivery effort and ordering, as well as spontaneous discovery. Furthermore, DDS offers new capabilities with respect to data-object lifecycle management, best-effort and predictable delivery, delivery ordering, resource management, and status notifications.
System Requirements
Our design objective is a generic DDS Message Logging subsystem. Specifically, we will design a facility that supports the ability to:
- log messages from one or more DDS domains, - log messages persistently,
- introduce minimal change and negligible performance impact on existing DDS enabled systems, - provide high-performance logging on low-cost off-the-shelf hardware,
- provide support for runtime logging configuration, - discover DDS message formats dynamically at runtime,
- provide individual DDS message stream logging configuration/control, - provide bounded and unlimited message logging duration.
Figure 1 illustrates the fundamental components of a generalized DDS message logging subsystem. The subsystem must interface with DDS applications as well as one or more database management systems. The design will leverage a relational storage model. One of the most compelling reasons for leveraging a relational database is its self describing capability and standards based compliance. In addition, storing information within a relational model opens a vast array of post-processing tools as well.
Runtime Configuration
The Runtime Configuration Engine controls which DDS messages will be logged to the database. In addition it provides various Quality-of-Service (QoS) settings :
- Whether to sub-sample (based on time) the message stream, or simply capture all samples, - the depth of the amount of logging to capture for each message stream,
- From which DDS network (i.e. domain) to log a particular DDS message stream from, - The ability to re-configure during runtime. This includes adding a new DDS message to log,
indicating which DDS message stream to stop logging, or altering the way in which a current DDS message stream is being logged.
Automatic Discovery and Storage
Once configured via the Runtime Configuration Engine, the logging subsystem must automatically discover both DDS topics (i.e. unique messages) and their associated message formats (i.e. IDL types). If the DDS message is keyed, then all instances must also be captured. For each message stream that must be logged, as indicated by the Runtime Configuration Engine, the ‘Message Discovery & Capture’ module simply creates a DDS subscriber for the corresponding topic and data type. The Runtime Configuration Engine controls the actual DDS QoS settings of the subscriber. By using DDS subscribers, the design will introduce minimal performance impact on the existing systems1. Once the DDS messages have been discovered and are being captured, they will be stored in a relational database. The ‘Relational Database
Figure 1. Generic DDS Message Logging Subsystem
Relational Database
Message Discovery & Capture
DDS Application DDS Application DDS Application
Relational Database Interface
Runtime Configuration
Engine
Interface’ module is responsible for inserting each of the DDS message samples into their respective relational tables within the database. Thus if the logging subsystem is configured to capture 50 DDS message types (i.e. topics), then 50 relational database tables (within one database) will be created to persistently store each distinct DDS message stream.
Applying RTI Real-Time Connect
Next we discuss how RTI Real-Time Connect implements the design of the generic DDS logging subsystem as laid out in the previous section. RTI Real-Time Connect seamlessly integrates DDS applications with enterprise applications by using a relational database as the interface. The database also serves as a buffer giving enterprise applications direct access to real-time data in a way that doesn’t negatively impact the performance of real-time applications. Furthermore, RTI Real-Time Connect is designed to integrate existing DDS enabled real-time applications with enterprise based systems with minimal modification to either working application. Figure 2 illustrates the RTI Real-Time Connect architecture.
RTI Real-Time Connect operationally consists of a configurable number of DDS publishers and subscribers, which act as a bridge between the DDS networks and the relational database. It further enables automatic logging of DDS messages by mapping the DDS Topic to a table in the database and storing an
Relational
Database
RTI Real-Time
Connect
DDS Application DDS Application DDS ApplicationFigure 2. RTI Real-Time Connect Architecture.
DDS
instance of a DDS Topic as a row in that table. To be complete, RTI Real-Time Connect can automatically manage the publication of database table changes out to a specified DDS network, but that capability won’t be utilized within the logging subsystem design.
RTI Real-Time Connect Salient Features
In order to better understand how RTI Real-Time Connect can be used as the basis of our logging design, let’s briefly list key features that we’ll build upon:
• Automatic discovery of DDS messages being published, • Determines DDS message format automatically via typecodes, • Automatic storage of received DDS messages within DBMS tables, • IDL/SQL Data Mapping,
• Runtime configuration via meta-tables and DDS QoS, • Message History support – both bounded and unbounded. Automatic discovery and mapping
A key RTI Real-Time Connect feature is that once configured and connected to a DDS network, it will automatically discover the DDS messages (and their type format) being published. Once discovered, DDS message format information is automatically mapped to SQL table schema, and a corresponding database table is created in order to allow the logging of the incoming DDS messages. So, without any user interaction, RTI Real-Time Connect can create, and populate relational tables that will log DDS messages. Each discovered DDS topic will result in a relational database table. Let’s explore how RTI Real-Time Connect handles each of these functional steps.
Automatic DDS Topic and Type Discovery
RTI Real-Time Connect can be configured to listen on one or more DDS domains for DDS message traffic and auto-discover the DDS topic and its data type. RTI Data Distribution Service provides built-in topics that can be used to discover remote DDS entities. By creating a subscriber to the built-in DDS topic “DCPSPublication”, RTI Real-Time Connect can automatically discover DDS topics. Once the topics have been discovered, their format can be determined by parsing the typecode information. Typecodes provide a standards-based mechanism to represent arbitrarily complex IDL types at run time. They are used to send information about the application defined data format on the wire, thus enabling applications to publish or subscribe to topics of arbitrary types. Since typecode information is automatically propagated during the discovery phase of DDS applications, RTI Real-Time Connect is able to discover and store all of the available typecodes associated with existing DDS Topics.
IDL/SQL Data Mapping
Once the DDS topic and message format (or type) is known, the relational table can be created in the database (if it doesn’t already exist). In order to create the table, the table schema must be decided upon. RTI Real-Time Connect maps the discovered IDL type information to table schema. Fields in IDL types are mapped to columns of a table, and the key fields, if any, form the primary key of the table. Each row in the table represents a different value of the IDL type. Unless RTI Real-Time Connect is configured to keep history data, which will be discussed later, the last value of a DDS instance is stored in the table. So a non-keyed topic will be stored in a single row whereas for non-keyed topics, there will be as many rows as there are topic instances. The table below shows how basic IDL types are mapped to SQL data types. Simple IDL structures containing only basic or primitive types directly map to SQL schemas with fields in the structure becoming columns in the table. IDL structures that contain more complex fields, fields that are structures, unions, or sequences and arrays of types other than “octet”, “char” or “wchar” are mapped to SQL tables by flattening the embedded structures so that their fields are all at the top (and only) level.
string<n> VARCHAR(n)
Boolean TINYINT
Short SMALLINT
Int INTEGER
Long INTEGER
long long BIGINT
Float REAL
Double DOUBLE
Octet BINARY(1)/RAW
octet[n] BINARY(n)
sequence<octet,n> VARBINARY(n)
RTI Real-Time Connect Runtime configuration – Meta-tables and DDS QoS policies
Upon startup, RTI Real-Time Connect creates “rtidds_subscriptions” and “rtidds_publications” meta-tables in the database. These two relational tables provide the application or user the ability to configure the RTI Real-Time Connect runtime behavior. For the sake of our logging subsystem use case, we’ll only focus on the rtidds_subscriptions table. The rtidds_subscriptions table is created empty. The table schema associated with this table is as follows:Table RTIDDS_SUBSCRIPTIONS:
*DOMAIN_ID INTEGER NOT NULL
*TOPIC_NAME VARCHAR (200) NOT NULL
TABLE_OWNER VARCHAR (128) NOT NULL
TABLE_NAME VARCHAR (128) NOT NULL
TYPE_NAME VARCHAR (200) TABLE_HISTORY_DEPTH INTEGER PROCESS_BATCH INTEGER PROCESS_PERIOD.SEC INTEGER PROCESS_PERIOD.NSEC INTEGER COMMIT_TYPE VARCHAR (17) CACHE_MAXIMUM_SIZE INTEGER CACHE_INITIAL_SIZE INTEGER IDL_MEMBER_PREFIX_MAX_LENGTH INTEGER IDL_MEMBER_SUFFIX_MAX_LENGTH INTEGER SUB.PRESENT.ACCESS_SCOPE VARCHAR (25) SUB.PRESENT.ORDERED_ACCESS TINYINT DR.TIME_FILTER.MIN_SEP.SEC INTEGER DR.TIME_FILTER.MIN_SEP.NSEC INTEGER DR.DURABILITY.KIND VARCHAR (30) DR.RELIABILITY.KIND VARCHAR (27) DR.DESTINATION_ORDER.KIND VARCHAR (43) DR.LIVELINESS.LEASE_DUR.SEC INTEGER DR.LIVELINESS.LEASE_DUR.NSEC INTEGER DR.DEADLINE.PERIOD.SEC INTEGER DR.DEADLINE.PERIOD.NSEC INTEGER DR.HISTORY.KIND VARCHAR (21) DR.HISTORY.DEPTH INTEGER DR.OWNERSHIP.KIND VARCHAR (23) DR.RES_LIMITS.MAX_SAMPLES INTEGER DR.RES_LIMITS.MAX_INSTANCES INTEGER DR.MULTICAST.RECEIVE_ADDRESS VARCHAR (15)
(primary key columns are indicated with *)
Each row in the rtidds_subscriptions table represents a DDS topic in a domain to be logged. The fields of the table expose many of the QoS attributes defined by the DDS standard thus providing the user control over the DDS quality of service when capturing real-time data from the DDS network. More specifically, when a row (corresponding to a specific DDS topic) is inserted into the rtidds_subscriptions meta-table, the RTI Real-Time Connect Daemon creates a DDS DataReader (and Subscriber) according to the provided QoS parameters – thus controlling how each unique DDS message is ultimately subscribed to, and logged. As additional records are inserted into the meta table, additional DataReaders are instantiated. By inserting rows, we expand the logging systems ability to capture additional DDS message types. It’s not required to specify every column when a row is inserted into the table. In fact, if a row’s column is not specified, then a default value will be used.
Generally by specifying column-values in the rtidds_subscriptions table the application can configure: • Which DDS message or data streams to log;
• Whether a message stream is captured completely or sub-sampled by time-based filter; • How many samples are kept, either all samples (unbounded) or a fixed-length history;
• Whether samples are committed to the database separately (for better response time) or in batches (for higher throughput);
• Whether messages are received reliably or best-effort for minimizing the effect on DDS publishers.
Once a message stream is being logged, it is possible to reconfigure the logging QoS or to cancel logging the stream by updating or deleting the corresponding row in the rtidds_subscriptions table.
For example, to configure RTI Real-Time Connect to log four unique DDS topics (i.e. message streams), the following four SQL statements would be executed on the rtidds_subscriptions table:
insert into rtidds_subscriptions
values (7,‘AirCraftPos’,'tableowner','AircraftPositionTable','PlanePosType',-1);
insert into rtidds_subscriptions (DOMAIN_ID, TOPIC_NAME, TABLE_OWNER, TABLE_NAME, TYPE_NAME, TABLE_HISTORY_DEPTH, DR.TIME_FILTER.MIN_SEP.SEC)
values (25,‘FuseTmp’,'tableowner ','FuseTempTable', 'FuseTempType', 20,);
insert into rtidds_subscriptions (DOMAIN_ID, TOPIC_NAME, TABLE_OWNER, TABLE_NAME, TYPE_NAME, TABLE_HISTORY_DEPTH, DR.TIME_FILTER.MIN_SEP.NSEC)
values (43,’Humidity’,'tableowner','HumidityTable','HumidType',5, 300000000); insert into rtidds_subscriptions
values (43,’Pressure’,'tableowner ','PressureTable','PressureType',0);
As a result, four DDS topics will be logged, each with differing QoS and table history depths. Let’s examine each message to be logged:
Aircraft Position Message - the DDS topic ‘AirCraftPos’ of IDL type ‘AircraftPosType’ will be captured from DDS Domain 7 and stored in a table named ‘AircraftPositionTable’. The history column is set to ‘-1’ indicating that all samples of the topic will be captured. If the DDS topic has key fields, all instances will be logged in separate rows.
Humidity Message - the DDS topic ‘Humidity’ of type ‘HumidType’ will be captured from DDS Domain 43 and stored in a table named ‘HumidityTable’. The history column is set such that the most recent 5 samples of the topic are kept. The time based filter QoS policy is set such that messages will be logged once per 300ms.
Pressure Message - the DDS topic ‘Pressure’ of type ‘PressureType’ will be captured in DDS Domain 43 and stored in a table named ‘PressureTable’. The history column is set to ‘0’, which indicates that only the most recent sample of the topic is stored.
Runtime re-configuration of the Logging Subsystem:
Inserting rows into the rtidds_subscriptions meta-table provides the ability to start logging DDS messages in accordance to specified QoS. At some point, it may be desirable to discontinue the capture of a specific DDS message stream. In this case, the row representing that specific message type can simply be deleted from the table. At that point, RTI Real-Time Connect will remove the DDS DataReader, and the message will no longer be logged. The table containing all previously recorded messages continues to be available, for instance for data analysis or simulation/replay.
Secondly, if for operational reasons the QoS of the DDS DataReader instantiated by the RTI Real-Time Connect daemon must be altered, one can simply perform an ‘update’ on the appropriate row within the meta-table, and the existing DataReader will be removed, and a new DataReader (with modified QoS) will be instantiated. This capability can be used, for instance, to dynamically adjust the sample-rate or history depth.
For increased flexibility, the rtidds_subscriptions meta-table can be configured and managed either by a dedicated user application or graphical enterprise tools to configure a DDS message logging subsystem with the ability to alter the systems logging functionality during runtime.
Performance Considerations: In-memory versus disk-based database
Depending on the volume and frequency of DDS messages, RTI Real-Time Connect can be configured to utilize either in-memory, or disk-based relational databases. More specifically, RTI Real Time Connect supports Oracle TimesTen In-Memory Database which, being memory-optimized, delivers exceptionally high performance, enabling it to easily support the high data throughput of real-time applications. Real Time Connect can also be configured to use Oracle Database 10g, for storing large volumes of data. High Performance Logging
When high performance logging is required, the TimesTen in-memory database provides the highest transaction rate on commodity hardware because insert and update operations are executed using memory-optimized data structures. Figure 3 shows the throughput that can be achieved when logging a single DDS message stream into a TimesTen data store. Note that for a single DDS message size of 64 bytes, RTI Real-Time Connect configured with TimesTen can sustain 28,000 messages per second. If RTI Real-Time Connect is configured with Oracle 10g (on commodity hardware), we would achieve a sustainable message capture rate of roughly 8,000 messages per second.
Message or data rate versus payload 0 5,000 10,000 15,000 20,000 25,000 30,000 16 64 256 1,024 4,096
Payload size (Bytes)
Oracle Timesten (sustainable rate) Oracle 10g (sustainable rate) TimesTen Cache Connect (burst rate)
Figure 3. RTI Real-Time Connect & Oracle TimesTen Performance (Transactions/Sec).
Due to RTI Real-Time Connects scalable design, performance can scale when running on multi-processor or multi-core systems. Figure 4 illustrates RTI Real-Time Connects scalable architecture when logging four DDS message types on a two-processor/dual core system. The results are due to the fact that RTI Real-Time Connect employs a multi-threaded architecture coupled with TimesTen extremely high-performance.
Aggregate message or data rate versus payload
0 10,000 20,000 30,000 40,000 50,000 60,000 70,000 80,000 90,000 16 64 256 1,024 4,096 16,384
Payload size (Bytes)
1 topic 2 topics 3 topics 4 topics
It’s important to note that there are architectural decisions that must be addressed when configuring RTI Real-Time Connect with TimesTen. If your logging system will be running on a 32 bit operating system, you’ll be limited to a TimesTen database of roughly 2Gbytes due to the fact that the entire TimesTen database must reside in RAM. In order to support ‘endless’ logging functionality, the TimesTen tables will need to be persisted to disk, and resized in order to continue to log incoming message samples.
This can be achieved by deploying both TimesTen and Oracle 10g with the addition of Cache-Connect to Oracle which allows the seamless flushing of TimesTen tables to Oracle 10g. When TimesTen tables are associated with a Cache-Connect Cache Group, once configured - the Cache Connect daemon will seamlessly flush the designated TimesTen tables to Oracle 10g, and then automatically resize the TimesTen tables in order to allow them to continue to grow with logging. The criteria utilized to determine when to flush the tables, and how the resizing operation occurs is all configurable for each Cache Group managed by the Cache-Connect to Oracle daemon.
By leveraging Cache-Connect to Oracle, we can achieve and support ‘endless’ persistent DDS message logging, with a reasonable amount of system RAM (on a 32 bit operating system). From a performance perspective, it’s important to realize that the TimesTen in-memory database serves as a buffer isolating the enterprise infrastructure from potentially overwhelming high-frequency real-time messages. Even though RTI Real-Time Connect with TimesTen can support greater than 80,000 transactions when capturing multiple DDS message streams, this throughput rate is not perpetually sustainable. Oracle TimesTen keeps transactions in an in-memory log buffer before TimesTen Cache Connect propagates these to Oracle 10g. The results for TimesTen Cache Connect shown in Figure 3 were generated using a transaction log buffer size large enough to store all transactions for the duration of a burst. Should the burst length exceed this setting, the data rate would drop to the level as shown by the previous benchmark. By employing TimesTen as an in-memory logging buffer, RTI Real-Time Connect can successfully absorb high message rate spikes without loosing DDS messages.
Alternatively, a 64-bit operating system can be deployed with sufficient RAM necessary to support the total volume of messages to be logged. The TimesTen data store can be configured as Persistent, which enables DDS message capture to disk without the use of Oracle 10g.
Lastly, as mentioned earlier, if the performance of TimesTen isn’t required, then RTI Real-Time Connect can be configured to log directly to Oracle 10g thus allowing virtually unbounded storage of DDS message traffic. In order to increase message throughput when logging to disk, message aggregation or sub-sampling can be applied if each individual DDS message isn’t required to be uniquely logged.
Conclusion
RTI Real-Time Connect is fully standards-based and is a solution that facilitates the design of a DDS logging capability. From our previously stated design objectives, we have been able to see that RTI Real-Time Connect addresses each of the logging requirements with little or no application specific design or software development. In summary, the RTI Real-Time Connect architecture provides the basis for developing an efficient distributed message auditing subsystem.
For more information about RTI Data Distribution Service, DDS, or RTI Real-Time Connect, please visit www.rti.com.
