This chstpter analyses how the generic R T D system was designed to take any form o f real time data from multiple sources and provide the data to a number o f client programs. T h is system was developed in collaboration with Black A ce Software E ngineering a com pany specialising in real time system s on PC’s.
5.1. Introduction
It was realised from the previous projects that it would be possible to create a system that could handle data in multiple formats from multiple sources. The architectures studied in the previous chapters have the ability to take data from more than one source, the problem was that the data has to be in a fixed format. Another drawback of the previous architectures was the lack of a general model for filtering the data.
This was going to have to change in order to create a more generic system, where the format of the data cannot be predetermined. In order to create such a generic system the architecture was designed based on previous architectures but addressing the issues of generic data types and an open model for filtering. This chapter looks at the software engineering techniques employed to create such a system along with the actual design. Also considered is the issue of how the system was designed in order to plan for future changes. This system is called RTD, which stands for Real Time Data The rest o f this chapter looks at the first three stages of the waterfall model as applied to the engineering of the RTD system.
5.2. System Engineering
The reasons for developing RTD were to create a system that could:
1. Take any type of data as its input; whether the data was from a financial institution or the position of a vehicle would not make any difference as
to how it is input to the system. There would need to be an input interface that could cope with multiple data types.
2. Take data from more than one feed; this input interface would also need to cope with input from more than one source.
3. Filter the data for information that was o f interest to the user; some mechanism for deciding what information the user wants would be required.
4. Provide the data to one or more client programs; an interface for exporting the data in a timely manner to client programs would be required.
5. Store the data for later analysis; the system would require some form of database system to archive the data to a storage medium.
From this list the diagram in figure 5-1 was drawn up.
Data Feed RTD Data Client Data Feed Data Client Data Feed Archive
Figure 5-1 How the system will act to control data flow.
5 3 . Analysis
The information domain of the system was not easy to define, as there was not a specific goal for this system but rather to provide a generic solution to a problem that had been encountered previously. All that could be said was that any data entering the system could be placed in this system and then filtered
This system would be required to filter the data and only keep that data which the user would require. It would also have to archive the data, it would be prudent to allow the user to choose what should be archived and how often. Depending on the inputs and filtering huge amounts of data could amass very quickly.
At this stage is would not be possible to gauge performance levels as the input volume is not a known factor. For this reason the system could not be defined as a hard-real time system.
The interfaces required are both for the input o f data as well as the extraction of data. There would also need to be some way of extracting the archived historical data. In order to provide a generic interface that was expandable an open and uniform data transfer protocol would need to be chosen.
5.4. Top Level D esign
The diagram in figure 5-2 was drawn up as an initial idea for the lines of communication between the code modules and the storage system. This was based on the architectures presented in the two previous chapters.
Storage
LiveData Historical Data
Data Director
Historical Data Object Manager LiveData Object
Manager
Hybrid Live/Historical Data Object Viewing and
editing tools Historical Data ObjectViewing and editing tools LiveData Viewing and
Editing Tools
Storage
The storage engine would need to be user definable as either an ODBC compliant relational database, or a proprietary storage mechanism which would be based on the Compound Storage Model using the COM Structured Storage implementation. It would be necessary to provide COM storage methods for the data objects in order to link and embed them in other container applications and so it would be fairly trivial to take this a stage further and provide the full storage mechanism if no database is required. An ODBC relational database would contain the tables shown in figure 5-3.
Historical Data ID Name
Contents Description Historical Data Archive Data Folder ID Name
Contents Description Live Data Folder
Tick Item Tick ID Data Item ID Field Type Field Value Date/Time Index Data Item ID Data Type Parent ID Name Contents Description Date/Time Data Item
Figure 5-3 Database Tables and Relationships
System Folder Table - used as a means of grouping the Data Items that are live.
Historical Data Archive Table - for grouping to o th er data items relating to the same thing over a period of time.
Data Item - can represent either a live piece of data or an archived piece of data, the Data Type field will indicate which. The parent ID will then reference either a System Folder or Historical Data Archive.
Tick Item - represents a field and its value at a given time for a Data Item. The Data Item ID is a foreign key pointing to the Data Item to which this field belongs. The Field Type field indicates which field this tick represents E.g. Trade price. The Field Value gives the value of that field. The index can be
LiveData Object Manager
The LiveData Object Manager would be a process in the system which accesses the data base and converts the live data into exportable COM objects conforming to the WOSA standards plus proprietary extensions.