The CBOS/CBIBS
Data Management Framework
December 4, 2007 Tucker Pierce
CBOS/CBIBS Data Management Consolidation
• We are currently undergoing a CBOS/CBIBS Consolidation to make NCBO’s data management activities for both CBIBS and CBOS less redundant, more cost effective
• The new data management framework will make broad use of state-of-the-art Web Services standards for both ‘front-end’ (user side) and
‘back-end’ (data provider side) interfaces
• Data-providers, modelers, and end-users will need less system specific knowledge to interact with the system
The consolidation of CBIBS/CBOS data management functions will allow us to do more with the available resources…
A Web Services approach will be employed to reduce the cost associated with NCBO’s Data Management Framework
• Easier incorporation of new data sources within an existing information management framework
• Quality assurance implementation for new data sources, as required • End-user information discovery, ‘sub-setting’, and data access tools • Metadata definition, implementation and management tools
• Business logic functions- applications and building blocks for tailored end-user products
WWeb Services architectures allow systems and applications to communicate
independently of operating system, programming language, database structure, and location
System Concept
Data Management Framework
Archiving Retrieving
Data Quality Control
Applications (e.g. Models) Web Client Services
Metadata Sensor Systems And Assocaited Databases Syndicated features, tools, and data Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications IVR application Web Browser Applications Web Browser Applications Web Browser Applications Server application • Sensor system interface
approach decouples framework from sensor hardware specifics
• Standards-based (XML) Web Services ingest
• Developer keyed access
‘Back End’
• Provides for data discovery, access, and supports third party apps development
• Standards-based (XML and JSON) data interchange format
• provides for syndicated data feeds such as RSS, Yahoo Pipes
System Consolidation
• The ‘old’ CBOS will become the system’s ‘back-end’ function, responsible for interfaces with current and new data sources and providers
• The ‘old’ CBIBS will become the system’s ‘front-end’ function,
responsible for interfaces with current and new users, and actively supporting the development of data products
• This change will be transparent to:
– End-users directly connected to a particular sensor database – Web-services clients of the CBIBS system
CBIBS buoys Mobile Sources Client Data Request Handler Data Retrieval Handler
New CBOS/CBIBS Web Services Framework Overview
DBASE DBASE Business Logic Business Logic Business Logic Applications T Public Internet Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications Web Services Interfaces National Weather Service Web Services
Interfaces Web ServicesInterfaces
QA/QC Metadata Cache Data Archive Handler Storage NERRS CDMO National Estuarine Research Reserve
External Data Sources
CBOS Sensors DBASE DBASE . . . DBASE
Sensor Data Acquisition and Storage T Data Source 1 Data Source 2 Data Source 3 * * Data Source n System DB System DB System DB System DB (To DMS Framework) Data & Info Storage
The Data Archive Handler is responsible for archiving the data received from the observing system sensors (or other sources) to the appropriate local CBOS/CBIBS data storage facility
The DAH - Transformation module performs any data modifications required for the newly acquired ‘raw’ data Metadata
QA/QC Data Archive Handler
Transforms
TCP/IP XML
The DAH - QA/QC module performs the necessary quality checks on the incoming data, and generates the appropriate system messages in the case of anomalies.
The Metadata module contains ‘data about the data’’, e.g. sensor type, calibration history, and other information that gives the user confidence in the validity of the data contained in the archive (Buoys, sensors, meters, etc)
The CBIBS Data QA/QC Function
T
The QA/QC module is a part of the Data Archive handler
Data Retrieval Handler and Business Logic
T
Data Retrieval Handler
Business Logic Business Logic Business Logic
External Data Sources OpenDap Connector
NT Archive Connector NDBC
(as required)
Services connector
Construct, send, receive OpenDap commands and data to and from NDBC
Manages access to the server’s local near-term archive to satisfy user service request
Manages access and retrieval of data from external sources, e.g. NWS, NERRS.
Models, procedures, functions, performing value added services on demand as required to fulfill user needs
This function represents ‘Push’ based feed from external data or information source, e.g. NWS, NERRS
The Data Retrieval Handler locates and retrieves data and products requested by the internet based ‘front end’ function. It also initiates business logic processing for either internal or external applications supporting value added data product development for target clients.
Data Communications Interface XML Data Storage (from Data Archive Handler)
Client Data Request Handler
and ‘front-end’ Web Services interface
Public Internet Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications Web Browser Applications IVR application Web Browser Applications Web Browser Applications Web Browser Applications Server application
Client Data Request Handler
Direct Access Mgr
SQS (as required)
The Client Data Request Handler manages all connections and data requests from client (internet browser) applications.
Browser applications use API key for access to framework data and/or services
(to the Data Retrieval Handler)
XML Data Comm interface
XML
The system first checks the Near Term Archive for any data requested. If not found, system reformats request as necessary to retrieve the data from the appropriate source The Near Term Archive is a queue which ‘pushes’ older data out in lieu of data which is in more frequent demand
Note: Provision has been made to accommodate the Amazon Simple Cue Service if and when the need arises to ‘load balance’ service requests from the
CBOS/CBIBS Web Services Capabilities Under Development
• Expansion of the QA/QC service function
• Build Out of New Interface Types - to support all the popular tools – SOAP
– REST
• Metadata module expansion
• Build Out of Discovery/Interoperability Capabilities – UDDI/WSIL
– Enterprise System Bus Architectures (BPEL)
• Build Out of Authentication and User Enrolment
– Self Service (or near Self Service) Developer Enrolment
– Provides fine-grained access and usage control, e.g. quotas, throttling, private data (as required)
• Enhanced Security (when needed) – HTTPS/SSL
• Enhanced Administration
SUMMARY
• Provides a current, industry-standard Means of Inter-operating with the CBOS/CBIBS data management framework
• Encourages the development of third party applications and data products • Reduces the cost of maintaining the system
• Reduces the cost of interacting with the system for providers and users • For the ‘Back End’ (data provider side)
– Increases choice of back-end data sources (databases, data sets)
– Increases options for sensor hardware and software • For the ‘Front End’ (data user side)
– Increases choice of service requester technologies
– Any tool conforming to standard may be used to access and retrieve data
The benefits of consolidation and adoption of a ‘Web Services’ Architecture
