Contract Routing No. 04 HAA 00063
Robert Wycoff CDOT ITS Office
Signature: ___________________________ Date: _______________________________
John Williams CDOT ITS Office
Signature: ___________________________ Date: _______________________________
Confidential ©EnRoute Traffic Systems, Inc. Page 2 of 17
Date Version Description Author
08-Mar-2004 0.1 First Draft Jason Westra
27-Apr-04 0.2 Scheduler and Communications are
services, not layers. Added Delegate Layer.
11-May-04 0.3 Added Notifications service and Services
21-May-04 0.4 Added New Architecture Drawing Sachin Saindane
07-March-05 1.1 Added more depth around layers and
Confidential ©EnRoute Traffic Systems, Inc. Page 3 of 17
Table of Contentsi. Executive Summary 5 1.Introduction 6 1.1 Purpose 6 1.2 Scope 6 1.3 Audience 6 1.4 References 6
2. Logical Architecture Reference Model 7
2.1 Architectural Layers 7 3. Application Layer 9 3.1 Client Layer 10 3.2 Delegate Layer 11 3.3 Messaging Layer 11 3.4 Services Layer 12
4. Application Support Layer 13
5. External Services Layer 14
6. Platform Services Layer 15
7. Tiers 16 7.1 Client Tier 16 7.2 Presentation Tier 16 7.3 Business Tier 16 7.4 Integration Tier 17 7.5 Resource Tier 17
Confidential ©EnRoute Traffic Systems, Inc. Page 4 of 17
Table of Figures
Figure 1: A Four-Layer Organizational Structure For Application Components ...7
Figure 2: Conceptual Application Layer ...9
Figure 3: CTMS Application Layer...10
Table 1: Application Support Layer ...13
Table 2: External Services Layer...15
Confidential ©EnRoute Traffic Systems, Inc. Page 5 of 17
This particular document describes the roles and responsibilities of the various layers that are central to the CTMS/CTIS architecture and catalogs the services available to the architecture.
Confidential ©EnRoute Traffic Systems, Inc. Page 6 of 17
This document describes the application architecture of CTMS and the various roles and responsibilities of each layer that contributes to the overall CTMS/CTIS software architecture.
The purpose of this document is:
• To supplement the technical information provided in the Messaging Architecture Assessment Document. • To capture and convey the significant messaging architectural issues and decisions which have been made
during meetings and discussions.
The scope of this document is the CTMS/CTIS system.
This document describes the underlying messaging architecture within the core CTMS architecture from various perspectives. Since most sections are technical in nature, the document is focused towards technical managers, technical architects and the software development team.
The following references were either used in the creation of this document or may be used to supplement the detailed descriptions of the CTMS/CTIS system:
• The CTMS/CTIS Architecture Assessment Document.
• The CTMS/CTIS Messaging Architecture Assessment Document.
• Design Patterns: Elements of Reusable Object-Oriented Software - Gamma, Helm, Johnson, Vlissides • Core J2EE Patterns: Best Practices and Design Strategies – Alur,Crupi, Malks
Confidential ©EnRoute Traffic Systems, Inc. Page 7 of 17
2. Logical Architecture Reference Model
The logical architecture makes it possible to communicate the design of the system with both technical and business audiences. It covers the structure, guidelines, and capabilities of the system without getting mired in the details of the technical implementation of the architecture such as protocols and interfaces. The logical architecture organizes broad definitions into categories and organizes them in such a way as to allow the implementation to be carried out effectively and efficiently. It provides a way to help determine build vs. buy for each category whether at a coarse-grained or granular level. For instance, it can help make decisions like whether or not to implement a whole subsystem with COTS (e.g., an ERP solution) or just a few services in the subsystem with a specific OTS (commercial off the shelf) product.
2.1 Architectural Layers
Application services may be organized into layers, each representing a gross classification of the degree to which services in the layer have to do with business or technology. Services in higher layers have more to do with business and are most likely custom built while services in lower layers tend to be more technical and are often purchased rather than built.
Figure 1, below, illustrates a four-layer classification scheme with Application, Application Support, External Services and Platform Services layers. The foundation of the application architecture is the Platform Services that represent the raw technology upon which a solution is built. Examples include operating systems, computer and networking hardware, operating systems, network operating systems, interface cards, and communication protocols.
Application Support Layer
External Services Layer
Platform Services Layer
Provides the user interface, business logic, and process flow custom-built for the application.
Example: UI, images, and services
Provides common, but custom services to the Application Layer that function as adapters/ wrappers to External Services.
Example: A wrapper around SNMP protocols or a J2CA Connector to a legacy system
Provides third-party APIs and components that the application must interface with. Example: DB, Java Mail API, Java Comm APIs
Network, operation systems, and hardware. Example; Routers, Unix on Solaris, firewalls
Figure 1: A Four-Layer Organizational Structure For Application Components
The External Services Layer represents purchased or pre-existing software with which your system must integrate. Examples include database management systems, messaging services, application monitoring services, e-mail, and legacy systems. These services are purchased or pre-existing and rely on underlying Platform Services Layer in order to operate.
The Application Support Layer sits above the External Services layer. Services in this layer may include application security (e.g., authentication and authorization), object identity and naming, exception and event handling, logging, and transaction management. These services form the common, architectural plumbing of your enterprise
infrastructure and are reusable across multiple applications. In addition to the services just mentioned, perhaps one of the most powerful services in this layer is the adapter service that can be used to encapsulate access to external and platform services, insulating Application Layer components from technology changes.
Confidential ©EnRoute Traffic Systems, Inc. Page 8 of 17 To understand how each layer provides insulation for the next, consider this example. Multiple components in the Application Layer require authentication and authorization services. Users and permissions are stored in a relational database. The components requiring security services call a Service Adaptor interface that exposes methods like “login(username, password)”, “logout(user)”, and ‘hasPermission(permission, user)”. Depending on the request, the Service Adaptor might query the relational database using SQL to validate login credentials or to see if the user has the permission specified.
As the company grows, an enterprise security provider is chosen and all applications must now conform to. It happens to use an LDAP Directory Server for the storage of users and permissions. The Service Adaptor simply needs to change its implementation to make LDAP directory calls now instead of SQL queries. Applications that used the Service Adaptor did not have to change. Unfortunately, applications in the company that did not use the Service Adaptor have to change each “touch point” in the system to conform to the new LDAP directory APIs. At the top of the architecture stack, the Application Layer represents services unique to a particular system—its presentation, business logic and data storage logic. These are nearly always custom built components because they have to do with a specific business problem.
Adherence to layering rules increases the technical agility of a system in that underlying technology can be replaced with minimal, if any, impact on the operation of higher-level services. The degree to which this type of layer encapsulation is required is a practical matter, depending on both the anticipated rate of change in the External and Platform Services layers and the number of Application Layer components that use a lower layer service.
Confidential ©EnRoute Traffic Systems, Inc. Page 9 of 17
3. Application Layer
The following section details the Application Layer of the CTMS in more depth. The Application Layer described above is further broken into sub-layers for purposes discussing sub-layer responsibilities as well as guidance for implementation of the Application Layer.
Supports Web, portable, and GIS-based thick clients
Loosely coupled services perform business logic and data access
Gateway to Services Layer orchestrating services into complex units of work
Transport bus for messages to and from clients and services
Figure 2: Conceptual Application Layer
The conceptual architecture of CTMS is depicted in Figure 2 as a service-oriented architecture where services can be implemented as anything from Web Services, EJBs, Java Management Extension (JMX) MBeans or even plain old Java objects (POJOs).
Requests are sent via the Messaging Layer to the remotely accessible Delegate Layer, which acts as a routing agent. The Delegate Layer routes the request to Services, which execute behind it and are only accessible from this layer. The architecture supports a Messaging Layer with any type of protocol including RMI, IIOP, JMS, SOAP, and HTTP/S and both asynchronous events as well as synchronous RPC-like calls. Because services are deployed behind the Delegate Layer, they are easily distributed and clustered. The following diagram gives a more technical look at the server-side layers and how services interact with other services using method invocations and events.
Confidential ©EnRoute Traffic Systems, Inc. Page 10 of 17 Delegate Layer Service Service Service Service Messaging Layer Event Event
Figure 3: CTMS Application Layer
In Figure 3, services never communicate directly to one another. All events whether synchronous or asynchronous are sent over the Messaging Layer and routed by the Delegate Layer to the appropriate Service to handle the request. The Delegate Layer may route the request to multiple services or just a single service.
The following layers are discussed in detail: 1. Client Layer
2. Delegate Layer 3. Messaging Layer 4. Services Layer
3.1 Client Layer
The Client Layer will be implemented as a thick, Java client across 3 screens (a GIS map, a notifications screen, and at later iteration, a CCTV (Closed Circuit Television) screen for camera input). It is responsible for displaying the user interface to the users of the application. It must be able to send and receive both synchronous and
asynchronous communications via the Messaging Layer. Examples of asynchronous notifications include alarms, errors, and replies to instructions from operators.
The main responsibilities of the Client Layer are: 1. Display the GUI (graphical user interface) 2. Initiate system requests
3. Handle messages from the Messaging Layer 4. Handle errors
Confidential ©EnRoute Traffic Systems, Inc. Page 11 of 17
3.2 Delegate Layer
The Delegate Layer is the “gateway” to the Services Layer. More importantly however, the Delegate Layer is responsible for orchestrating individual services in the Services Layer into more complex units of work. It controls the transactional nature of a request and provides subsystem abstraction for services and legacy systems. Clients use the Messaging Layer to contact the Delegate Layer, passing it a message either synchronously or asynchronously and the Delegate Layer routes each message to the appropriate services.
The main responsibilities of the Delegate Layer are: 1. Routing of requests to proper services
2. Coordinate multiple services into a coarse-grained unit of work 3. Abstract core subsystems into more manageable interfaces 4. Apply security controls to clients for services they request
5. Act as an auditing mechanism for requests in and out of the system
The Delegate and Services Layers of the architecture should be consistent across a clustered deployment. This means that the services and orchestrations deployed across a clustered application are exactly the same on each node. This makes it easier to configure, deploy, and implement.
As stated earlier, the Delegate Layer must provide a remote interface for clients to connect to and call services. However, the Delegate Layer must also be smart enough to know that it is already within the server space and when to use non-remote APIs, which are more scalable. An orchestrated sequence of service calls should be local to the server space. Likewise, calls from services to the Delegate Layer should be localized. Since the same services are deployed across all cluster nodes, any service is always locally accessible from the Delegate Layer once inside the server space.
3.3 Messaging Layer
The Messaging Layer represents the protocol, protocol handlers and underlying infrastructure used to communicate to and from the Delegate Layer. The layer can be implemented using protocols such as SOAP, RMI, IIOP, or JMS and protocol handlers that know how to connect, make requests, and receive replies on the protocol of choice. The Messaging Layer is the bus that messages are passed on between services either synchronously or asynchronously. The implementation must be clustered for load balancing and high-availability. For instance, if Java Web Services are used, they should be servicing requests from a distributable Servlet Container. If Stateless Session Beans are used the EJBs should be clustered to load balance requests across EJB Containers. If the implementation is JMS, the JMS Provider should support fail-over using a persistent message store and it should support load-balancing
messages across numerous JMS receivers.
The Messaging Layer has the following major components:
1. Channel – the destination upon which messages are sent and received. In a JMS implementation, this is a Queue or Topic.
2. Consumers – Handle the messages on the Messaging Layer 3. Producers – Send messages to the Messaging Layer The main responsibilities of the Messaging Layer are:
1. Secured, anonymous connectivity to services over an asynchronous message bus 2. Capabilities to administer channels (add, remove, query channels)
3. Allow remote notifications to groups of listeners (e.g., JMS Topics) 4. Allow remote notifications to a specific listener (e.g., JMS Queue) 5. Provide fail-over capabilities in the event of a communications failure.
a. For example, a JMS Provider that stores messages persistently
b. For example, a JMS Provider that provides automatic fail-over of JMS consumers to a secondary JMS Server
c. For example, a JMS Provider that provides automatic fail-over of JMS producers to a secondary JMS Server
Confidential ©EnRoute Traffic Systems, Inc. Page 12 of 17 d. For example, if a node is down, an RMI client might contact another node in the cluster and
resume request processing.
As an architectural guideline, the number of channels should be kept to a minimum to ease the burden of channel and message management on the provider. This generally leads to a pattern where a message consumer handles a message by passing it off to a secondary handler in the form of a Command pattern.
3.4 Services Layer
A service has a well-defined interface describing the services it provides to clients. A service can be invoked remotely not assuming that the client of the service is operating in the same environment. The CTMS architecture designates the Delegate Layer as the gateway to services, so services must expose their APIs through the Delegate Layer if they are to be remotely accessible. Consequently, services exist within a computing space (e.g., a J2EE Server) and can be designated as “local” to this space.
Services are self-contained, that is, they can be used independently of any other services. Services are very loosely coupled and do not interact with each other, but instead perform very succinct tasks and rely on the Delegate Layer to coordinate more complex, multi-step services to fulfill a request.
A service is loosely coupled. This characteristic is achieved by enforcing that no service references another service directly. Instead, the services live under these guidelines:
1. Services use the Messaging Layer to make anonymous requests that are handled by the Delegate Layer as it sees fit.
2. The Delegate Layer handles coordination of independent services to achieve complex request handling. 3. References to services can be discovered rather than fixed at development or deployment time. This once
again, is handled by the Delegate Layer, which has the capability to find services dynamically.
Not only are services decoupled from each other, the Services Layer is decoupled from the Client Layer as well. Thus, services must never connect directly to a client to make requests or replies directly to the client. This would tightly bind clients with services and vice versa. It would also make security and auditing more difficult to provide if each service was opened up to the Client Layer.
Confidential ©EnRoute Traffic Systems, Inc. Page 13 of 17
4. Application Support Layer
Components in the Application Support layer may be common to multiple applications within an enterprise. They are considered “plumbing” and therefore do not offer significant business value to individual applications using their services.
Services in this layer often include the following:
Category Name Description of Service
Application Security These are components that have to do with security the application and typically involve user authentication, access authorization, and non-repudiation. May also include auditing and management of user accounts and account activity.
Common Libraries This represents 3rd party utilities that perform basic operations
such as parsing XML, regular expressions, and data validations. These are used within the same process as the application; whereas, an External Services Layer component might function in a separate address space altogether.
Exception Handling Handles errors in the application in a standard way. Usually integrates with the Logging service to provide information about the errors to system administrators and developers alike. Event Handling Handling events posted from application components. Applies
persistence to the events to prevent loss during application failures.
Logging A specific type of service adapter called out because of the significance that logging plays in production operations. The service provides ways to log various levels such as DEBUG, WARN, and ERROR as well as guidance to development teams on which to use.
Object Identity and Naming This service generates globally unique ids (GUIDS). It usually requests the service from a database or other shared resource. It can provide caching algorithms to enhance performance where the number of transactions per second is high.
Service Adapters In simple terms, these are “wrappers”. Service adapters represent a interfaces to external services to isolate the implementation of a service from the protocol the application uses to access it. In J2EE, these are J2CA Connectors.
Transaction Management Implements services to track business data changes. Represents them in a “logical unit of work” (LUW) to promote ACID transaction processing at the resource level.
Confidential ©EnRoute Traffic Systems, Inc. Page 14 of 17
5. External Services Layer
External Services are almost always 3rd party products when discussing buy vs. build options. External Services generally provide the capability to run on their own and provide an API for an application component to request services from it. Often external services are grouped together to form more useful products. For instance, Application Servers bundle resource pooling, a transaction coordinator, a message bus and a web server together into one product offering. Nonetheless, there are an infinite number of combinations of external services and many are not listed here at all. However, the table below does provide some of the more common services in this layer.
Category Name Description of Service
Application Server Generally, a class of server that bundles multiple Application Support Layer and External Services Layer categories into a single product. There are two standards available: J2EE and .NET. Examples of J2EE servers include BEA WebLogic, IBM WebSphere, and JBoss. The most notable .NET example is Microsoft’s .NET platform.
Application Monitoring Services Applications that monitor the health of the system and report anomalies to any number of media including emails, pagers, logs, and monitoring consoles. Generally, JMX and SNMP are used to integrate systems with console vendors.
Business Process Management Software that allows the creation, deployments, and monitoring of business processes. Usually includes workflow, automation, and business activity monitoring capabilities.
Clustering and Replication Services Hardware and software that enable high-availability operation of the system wherein two or more homogeneous components operate as single entities and back up each other in the event that one fails. Includes session-state replication, and database synchronization, for example.
Content Management Services Services that provide a user-friendly interface to manage the content of a web application. Generally, used by business users to alleviate the technical users from this task.
Database Management System Any data storage system including relational databases and object databases. Examples: Microsoft Access and SQL Server, Oracle, and Sybase.
Directory Server Directory services are used in distributed applications to provide classifications/properties, lookup services, and authentication. They are generally LDAP and accessed by an API like Java Naming and Directory Interface (JNDI).
Distributed Timing Service A distributed timing service that synchronizes all clocks on all computing devices on the network.
Message Bus Servers that that enable and guarantee delivery of asynchronous communications between components.
Confidential ©EnRoute Traffic Systems, Inc. Page 15 of 17
Category Name Description of Service
Network Security Hardware and software components used to lock down the network from viruses and intruders. Generally provided by firewalls and intrusion alert software.
Personalization Services Services that track and profile users behavior to personalize content and campaigns to individuals.
Resource Pooling Supports pooling of expensive resources such as remote connections and database connections. Provides and API to get and return the resources to and from the pool.
Runtime Environment Provides a runtime environment for application components. The best example is Java Virtual Machine, while the C# runtime is also gaining popularity. Other systems like Visual Basic require DLLs to run properly.
Transaction Coordinator Services that enable the two-phase commit protocol and/or compensating transactions for components whose functions span multiple resources.
Web Server This type of server generally excels at serving static and dynamic content over the HTTP protocol. Examples include Apache and IIS.
Workflow Engine Coordinates manual tasks between humans. Task lists are shared and individual allowing tasks to be acknowledged and completed by a single person. Tasks are usually persistent allowing them to be tracked and performed over extended periods of time.
Table 2: External Services Layer
6. Platform Services Layer
The Platform Services layer is the foundation of the application architecture. It represents the raw technology upon which a solution is built. Examples include operating systems (OS), computer hardware, networking hardware such as routers, switches, and hubs, interface cards, communication protocols, mass storage, and replication technologies. This layer is 100% technology based with zero business affinity and is almost always purchased as a 3rd party product rather than built by the development team.
While this layer has zero business affinity, each layer above the Platform Services Layer must be able to co-exist and operate efficiently upon this foundation. Luckily, many standards exist today that negate the need to design solutions around proprietary hardware and networking specifications. For instance, standards exist for network protocols such as TCP/IP and wireless standards like IEEE 802.11b and IEEE 802.11g that allow network hardware to be easily swapped with little worry of interoperability.
Table 3 covers the more common Platform Services in more depth.
Category Name Description of Service
Computer Hardware Includes components such as cables, UPSes, monitors, motherboards, AGP and PCI cards, CD/DVD-ROM drive, keyboards, memory sticks, processors, and mice.
Confidential ©EnRoute Traffic Systems, Inc. Page 16 of 17
Category Name Description of Service
Communications Protocols Describes how low level communications are transmitted over hardware and network devices including TCP/IP, 802.11b and IEEE 802 for wireless, and RS-232 for serial communications. Network Hardware These components include CAT5 cable, fiber, routers, hubs,
switches, NIC cards, wireless cards and wireless access points. They are hardware devices designed to connect other devices over a wired or wireless network (usually Ethernet-based) and compatible with standards such as TCP/IP, IEEE 802.11b and IEEE 802 depending on whether it is for wireless networking or not.
Operating Systems Provides insulation over the hardware to make it easier to work with hardware devices through plug-and-play. The most common are Windows and various flavors of Unix and Linux such as HP-UX, Solaris, Red Hat, and Suse.
Other devices This category is broad and includes anything non-typical such as cameras, TVs, radars, etc.
Table 3: Platform Services Layer
Layers provide the language necessary to discuss the logical structure of systems independently of their physical deployment. We use the term tiers (or optionally, partitions) to describe the run-time topology of the system; that is, which of the services are deployed on which of the network computing nodes.
Node types (or tiers) common to J2EE applications are Client Tier, Presentation Tier, Business Tier, Integration Tier, and Resource Tier (Core J2EE Patterns, pg. 120-121).
7.1 Client Tier
This tier represents where the presentation is rendered. Typically, a web application presentation will render in a browser. A wireless application may be rendered in a handheld device such as a PDA or phone. On the other hand, a thick client runs on a desktop computer using a graphics card to render the display. The desktop computer may have one or more monitors depending on the complexity of the application. Finally, a Client tier can also be represented as a headless client such as a batch process that renders nothing or only outputs information to logs.
7.2 Presentation Tier
The Presentation tier is responsible for providing a view into the system by serving up dynamic web content or client windows. If it is a web application, it typically includes one or more computers in the network demilitarized zone (DMZ) where browsers make contact with the application and which serve static content. The Presentation tier for a Java client shows views of the application such as Java Swing windows and widgets on a desktop computer. Usually, initial authentication and authorization take place at this tier. If using OS based authentication such as an NT Domain login, this logic already took place in the Client tier.
7.3 Business Tier
The Business tier represents one or more computers in the secure area of the network that provide common services and business logic to application clients. Components in this tier are EJBs and Web Services that serve the business
Confidential ©EnRoute Traffic Systems, Inc. Page 17 of 17 logic and can be deployed across multiple nodes in a cluster ensuring load balancing and high-availability for the services.
This tier is also responsible for managing transactions, which can span multiple Business and Integration tier components. For instance, a transaction in the Business tier could include calls to an EJB, a Connector, and then send a message via JMS, all of which must commit or rollback together. Transactions in the Client, Presentation, or Integration tiers are not recommended for many reasons. Some tiers are inherently unstable (e.g., a wireless PDA in the Client tier), which could leave resources locked in transactions until a timeout occurs. If the Client and
Presentation tiers start transactions they will most likely be distributed across multiple nodes, which is costly as well. Finally, if the Integration tier manages transactions, then it’s possible for components in the Business tier to be left out of the transaction.
7.4 Integration Tier
The Integration tier communicates to external resources, which can be hosted on any number of computers on a secure network. Components in this tier include JDBC, J2CA Connectors, and JMS for messaging. They could also include proprietary software such as the case with 3rd party vendors who provide adaptors to their services.
7.5 Resource Tier
The Resource tier contains business data and resources shared across applications. This tier may include databases, legacy systems, 3rd party services, or directory services such as LDAP servers to provide authentication and authorization capabilities to the application. Some components in this tier may contain business logic especially if they are legacy applications or ERPs.
Multiple tiers generally rely on the Resource tier. For instance, the Client tier may initiate login to a workstation by verifying credentials against an LDAP server. Likewise, the Presentation tier will authenticate and authorize access to web resources and the Business tier may authenticate, authorize, or even administer the directory. Similarly, a relational database is usually the preferred data store for today’s applications and a single database may be shared by many applications.
7.6 Achieving Deployment Agility
As it turns out, strict adherence to architecture layers, rules and design patterns provides a great measure of deployment agility as well, making it easier to scale up to a larger number of users or processes. An initial
deployment of a web application, for example, might host presentation, business and integration components all on one server or server cluster. However, as the application scales, it may be necessary to move presentation, business and integration behavior to different nodes so that each node’s resources are better utilized.
Without proper planning, redeployment of components may incur unnecessary communication overhead (e.g., too “chatty”) and may require software modifications to handle remote communications efficiently. If the architect anticipates redeployment as a possibility, he or she may design scalable communications into the software with an eye on future deployment strategies.