Middleware Architecture for Federated Control
Systems
Girish Baliga and P.R. Kumar • University of Illinois at Urbana-Champaign
A federated control system (FCS) is composed of autonomous entities, such as cars, that cooperate to provide a global behavior, such as traffic control. A middleware is a convenient mechanism to separate software infrastructure, as a layer, from other FCS components. Element abstraction is a model for designing, implementing, and operating software components on this infrastructure layer.
W
e focus on systems composed of entities operating with autonomous control, such as automated production plants and traffic control systems. Traditionally, the software controlling such systems has been tailored to each system’s requirements. However, creating custom, one-of-a-kind designs is costly and not conducive to proliferation. In recent years, technology has evolved to the point at which powerful processorsembedded in the environment can support generalized approaches. Such approaches use formal theory and well-defined design methodologies, and can be made available as commercial technologies. However, generalizations must evolve from experience and be validated in practice.
This article introduces the notion of federated control systems through a study involving a complex traffic control system under development. We use the notion of elements to represent a service abstraction for software components in the FCS and present a concomitant middleware architecture to support this service abstraction.
Prototype system
Figure 1 illustrates the prototype system, which consists of a set of cars moving on a track. The cars are controlled remotely using an RF transmitter, which is connected to the serial port of a dedicated laptop. A program controls the car’s speed and direction by executing a control algorithm on the dedicated laptop.
Controls are sent to the serial port as a stream of alphabetic characters. The port is connected to a microcontroller that converts the character stream into a sequence of signals to operate the car. The control program that runs the car performs closed-loop control and obtains the necessary feedback from an array of cameras (currently two). At
intersections, traffic lights illustrate traffic arbitration.
Figure 1. Prototype federated control system.
A set of laptops operates the system, with each laptop controlling one car. A set of desktops process vision feed from the cameras and send the feedback to the laptops. The desktops run on Windows, and the laptops on Linux. The computers are connected by Ethernet and an 802.11 wireless LAN.
The prototype represents a class of systems, each composed of sensors, actuators, computers, and communication networks, in which various control algorithms operate on subsystems with strict real-time constraints. We categorize such systems as the class of federated control systems fitting the following definition:
A federated control system (FCS) is a networked system of sensors, actuators, and computers that are composed into hierarchical sets of cooperating entities operating with autonomous control.
Sensors and actuators represent the interface between the system and its environment. Computers represent programmable system components, which are networked by a topology of connections imposed by system design, technology, and convention.
Compositions of components represent independent system entities, which operate with autonomous control. In the prototype system, a car exemplifies this type of
composition. It consists of a controlling laptop, an RF connection, and actuators on the car itself. Useful behavior is obtained when these autonomous entities cooperate to
achieve desirable goals. We can compose a set of autonomous entities, such as cars, into a higher-level autonomous entity, such as a fleet of cars, and so on. This hierarchy of sets of autonomous entities represents an FCS.
Simple client-server architecture
Our study of the prototype FCS has paralleled our implementation of a simple working system. The system design emerged as a bottom-up composition of code fragments that were written for simple use cases 2 of various subsystems. Figure 2 shows the overall
software architecture of the first working implementation. The vision sensor is a camera providing a vision feed to a dedicated desktop that executes an image-processing
application and determines the position and orientation of cars on the track using a local coordinate system. A central data server connects to the vision sensors (the programs on the desktops), and obtains the vision information for all cars in the system, which it then transforms to a global coordinate system. A central trajectory server computes and serves trajectories for all cars in the system. A trajectory is a sequence of car position coordinates in time. The trajectory server obtains location information about the cars from the data server.
Figure 2. A simple client–server architecture for the prototype system.
Each laptop executes a control algorithm that moves the car according to a trajectory provided by a trajectory server. The control program obtains feedback from the data server. This implementation is working quite well. We have tested scenarios with up to eight cars operating simultaneously on the track and closely following prespecified trajectories. However, we encountered some fundamental problems during system implementation.
l Location independence. Operating the system required considerable
preconfiguration. For example, all the clients needed to know the addresses and ports of the respective servers. Moving the servers to different machines entailed
much reconfiguration.
l _Bootstrap. Starting up the system was an elaborate process. The servers and
clients needed to be started in the right order and on prespecified computers. Synchronization (implemented at the trajectory server) was needed to ensure that all cars were ready before any car started to move.
l Software update. We had to propagate a change in the car control program to all
the laptops, requiring a system-wide reboot.
l Software model and architecture. Multiple team members working on the different
problems of image processing, car control, car scheduling 3, and infrastructure
developed the code concurrently. Separating the infrastructure code as an independent body in this context was particularly challenging. Maintaining and updating this code required changing and testing the other code using it.
The next iteration of the system implementation addresses these and other problems. The following sections outline some of our ongoing research.
Related research and applicability
Some of the problems mentioned earlier, such as location independence and
preconfiguration, have elegant solutions in commercial middleware technologies such as CORBA 4, DCOM 5, and Jini 6. However, using these technologies as the base
infrastructure has some fundamental problems. One key problem is that they are intended for transaction-based applications, where the focus is on providing a rich interface. FCSs require an event-based infrastructure, in which data or control update can proceed asynchronously.
The Open Control Platform (OCP) 7 extends CORBA to support event streams and other
primitives. However, some unsupported features remain. For example, OCP doesn’t address the software update problem, which must be engineered into the system. In addition, OCP is essentially a transaction-based system with relevant functionality imposed as a new layer.
Message-oriented middleware such as IBM’s WebSphere Message Queuer 8, Sun’s Java
Message Service 9, and Microsoft’s Message Queue Server 10 represent commercial
event-based middleware technologies. These middleware platforms focus on business applications, however, which value secure data delivery over timeliness and other FCS requirements.
Other interesting approaches to providing software infrastructure for control systems include Cactus 11 and Giotto 12. Control frameworks such as AOCS 13, Realtime
Framework 14, and Nexus 15 have interesting approaches to specific classes of systems.
Bonnie S. Heck, Linda M. Wills, and George J. Vachtsevanos 16 give a good overview of
The element pattern
As this brief survey shows, middleware is one of the predominant architectural models that have been applied to FCSs. Middleware nicely separates infrastructure from the rest of the software. However, designing, implementing, and operating software components on the infrastructure layer require a good service model. Figure 3a presents one
candidate model, the element abstraction, that models software components in an FCS.
Figure 3. Element service model: (a) element abstraction in an FCS; (b) elements in an FCS; and (c) the element pattern
An element encapsulates a control algorithm operating on some subsystem in the FCS. Elements participate in control hierarchies. For example, in Figure 2, if car control were a representative element, the trajectory server would be a higher-level element, and the car actuator would be a lower-level element. Elements also participate in data flows. In Figure 2, the data server is an input element to car control.
Not all elements need all the connections shown in Figure 3a, however. Figure 3b, for example, shows connections that would exist for sensors, actuators, and controllers in an FCS. The element abstraction can be represented in many ways. We use a design pattern 17. As Figure 3c shows, the element pattern is essentially a composition of
design patterns 17:
l A strategy represents the control law implemented by the element.
l A memento corresponds to the concept of state in a control law.
l The proxy operates as the conduit for communicating with other system elements.
l The composite pattern denotes the idea that multiple elements can be handled as a
collection.
Strategy and memento represent the service model for elements in an FCS. Proxy is an element’s interface to the rest of the system and is analogous to CORBA’s skeletons and stubs 4. The composite pattern represents the infrastructure’s view of the set of
elements in the system.
We model connections to other elements as streams of asynchronous events. This represents the typical case, in which a left element or sequence of control points from an upper element periodically updates an element. Synchronous communication, such as a handshake, can occur as connections are established between elements. A pair of asynchronous event streams hidden by the proxy can represent this communication. Further details of the element abstraction and other aspects of element behavior are available elsewhere. 18
We conclude by briefly considering how this architecture addresses some of the issues raised previously. Because elements have addressable profiles that identify and address them, elements are location independent. They must also have a well-defined state (communicated as mementos 17) that can be used to pause and restart them.
Downloading code and initial state from one central location permits bootstrapping and update of elements. The middleware architecture also clearly separates application code from the infrastructure.
Element-based architecture
One way to provide infrastructure for FCS elements is through a simple architecture that directly supports elements as primitive components. Figure 4 shows an architecture for element-based systems. Each element communicates with its shell, which represents the element to the system, and vice versa. The shell also provides a convenient mechanism for maintaining element-specific data structures and modifications for additional functionality (for example, adaptation for synchronous communication). A manager maintains and operates on the elements as a collection (the composite pattern 17) of similar entities. Because each element will potentially operate in its own
thread, the manager schedules the threads and supports various policies. Similarly, it schedules the events flowing between elements. The events are maintained in an event board, which the element shells and the rest of the system access uniformly.
Figure 4. An architecture for supporting the element abstraction.
In addition to the manager, the element-based architecture provides three sets of utilities:
l Reactive utilities are passive components that are activated by appropriate events
requesting their services. For example, a reflector service consumes the reflection
19 events generated by the elements and exposes the system infrastructure as
metadata.
l Proactive utilities are active components that both generate and consume events.
One such component is the global event dispatcher, which forwards events generated by the local elements to remote elements, and vice versa.
l Optimization utilities are components that do not participate directly in
event-processing activities. Instead, they observe other components (through notification mechanisms, for example) and perform optimizations as appropriate.
Previous work considers the rationale, functions, and responsibilities of the architectural components in more detail. 18
The notions of elements and element architectures have been further developed to EtherArch—a software architecture for FCSs. This work is under review for publication.
Acknowledgments
Research under contract number F49620-02-1-0217, DARPA under contract numbers F33615-01-C-1905 and N00014-01-1-0576, the US Army Research Office under contract numbers DAAD19-00-1-0466 and DAAD19-01010-465, DARPA/AFOSR under contract number F49620-02-1-0325, and the US National Science Foundation under contract number NSF ANI 02-21357. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the above agencies.
References
1. J.S. Bayne, “Automation and Control in Large-Scale Interactive Systems,” Proc. 5th IEEE Int’l Symp. Object-Oriented Real-Time Distributed Computing (ISORC), IEEE CS Press, 2002, pp. 3–12.
2. R.S. Pressman, Software Engineering: A Practitioner’s Approach, 5th ed., McGraw-Hill, 2001.
3. A. Giridhar, Scheduling Traffic on a Road Network, master’s thesis, Dept. of Electrical and Computer Eng, Univ. of Illinois at Urbana-Champaign, 2002.
4. Common Object Request Broker Architecture: Core Specification, Object Management Group, Nov. 2002; http://www.omg.org/cgi-bin/doc?formal/02-11-01.
5. DCOM Technical Overview, Microsoft Corp., Nov. 1996; http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dndcom/html/msdn_dcomtec.asp.
6. K.W. Edwards, Core JINI, Prentice Hall, 1999.
7. L. Wills et al., “An Open Control Platform for Reconfigurable, Distributed, Hierarchical Control Systems,” Proc. Digital Avionics Systems Conference, vol. 1, IEEE Standards Office, 2000, pp. 4D2/1–4D2/8.
8. A. Banks, “WebSphereMQ: Providing More Processing Power through Clustered Queue Managers,” white paper, IBM, July 2002,
http://www-3.ibm.com/software/integration/mqfamily/library/whitepapers/g325-2032-00.pdf. 9. K. Haase, “Java Message Service API Tutorial,” Sun Microsystems, 2002;
http://java.sun.com/products/jms/tutorial/.
10. P. Houston, “Building Distributed Applications with Message Queuing Middleware,” Microsoft Corp., 1998,
11. M. Hiltunen and R. Schlichting, “The Cactus Approach to Building Configurable Middleware Services,” Proc. Workshop Dependable System Middleware and Group Comm. (DSMGC), IEEE CS Press, 2000.
12. T. Henzinger, B. Horowitz, and C. Kirsch, “Giotto: A Time-Triggered Language for Embedded Programming,” Proc. 1st Int’l Workshop Embedded Software (EMSOFT), Springer-Verlag, 2001, pp. 46–60.
13. T. Brown et al., “A Reusable and Platform-Independent Framework for Distributed Control systems,” Proc. IEEE/AIAA 20th Digital Avionics System Conf. (DASC), vol.2, IEEE Standards Office, 2001, pp. 6A1/1–6A1/11.
14. A. Traub and R. Schraft, “An Object-Oriented Real-Time Framework for Distributed Control Systems,” Proc. IEEE Conf. Robotics and Automation, vol. 4, IEEE CS Press, 1999, pp. 3115–3121.
15. J. Fernandez and J. Gonzalez, “Nexus: A Flexible, Efficient and Robust Framework for Integrating Software Components of a Robotic System,” Proc. IEEE Conf. Robotics and Automation, vol. 1, IEEE CS Press, 1998, pp. 524–529.
16. B.S. Heck, L.M. Wills, and G.J. Vachtsevanos, “Software Technology for
Implementing Reusable, Distributed Control Systems,” IEEE Control Systems Magazine, vol. 23, no. 1, Feb. 2003, pp. 21–35.
17. E. Gamma et al., Design Patterns, Addison-Wesley, 1995.
18. G. Baliga, A Software Architecture for Federated Control Systems, master’s thesis, Dept. of Computer Science, Univ. of Illinois at Urbana-Champaign, 2002.
19. F. Kon et al., “The Case for Reflective Middleware,” Comm. ACM, vol. 45, no. 6, June 2002, pp. 33–38.
Girish Baliga is pursuing a PhD in the Department of Computer Science at the University of Illinois at Urbana-Champaign. His research interests include software architecture and federated control systems. Contact him at Coordinated Science Laboratory, Univ. of Illinois, Urbana-Champagne, 1308 W. Main St., Urbana, IL 61801; [email protected]; http://black.csl.uiuc.edu/%20~baliga/.
P.R. Kumar is the Franklin W. Woeltge Professor of Electrical and Computer Engineering at the University of Illinois at Urbana-Champaign. He was the recipient of the Donald P. Eckman Award of the American Automatic Control Council, and is a Fellow of the IEEE. Contact him at [email protected].