ORB Middleware for Networked Embedded Systems: A Case Study

In this section, we describe the design and implementation of nORB, and the rationale behind our approach, to address the networked embedded system design and implementation challenges described in Section ...

2.3.1 Message Formats

We support a limited subset of the messages supported by the CORBA specification, so that we do not incur unnecessary footprint, but at the same time support the minimum features required by the application. The following messages are supported by nORB: Request, Reply, Locate Request, and Locate Reply.

Figure . shows the formats of the messages supported by nORB. The format of the Request and Reply messages in nORB closely resembles that of the GIOP Request and Reply messages, respec-tively. We use the Common Data Representation [] to encode the messages themselves. The nORB client builds a Request message and sends it to the nORB server, which sends a Reply back to the client.

Richard Zurawski/Networked Embedded Systems K_C Finals Page  -- #

Middleware Design and Implementation for Networked Embedded Systems 2-9

Request Id

Two-way flag

Op Name

ObjectKey

Length ObjectKey Priority Parameters

Request Id

nORB Request message format

nORB Reply message format

Status Results

nORB Locate Request message format

Locate request Id Corbaloc style Key

nORB Locate Reply message format Locate Reply Id IOR string

nORB IOR format

Repository Id ObjectKey Transport

address Priority

Profile-1 Profile-n

FIGURE . nORB IOR and message formats.

2.3.2 Object Adapter

In standard CORBA, each server-side ORB may provide multiple object adapters []. Servant objects register with an object adapter, which demultiplexes each client request to the appropriate servant.

Each object adapter may be associated with a set of policies, e.g., for servant threading, retention, and lifespan []. In standard CORBA, multiple object adapters are supported by each ORB. This allows heterogeneous object policies to be implemented in a client-server environment, which is desirable in applications such as online banking, where each object on a server may be configured according to preferences of the server administrator, or even the end user.

In nORB, however, there is no assumption of multiple object adapters. Instead, a single object adapter per ORB is considered preferable for simplicity and footprint reduction. In nORB, the num-ber of objects hosted on an embedded node is expected to be small, which reduces the need for multiple policies and thus for multiple object adapters. Even though the resulting object adapter does not conform to the Portable Object Adapter specification, a significant degree of footprint reduction is achieved because of the reduced object adapter functionality.

We have also simplified the process of object registration, to free developers from writing repetitive code as is seen in many CORBA programs. In the object adapter, we maintain a lookup table of object IDs and pointers to servant implementation objects. The lookup table is synchronized using a Read-ers/Writer lock. We have also consolidated object registration with other middleware initialization functions, by moving it from the object adapter interface to the ORB interface.

2.3.3 Message Flow Architecture

Themessage flow architecture in nORB uses strategies and patterns similar to those in TAO. We briefly mention the strategies we used and refer the interested reader to Ref. [], which discusses these strategies in detail.

Richard Zurawski/Networked Embedded Systems K_C Finals Page  -- #

2-10 Networked Embedded Systems

2.3.3.1 Reply Wait Strategy

When a client makes a remote two-way function call, the caller thread needs to block until it receives a reply back from the server. The two-way function call is made on the client stub, which then marshals the parameters into a Request and sends it to the server. The two-way function call semantics requires the caller thread to block until the reply comes back from the server. There are two different strategies to wait for the reply in TAO—“Wait on Connection” and “Wait on Reactor” []. nORB uses the Wait on Connection strategy to wait for the reply.

2.3.3.2 Upcall Dispatch Strategy

On the server side, there are different strategies to process an incoming request and send the reply back to the client. Two well-known strategies are “Direct Upcall” and “Queued Upcall” []. In the Direct Upcall strategy, the upcall is dispatched in the same thread as the I/O thread, which listens for incoming requests from the connection stream. The Queued Upcall strategy follows the Half Sync Half Async pattern [,]. In contrast with the direct upcall strategy, a network I/O thread is dedicated to receiving requests from clients. Once the request is received, it is encapsulated into a command object and then put into a queue. This queue is shared with another thread, in which the upcall dispatch is done. TAO uses the Direct Upcall strategy, whereas nORB uses the Queued Upcall strategy to address the simulation issues presented in Section .., as is discussed in Section ...

2.3.4 Time-Triggered Dispatching

In the Active Damage Detection application described in Section .., control events must be trig-gered predictably at various rates. We developed a dispatcher for nORB to trigger these events predictably, based on the Kokyu [] dispatching model. Kokyu abstracts combinations of funda-mental real-time scheduling and dispatching mechanisms to enforce a variety of real-time policies, including well-known strategies such as Rate Monotonic Scheduling (RMS) [], Earliest Deadline First [], and Maximum Urgency First [].

Thisdispatcher is used to trigger events on the client side and for dispatching upcalls on the server side. A “Dispatchable” interface is provided, which is to be implemented by an application object that needs to be time-triggered. The “handle_dispatch()” method is called on the Dispatchable object. In the ping-scheduling application, some application objects are both Dispatchable and make remote calls when triggered. Figure . shows the message flow from the client to the server when a timer expires on the client side, leading to a remote object method call on the server side.

nORB

Dispatcher Dispatcher

LO HI

SOA

Skeleton Skeleton

Servants

Stub Stub

Clients

FIGURE . Kokyu dispatcher in nORB.

Richard Zurawski/Networked Embedded Systems K_C Finals Page  -- #

Middleware Design and Implementation for Networked Embedded Systems 2-11

HI LO

2 KHz 100 Hz Client

Stub

HI LO

Skeleton

nORB

nORB Reactor port:10000

HI

Reactor port:20000

LO

RequestId = 23 ……. Priority = LO

ObjectKeyA ……. Priority = LO,hostA:20000 Priority = HI,hostA:10000 IOR

Server

Dispatcher

FIGURE . Priority propagation in nORB.

2.3.5 Priority Propagation

nORB implements real-time priority propagation in a way similar to TAO [,]. The client ORB uses the priority of the thread in which the remote invocation is made. nORB then looks for a matching priority from the set of profiles in the IOR and then makes a connection to the appropriate port. We use a cached connection strategy [] to avoid the overhead of connection setup every time a remote invocation is made. To alleviate priority inversion, each connection endpoint on the server is associated with a thread/reactor pair, thus forming a “dispatching lane.” The priority of the thread associated with each dispatching lane is set appropriately so that a request coming into a higher pri-ority lane will be processed before a request coming into a lower pripri-ority lane. Figure . shows an example in which RMS [] was used to assign priorities to the different rates on the client side.

The priority of the client thread making the request is propagated in the request. This priority is used on the server side to enqueue the request if necessary as explained in Section ...

2.3.6 Simulation Support

Our solution to the engineering life-cycle issues described in Section .. is to have nORB maintain a logical clock [,] at each node to support synchronization among nodes. We distinguish between

“logical” times and priorities that are maintained by the simulation, and “actual” times and priorities that are maintained by the operating system. Logical time is divided into discrete frames. Each logical clock is incremented in discrete units by nORB after any incoming message is processed. If there are no incoming messages to be processed, then the logical time is incremented to the start of the next frame.

At the start of each frame, each node registered for a timed upcall in that frame is triggered, which results in the node waiting for sensor values from a physical simulation tool like Matlab or Simulink.

While a node is waiting for the sensor values, we “freeze” the logical clock on that node, i.e., we

Richard Zurawski/Networked Embedded Systems K_C Finals Page  -- #

2-12 Networked Embedded Systems

prevent the logical clock from advancing. As a result, no messages are processed on that node while it is waiting, and all incoming messages are queued. The queuing of incoming messages allows messages from the future to be delayed, and then processed after a slower node “catches up.”

Any request or reply leaving a node carries the current logical clock value of that node. This value is used by the receiving node to sort the incoming messages based on their time of release and the logical clock of the receiving node. If the logical clock of the receiving node is later than that of the incoming message, then the message is stamped with the value of the logical clock at the receiving node.

Each “item” (dispatchable or request/reply message) carries its logical execution time, which is predefined for each item. When an item is ready and most eligible for execution, the clock thread dequeues it and checks whether it can complete its execution before the earliest more eligible item’s release time. If it can complete its execution before another more eligible item must be processed, the clock thread enqueues the current item in the appropriate “lane” for “actual” execution on the pro-cessor. If not, the clock thread “simulates” the partial execution of the current item without “actually”

executing it, by () storing the remaining logical execution time in the item itself and () enqueuing the updated item back into the clock thread’s queue so that it can compete with other enqueued items for its next segment of execution eligibility.

A lane can be configured to run a single thread or a pool of worker threads. As described in Section .., without clock simulation each lane thread is run at its own actual OS priority. In the simulation environment, time and eligibility are accounted for by the logical clock thread, so all the lane threads are run at the same actual OS priority. Each lane still maintains its logical priority in thread-specific storage [], so that the logical priority can be sent with the request messages and used for eligibility ordering, as it will be in the target system.

Figure . shows an illustrative sequence of events that results from the addition of the logical clock thread to nORB, assuming for simplicity that all items run to completion rather than being preempted.

. When the logical clock advances, the clock thread goes through the list of dispatchables to see whether any are ready to be triggered. A “ready” dispatchable is one whose next trigger time is less than or equal to the current logical clock and whose previous invocation has completed execution. In general, the clock thread determines the earliest time a message or dispatchable will be run, and marks all items with that time as being ready.

. Any ready dispatchables are released to the clock thread’s queues, according to their assigned logical priorities.

. The clock thread selects the most eligible ready item (message or dispatchable) from among its priority queues. The clock thread then enqueues the selected item in the appro-priate priority lane of the dispatcher, where it will compete with other messages and locally released dispatchables.

. The corresponding lane thread in the dispatcher dispatches the enqueued item. The resulting upcall might in turn invoke a remote call to a servant object, which we describe in the following steps –.

. The logical priority of the dispatchable or the message is propagated to the server side.

Currently, the application scheduler uses RMS to decide the logical priority of the dis-patchable based on its simulated rate. Each lane thread stores its assigned logical priority in thread-specific storage []. The actual OS priorities of all the lane threads are kept the same under the clock simulation mechanism.

. An incoming message is accepted by the server’s reactor thread and is enqueued for tem-poral and eligibility ordering by the clock thread. Note that there is only one reactor thread, which runs at an actual priority level between the clock and the lane threads’ actual

Richard Zurawski/Networked Embedded Systems K_C Finals Page  -- #

Middleware Design and Implementation for Networked Embedded Systems 2-13

nORB Client

Stub

HI HI

Skeleton

Reactor port:10000

ME

RequestId = 23 ……. Logical priority = LO

ObjectKeyA ……. hostA:10000

IOR

Server

Logical clock LO nORB

HI HI

Dispatcher

Logical clock LO

100 Hz 2 KHz