This section describes the general FT framework implementation, which consists of patterns and mechanisms used in all FT strategy development.
5.4.1 Timing behavior
The MiddlewareScheduler (MS) thread runs at the beginning of every clock tick interval (e.g. 1ms; see Section 4.2.1 ) and controls the behavior and execution of each FT thread and voter. Besides, this thread is also responsible for activating the middleware thread that delivers external incoming messages.
Figure 5.5 shows an example of the execution of a Recovery Blocks (RB) strategy. The MS thread runs periodically and releases the message reception thread each two activation periods. The message reception thread is not executed in every cycle in order to reduce CPU utilization and to provide at least one cycle in two for FT threads free execution. In the first cycle, the FTThread receives a message and starts the FT execution. This example shows a failure in the primary block and a success in the recovery block.
Figure 5.5: RB execution timing example.
Figure 5.6 contains an activity diagram that shows the interaction between the
FTThread and the MiddlewareScheduler thread in the execution of the RB strategy.
After being activated, an FT thread sets up a deadline for execution, based on the actual time and the maximum allowed response time, the thread suspends. In subsequent MS activations, this thread restarts the FTThread if the deadline has expired. This situation represents a failure in delivering the correct response on time, but after restarting the FTThread is ready to receive the next request. If the deadline has not expired, the MS thread commands the next actions to be performed by the
FTThread thread and schedules it for execution. After executing the right operations
(save/restore state, run primary/recovery block, run acceptance test) the RB thread suspends again and the MS thread checks the acceptance test (AT) result. If the
FTThread succeeds in the AT, the MS thread allows it to send its results and the
interaction finishes. If the FTThread fails in both blocks, it is restarted by the MS thread.
receives message or wakes up
start
sets deadline and suspends
commands save state and acceptance test. Resumes thread saves state, runs primary block, runs acceptance test and
suspends
Passed AT
sends result
commands send result and resumes
thread
commands restore state and acceptance test. Resumes thread
restores state, runs recovery block and acceptance test and
suspends Final Final Passed AT commands send result. Resumes thread sends result Final Final Deadline Deadline restart thread Deadline restarts thread Final FTThread (RB) MiddlewareScheduler [Yes] [No] [No] [Yes] [No] [No] [Yes] [No] [Yes] [Yes]
Figure 5.6: RB execution activity diagram.
5.4.2 Class structure
Any FT strategy is executed in the context of two separate threads: the FTthread and the MiddlewareScheduler thread. The FTThread executes methods in response to the control algorithm performed by the MiddlewareScheduler thread. However, all the code related to a given FT strategy is defined by its FTStrategy concrete class. Figure 5.7 shows a class diagram describing the main methods involved in the execution of an FT strategy. MiddlewareScheduler + sendMiddlewareMsg(...) : void FTThread + ftStrategy: FTStrategy* FTStrategy + ftThread: FTThread* + executeFT() : void + executeMSControl() : void + processMiddlewareMsg(...) : void + startPeriodicMsg() : void Thread + myFTType: FTType + isRunningFT: bool + isVoting: bool «enumeration» FTType + NONE: int + FT: int + VOTER: int VoterThread
+ executeVoting(unsigned short) : void + checkTimeoutVoting() : void + processMiddlewareMsg(...) : void + startPeriodicMsg() : void
Figure 5.7: FT strategy execution class diagram.
Every FTStrategy subclass must implement the executeFT method, which performs the FT control algorithm that runs in the context of the FT thread (upper part of Figure 5.6, excluding message reception). It must also implement the
executeMSControl method, which performs the MS control algorithm for that stategy
(botton part of Figure 5.6). Using this approach, the MiddlewareScheduler class does not depend on any FT strategy implementation, and FT strategies can be added to the framework transparently.
The MS thread controls the execution of the voter threads in a similar way. However, the MS control is simpler, as it only have to detect if the voting deadline has elapsed. The executeVoting method is executed by the VoterThread, while the
checkTimeoutVoting method is called by the MS thread.
In contrast with the RB strategy presented so far, other FT strategies involve the utilization of multiples instances of the FT thread, running in different nodes. These FT threads have to communicate in order to coordinate, establish roles and initialize states. In this framework, the required communication between FT threads is executed by message passing between the MiddlewareScheduler threads of each node. If an FT thread needs to send a message, it calls the sendMiddlewareMessage method of MS. The sending message is broadcasted to all other nodes and their MS threads will distribute it to the related FT threads in their nodes, if any, by calling the
processMiddlewareMessage method of the corresponding FT strategy. The same
applies to VoterThreads that can communicate using the same methods described above.
Another feature performed by MiddlewareScheduler is the activation of the
startPeriodicMsg of FTStrategy and VoterThreads periodically (e.g. 300ms), in order
to trigger the execution of periodic tasks as, for instance, role conflicts detection in the DRB strategy.
Finally, the MiddlewareScheduler thread is responsible for changing the FT threads priorities according to the Earliest Deadline First (EDF) scheduling. Therefore, in each MS activation the FT thread with earliest deadline is found and its priority is raised to a maximum among application threads. This feature can be enabled or disabled in the framework.
Figure 5.8 contains a sequence diagram that describes the activities performed by
MiddlewareScheduler each time it runs. First it reads messages coming from other MiddlewareScheduler objects in other nodes. These messages are sent by external
FTThreads and VoterThreads and must be delivered to internal objects of the same
type and name, if any. Therefore, MiddlewareScheduler checks if there is any local thread with the name received in the incoming message. If there is, it determines if it is an FT thread or voter, based on the myFTType attribute of the Thread class, and calls the processMiddlewareMsg method of the related class (FTStrategy or
VoterThread). Next, if a predefined number of MiddlewareScheduler activations have
been executed; periodic messages of FT threads and voters are triggered, using the
startPeriodicMsg method. Finally, the control algorithm of active FT threads is
performed by running the executeMSControl method. Similarly, MS checks the timeout for active voting threads by calling the checkTimeoutVoting method. Active FT threads and voters are represented by a true value in the isRunningFT and isVoting Boolean attributes of the Thread class.
:MiddlewareScheduler :FTStrategy
[myFTType==FT]: processMiddlewareMsg
:VoterThread
[myFTType==VOTER]: processMiddlewareMsg reads message & finds thread by name
While there are new messages * [myFTType==FT]: startPeriodicMsg * [myFTType=VOTER]: startPeriodMsg Each n activations * [isRunningFT]: executeMSControl * [isVoting]: checkTimeoutVoting
Figure 5.8: MiddewareScheduler thread sequence diagram.
Figure 5.8 does not represent FT thread scheduling, but this operation is performed, if selected, at the end of each MS activation.