CFI is applied to the system model introduced in Chapter 4. While this model makes a couple of assumptions regarding the nature of services, it is not restricted to just one concrete type of environment. In particular, it includes distributed or non-distributed system environments as discussed in this section. By characterizing basic properties of these environments we can precisely identify those types of failures within these environments that are covered by CFI, which we will discuss after supported system environments and their basic properties have been described.
5.2.1
System Environments
Following distributed systems theory, we characterize concrete system environments in terms of three basic abstractions: different types of processes that run on different types of computing machines and messages exchanged among processes that rely on links that enable exchange of messages. Notably, there are three different types of processes:
• Sub process
• Service instance process • Server process
First, a sub process reflects the execution of the implementation of an operation. Therefore, one can say that the process of a service instance Sc• involves n = |Sc0.U |
sub processes (n ≥ 1) where Sc0 is the unfolded service of Sc. Clearly, a sub process is activated with invocation and ends upon completion. Both the invocation and the sub process itself may fail as detailed below.
Second, a server is a computing machine at which sub processes run. Obviously, a server requires software that provides features to activate a sub process on invocation and to further manage its lifecycle. The execution of this software is reflected by a pos- sibly infinitely running server process. A server may run any number of sub processes in parallel, virtually limited only by its technical computing resources. An execution en- gine is the software that activates and manages the process of a service instance (besides taking care on the actual execution task). Again, the execution of an engine itself is re- flected by a possibly infinitely running process. For now it is not important whether one engine manages the process of a service instance exclusively or whether it cooper- ates with other execution engines (i.e., distributed service instance process). Analogous to a server, an execution engine may manage multiple processes of different service instances in parallel. Furthermore, an execution engine may run on a server or at a separate (and dedicated) computing machine.
Obviously, this setup results in a distributed system environment as soon as sub processes and processes of service instances run at different computing machines and
implies communication between them, which is assumed to be done by means of mes- sage exchange.
However, it should be noted that the applicability of CFI is independent of whether a concrete system environment is distributed or not. The reason is that the types of runtime failures that are considered in this work apply to both distributed and non- distributed environments. On the distributed end, this includes partially synchronous and synchronous system environments.1 Since the step to real-time systems is “only” of a temporal nature, CFI would in principle also be applicable there, provided that appropriate means of ensuring timing constraints exist, which is, however, an aspect not addressed in this thesis. The main properties that we require to be provided by any concrete system environment are captured by the following assumption:
(A7) There is an (eventually) perfect failure detector. Messaging among (sub) pro- cesses is reliable.
These two properties are reflected in the so-called fail-stop model [SS83] for synchronous distributed environments and its variant the fail-noisy model [CGL11a, Chapter 2] for partially synchronous distributed environments. The basic property shared by both models is that processes2 may crash (i.e., halt prematurely). Moreover, processes can communicate with each other through point-to-point message-passing. The difference between the fail-stop and the fail-noisy abstraction lies in the accuracy of failure detec- tors and is inherent in the synchronous versus partially synchronous nature. Accuracy, in short, describes quality-of-service restrictions on the mistakes that a failure detector can make. In summary, the properties of these two models are as follows:
• A process that is not crashed follows its specification. It is called correct if it never crashes; otherwise it is called faulty.3
• Crashed processes do not perform anything. However, in the system environ- ments that we consider they may recover as detailed inSection 5.2.2.
1Recap, an asynchronous distributed system – which is not to be confused with asynchronous versus synchronous message sending as this refers to non-blocking versus blocking senders [BA06, Chapter 8] – is characterized by the following two properties:
• relative processor speeds and message transmission times are unbounded, which is introduced in practice when best-effort computing machines and networks are subject to by unpredictable loads; • (sub) process’s local clocks are not synchronized (i.e., there may be arbitrary drifts) since there is
no access to a global synchronized clock.
In contrast, a synchronous distributed system is characterized by the assumption that processing, communication delays and clock drifts have an upper bound that is known a priori. Partial syn- chrony [DLS88] captures the cases where either fixed bounds exist but are not known a priori or where the bounds are known but only hold after some unknown time. Finally, in a real-time system, upper bounds are not only known a priori but subject to strict constraints in the sense of deadlines that are to be met. For more information we refer to general text books such as [BA06, CGL11a].
2For the sake of convenience, we do not distinguish between sub processes, server processes, and processes of service instances here and just speak of processes.
3The formal definition of correct and faulty in [CT91] relates these terms to a run, which is understood as an infinite execution of a system.
• Every crashed process is (eventually) detected by each correct process based on a failure detector, which is accessible to each process (Completeness).
• Every correct process is (eventually) not erroneously suspected of having crashed by any correct process (Accuracy).
• A message sent to a correct process is eventually delivered. No message is deliv- ered more than once (i.e., no duplicates). No message is delivered without having been sent (i.e., no creation).
• Messages sent between the same processes are delivered in the order sent (FIFO). The feasibility of an (eventually) perfect failure detector necessitates that processes are not arbitrarily slow and latencies between message sending and delivery are not arbi- trarily long. Finiteness of bounds is therefore indeed crucial since it would otherwise be impossible to distinguish a correct process from a crashed process [FLP85, DDS87]. An eventually perfect failure detector may erroneously suspect a correct process of having crashed, but there is a finite while unbounded time after which it eventually accurately detects a correct process as correct (i.e., it may make mistakes). It is well known that eventually perfect failure detectors can be implemented either using timeouts [CT91] or heartbeats [KACT97]; the latter being advantageous over the former since it does not rely on timeouts and is quiescent (i.e., eventually it stops sending messages). In contrast, a perfect failure detector is devoid of mistakes and detections are permanent (i.e., once a crashed process has been detected, a perfect failure detector will not change its mind). Finally, it should be evident that “applications that have timing constraints require failure detectors that provide a quality of service with some quantitative timeli- ness guarantees”, as addressed in [CTA02].
5.2.2
Failure types
CFI as a method for failure handling aims at covering runtime failures that can be fur- ther classified as either
1. invocation failures or 2. execution failures.
Such failures can occur for many reasons. Apart from software and hardware design errors made by humans, the main reasons are the following:
• Hardware systems are subject to various phenomena of a stochastic nature that may suddenly disrupt regular operation.
• Remote distribution of resources and the possibly large number of resources makes environments only partially observable. It is often impossible to have com- plete knowledge about the current state of all available resources.
• Even if one would have complete knowledge about the environment at some in- stant in time, it may be subject to arbitrary changes that cannot be anticipated beforehand.
Note here that detection and handling of Byzantine failures as first discussed in [LSP82] in the context of distributed systems is beyond the scope of this thesis. We see means to detect, mask, or protect systems from this class of failures – the importance of which are beyond question – as a separate matter. It is therefore assumed that systems do not expose malicious behavior, which is, in fact, implied by the correct versus faulty property stated earlier on the fail-stop model.
It should also be mentioned that we expect the crash of an execution engine to be transient; that is, processes of service instances that were active and affected by the crash, meaning that they have crashed as well, will eventually recover and resume. Con- sequently, implementations of execution engines need to provide appropriate means based on stable storage for correct recovery.
Invocation Failures
There are mainly the following reasons for invocation failures:
1. A server that hosts implementations of operations is (temporarily) unavailable. This can be due to (i) network partitioning (infrastructure related failure), (ii) be- cause the server is down (e.g., for maintenance purposes), or (iii) because it has crashed (local hardware or software fault).
2. The profile or implementation (grounding) of an operation has been changed in a backwards-incompatible way by its provider and some service specification that makes use of it still assumes the meanwhile outdated version; hence, a malformed invocation request is created on its execution (incompatibility fault).
Characteristic for an invocation failure is that a sub process was not activated; hence, no further assumptions on the failure behavior of the sub process need to be made. An invocation failure is detected by an execution engine based on a timeout in case of unavailability of a server, which means that guaranteed eventual delivery as ensured by reliable messaging is explicitly disabled in this case. The second type of invocation failure is detected by an execution engine based on an error reply message sent be the server. Finally, it should be clear that erroneously suspecting a server of having crashed by an eventually perfect failure detector increases the amount of invocation failures.
There is, however yet another type of runtime situation that can be considered an invocation failure though occurring ultimately before invocation: the case of unsat- isfied preconditions. Given a transition t and an execution state s = (M,K) such that t is token-enabled in s, the operation Op = f u(t) is obviously not invokable if
fchk(K, Op.Pr.P) =false.
Execution Failures
Contrary to an invocation failure, an execution failure happens after an invocation was successful; that is, after a sub process has been activated and where the sub process is subject to a failure itself. In addition, we also subsume application-level execution failures under this category. By this we mean a prematurely ending but not crashing sub process whose functionality cannot be performed completely due to an unexpected
application-level constellation that prevents this. As an example, imagine an order op- eration within the order & pay service of the book seller scenario fromSection 2.1that is successfully invoked but nevertheless fails because a book that the customer wants to order is out of stock. One might counterargue that such application level cases should all be modeled as preconditions so that precondition checking would already catch them. However, it is often impractical even infeasible to model all conditions required for successfully performing an operation (or service). In fact, this refers to the qualifica- tion problem – the insoluble dilemma we are faced with when trying to fully enumerate all requirements that may otherwise prevent successful use, as “anyone will still be able to think of additional requirements not yet stated” [McC90].
For execution failures, we need to further detail the fail-stop behavior of sub pro- cesses. As stated in Section 5.2.1, a crashed process in the fail-stop and the fail-noisy model does not perform anything. However, it is not clear yet what this means regard- ing outputs and effects. The precise understanding of fail-stop is made explicit by the following assumption.
(A8)Given a sub process sp that reflects execution of an operation Op, none of Op’s outputs O and effects E materialize if sp fails permanently, nor will any other erroneous side-effect be made permanent in the underlying sub system. If the functionality and/or effects of Op makes it infeasible to ensure this property then a failure must be transient.
A permanent failure means that sp does not recover. This does not exclude recovery of the server at which sp ran in case the failure (crash) of sp coincided with a crash of the server. A transient failure of sp means that sp recovers transparently and resumes execution with a sufficiently small delay. In fact, transient failures are not directly rel- evant for CFI since it is reasonable to assume that sp will eventually complete in any case, even if it was subject to repeated crashes.
Not materializing any effect, output, nor any other side-effect for a permanent failure essentially necessitates the fail-safe or fail-fast property. The transactional approach is one means to assure that implementations of operations are fail-safe: already created effects are undone in the presence of a failure (rollback). Fail-fast basically means that a process stops normal operation already before entering a flawed state and immediately reports an error. In both cases we assume that some form of an error message is reported back to the execution engine.