The testing techniques

Our development process aims at gaining timing assurance of the software by using formal methods and systematic code generation approaches throughout the development cycles, ideally to reduce (or remove) the cost spent in testing. However, the models (e.g., the platform-independent and the platform-dependent model) that we create are anabstraction of an implementation by hiding some details of the implementation. In practice, such an abstraction is necessary in order to keep the model size manageable for formal verification due to several limitations that come from a chosen technique (e.g., scalability issues from the model checking). This makes testing necessary in concert with using formal methods.

There have been several attempts at using UPPAAL to test implementations. Larsen et al. [36] introduced a tool for online black-box testing for real-time embedded systems using UPPAAL specifications. In Larsen’s work, the system behavior (e.g., the PCA pump be- havior) is modeled using an UPPAAL automaton that is composed with another UPPAAL automaton that models the behavior of the environment (e.g.,the patients) with which the system will interact. This information is fed into a UPPAAL-TRON for checking input- output conformance of the system. This tool can be used in our work to automatically generate test cases and to execute the tests in order to systematically check the correct- ness of the integration between the platform-independent and the platform-dependent code. Hessel et al. [30, 29] also used UPPAAL to derive test suites specifying the expected behav- ior of the implemented systems based on structural coverage criteria. We expect that these systematic online testing approaches can reduce test costs compared to manual testing.

Gay et al. [25, 23] studied the testing problem of using a model as an oracle to test implemented systems. Gay’s work highlights the challenges of using a model as an oracle due to several aspects that are not properly abstracted in the model, but should be considered at the implementation level to perform more accurate testing such as input processing delays, execution time fluctuation, and hardware inaccuracy. Neglecting such gaps leads to excessive amounts of false positive testing results that need to be inspected further by adding

extra costs. The proposed solution is to adapt the model behavior to the implementation by comparing the state of the model to that of the implementation. In order to realize such an adaptation, Gay’s work introduces a notion of the tolerance constraints that defines allowable changes to certain variables of the model. Next, the dissimilarity function is used to compare the state of the model to the observable state of the System-Under-Test (SUT). Finally, using the SMT-based bounded model checking technique, the framework finds a solution to the constraint that gives a new model state that is more similar to the behavior of the SUT; as a result, the new model can be used as a better testing oracle.

In comparison to our work, Gay’s work addresses the similar issue in a sense that, in order to reason about the correctness of an implementation, there should be additional consideration about the impact of the timing gaps between the model and the implementation. One major difference is that our work focuses on constructing a new model (i.e., PSM) for model checking; on the other hand, Gay’s work creates a new model for testing. More specifically, our approach relies on abstracting the platform-specific portion in a composable way so that it can be integrated with the platform-independent model. The resultingPSM is eventually used to formally verify the correctness of the integration between the platform-independent and the platform-dependent code. On the other hand, Gay’s work gives more focus on transforming a model into another model through a steering process so that the resulting model behavior becomes closer to the implementation; the resulting model can then be used as a better testing oracle by reducing false positive testing results. We believe that these two works can complement each other to add more timing assurance to reason about the correctness of the composition of the platform-independent and the platform-dependent aspects of the implementations.

Mathwork introduces a concept of Model-in-the-Loop (MIL), Software-in-the-Loop (SIL) and Processor-in-the-Loop (PIL) testing [19] to validate the constructed artifacts from Simulink/Stateflow models (e.g., models, source code, and microprocessors running the source code) throughout the development cycles in a systematic way. This multi-layer

testing approach enables a system designer to develop a model of a system in the early de- velopment stage from which a test suite is automatically generated to perform MIL testing; this test suite is used to check whether the model conforms to the requirements. Next, SIL testing is performed to check whether the behavior of the source code generated from the model matches that of the model by elaborating the test suite that was used in MIL. After that, PIL testing enables the microprocessor that is running the source code to be checked for the requirement conformance by further elaborating the test suite.

Such a layered approach is useful in our work to test the behavior of three different artifacts: (1) the platform-independent model (using MIL) (2) the platform-independent code (using SIL) (3) the final implemented system that is composed of the platform-independent and the platform-dependent code (using PIL). However, this approach does not explicitly explain how a test suite used in MIL should be refined to perform the next level of testing (e.g.,SIL and PIL). Particularly, such a test suite refinement is quite challenging when the notion of input and output used in the test suite of MIL are not well matched to those used in SIL and PIL for some reasons. For example, such scenarios may include a single input (e.g., alarm condition) in MIL which is matched to two different inputs (e.g.,empty reservoir and occlusion condition) in SIL and PIL. Feeding an input (or observing an produced output) at SIL and PIL takes non-zero execution time as opposed to MIL which assumes zero delay for processing such input and output.