We are developing a prototype tool, called FBTT, which supports our testing approach.
From a user’s perspective, the tool is very helpful in guiding the test-design phase. FBTT Figure 7. EFSM of the testing model
Reading
allows creating statecharts, describing the results of FTA expressed in terms of duration calculus formulas, integrating statecharts and formulas, and slicing statecharts.
The tool was implemented in Delphi 5.0 and runs under Windows XP and Windows 2000.
We have concentrated on FBTT usability and scalability, as it is required for technology transfer. All the information related to a project is stored in Microsoft Access databases;
this will allow users to produce customized reports. However, the tool is coded in a way that it can easily switch to another database engine. We have adopted a layered application architecture. The logical architecture provides separation between graphical user interface (GUI) and process algorithms and database management. The domain processes are organized using a pipe and filters architecture. The layered architecture is shown in Figure 8 with each of the three layers — GUI, process, and database — consisting of one or more function blocks.
The FBTT tool has been used successfully on a number of small case studies. The algorithms used for integration are based on a data structure enabling the optimization of operations like searching.
FBTT supports the elaboration of fault-based testing models. However, when human assistance is needed during integration, it is sometimes difficult to see why. In the future we plan to extend the tool with a graphical interface that would allow the user to visualize the step-by-step results of the integration.
Conclusion
A review of the state of the art and practice in fault-based testing has been made. The capabilities and limitations of various strategies used have been presented and dis-Figure 8. FBTT architecture
GUI Layer
I_Statechart
I_CutSets
I_Syntax Tree
I_Slicing
Process Layer DataBase Layer
Statechart
CutSets
Syntax Tree
S l i c i n g Integration
FBTT DataBase
cussed. As a conclusion, we can say that the existing fault-based testing techniques are quite limited and our testing approach extends previous results on fault-based test case generation by including semantic errors and considering combinations of faults. If the fault-tree analysis is complete, then our testing approach assures that all conditions that enable a fault situation will show up as test cases.
Our approach can also be related to the concept of software testability. In our testing model, events that can contribute to a failure state may be hardware faults, software faults, or any other condition. An event representing a software fault indicates which faults can be injected to quantify the effects of probable hidden errors. Since injected faults arise from fault-tree analysis, we can focus on the faults related with our testing objective; and we do not assume single fault hypothesis, on the contrary, we consider combinations of faults.
It may be argued that a thorough understanding of the system is essential for the integration step. However, the analyst has to think about the system in great detail during tree construction. The most useful fault-trees require detailed knowledge of the design, construction and operation of the systems (Leveson, 1995).
The combination of fault-tree analysis and statecharts, poses another problem, such as the integration of heterogeneous specifications. We directed our efforts towards developing an approach that requires as little human intervention as possible. Most of the tasks involved — that is, the conversion of duration calculus formulas to statecharts and the slicing and generation of test sequences — can be automated.
One of the main problems of testing is the definition of an oracle. An oracle is a mechanism that specifies the expected outcome. In most testing proposals, the existence of an oracle is assumed. This is rather difficult to satisfy in practice since its creation is expensive and sometimes provides incorrect results. In our approach, since we do not aim to prove correctness, we do not need an oracle. Our objective is to demonstrate the absence of prespecified faults, and this is determined by observation if an undesirable state (given by the root node of a fault-tree) has been reached.
An assumption made in fault-based testing is the coupling-effect hypothesis. Research into the fault coupling effect demonstrated that test data sets that detect simple types of faults are sensitive enough to detect more complex types of faults (Offutt, 1992). The empirical investigations presented in Offutt consider mutation operators that describe syntactic changes on the programming language. Hence, the results cannot be inter-preted in the context of our work in which we consider semantic information.
The testing method has been demonstrated by an example on a simple Web application.
Most of the literature and tools on testing Web applications test nonfunctional aspects of the software (e.g., HTML validators, capture/playback tools, security-test tools, and load and stress tools). However, the different ways that pieces are connected in a Web application give rise to other problems. Andrews, Offutt, and Alexander (2005) categorize testing Web applications in terms of the type of connection: static links, dynamic links, user/time specific GUIs, operational transitions that the user introduces into the system outside of the control of the software, software connections among back-end software components, off-site software connections, and dynamic connections when Web components are installed during execution. The work of Andrews, et al. considers functional testing. They propose a system-level testing technique (regarded as FSMWeb)
that combines test generation based on finite state machines (FSMs) with constraints.
The approach builds hierarchies of FSMs that model subsystems of the Web applications and then generates sequences of actions labeled with parameters and constraints on parameters. The constraints are used to select a reduced set of inputs. The authors indicate that one limitation of the technique is that Web applications have low observability.
Some of the output is sent back to the user as HTML documents, but Web applications also change state on the server and the database and send messages to other Web applications and services. In our approach, we alleviate this problem by avoiding the use of an oracle.
Another limitation in the FSMWeb technique is that it has limited support for unantici-pated usercontrolled transitions (e.g., a user going directly to an internal Web page with a bookmark or use of the back button). The authors suggest modeling those transitions.
However, this significantly increases the number of transitions. This problem is ad-dressed in our approach by only generating test sequences that reach some prespecified undesirable states.
References
Aho, A., Sethi, R., & Ullman, J. (1986). Compilers: Principles, techniques and tools.
Reading, MA: Addison Wesley.
Andrews, A., Offutt, J., & Alexander, R. (2005). Testing Web applications by modeling with FSMs. Software Systems and Modeling, 4(2), 326-345.
Booch, G., Rumbauch, J., & Jacobson, I. (1998). The unified modeling language: User guide. Reading, MA: Addison Wesley Longman.
Budd, T., & Angluin, D. (1982). Two notions of correctness and their relation to testing.
Acta Informatica, 18(1), 31-45.
Chen, T. Y., Feng, J., & Tse, T. H. (2002). Metamorphic testing of programs on partial differential equations: A case study. In I. Sommersville (Ed.), Proceedings of the 26th Annual International Computer Software and Applications Conference (pp.
327-333). Los Alamitos, CA: IEEE Computer Society Press.
Chen, T. Y., Tse, T. H., & Zhou, Z. (2003). Fault-based testing without the need of oracles.
Information and Software Technology, 45(1), 1-9.
Denaro, G., Morasca, S., & Pezzè, M. (2002). Deriving models of software fault-prone-ness. In G. Tórtora & S. Chang (Eds.), Proceedings of the SEKE. Ischia, Italy.
Dugan, J., & Doyle, S. (1996). Incorporating imperfect coverage into a BDD solution of a combinatorial model. Journal of Automatic Control Production Systems, special issue on Binary Decision Diagrams for Reliability Analysis, 30(8), 1073-1086.
Foster, K. (1980). Error sensitive test cases analysis (ESTCA). IEEE Trans. on Software Eng., 6(3), 258-264.
Goodenough, J., &. Gerhart, S. (1975). Toward a theory of test data selection. IEEE Trans.
on Software Eng., 1(2), 156-173.
Hamlet, R. (1992). Are we testing for true reliability? IEEE Software, 9(4), 21-27.
Hamlet, R. (1994). Foundations of software testing: Dependability theory. In T. Ostrand (Ed.), The 2nd ACM SIGSOFT Symposium on Foundations of Software Engineering (pp. 128-139).
Hamlet, R., & Taylor, R. (1990). Partition testing does not inspire confidence. IEEE Trans.
on Software Eng., 16(12), 1402-1411.
Hamlet, R., & Voas, J. (1993). Faults on its sleeve: Amplifying software reliability testing.
In T. Ostrand & E. Weyuker (Eds.), Proceedings of the International Symposium on Software Testing and Analysis (pp. 89-98). USA: ACM Press.
Hansen, K. M., Ravn, A. P., & Stavridou, V. (1998). From safety analysis to software requirements. IEEE Trans. on Software Eng., 24(7), 573-584.
Hansen, M., & Zhou, C. (1992). Semantics and completeness of duration calculus. In J.
W. de Bakker, C. Hizing, W. de Roever, & G. Rozenberg (Eds.), Real-time: Theory in practice, REX Workshop (LNCS 600, pp. 209-225). The Netherlands: Springer-Verlag.
Harel, D. (1987). Statecharts: A visual formalism for complex systems. Science of Computer Programming, 8, 231-274.
Harel, D. (1996). The statemate semantics of statecharts. ACM Transactions on Software Engineering and Methodologies, 5(4), 293-333.
Heimdahl, M., & Whalen, M. (1997). Reduction and slicing of hierarchical state machines.
In R. Conradi (Eds.), Proceedings of the Internationall Conference on Founda-tions of Software Engineering (pp. 450-467). New York: Springer-Verlag, Inc.
Howden, W. (1976). Reliability of the path analysis testing strategy. IEEE Transactions on Software Engineering, 2(3), 208-214.
Kim, Y., Hong, H., Bae, D., & Cha, S. (1999). Test cases generation from UML state diagrams. IEE Proceedings: Software, 146(4), 187-192.
Leveson, N. G. (1995). Safeware: System safety and computers. Addison Wesley.
Morell, L. J. (1990). A theory of fault-based testing. IEEE Transacions on Software Engineering, 16(8), 844-857.
Offutt, J. (1992). Investigations of the software testing coupling effect. ACM Trans. on Software Engineering and Methodology, 1(1), 3-18.
Offutt, J., & Untch, R. (2000). Mutation 2000: Uniting the orthogonal. In Proceedings of Mutation 2000: Mutation Testing in the Twentieth and the Twenty First Centu-ries, San Jose, CA (pp. 45-55).
Podgurski, A., & Clarke, L. (1990). A formal model of program dependencies and its implications for software testing, debugging, and maintenance. IEEE Transactions on Software Engineering, 16(9), 965-979.
Richardson, D., & Thompson, M. (1993). An analysis of test data selection criteria using the RELAY model of fault detection. IEEE Trans. on Software Eng., 19(6), 533-553.
Sánchez, M., Augusto, J., & Felder, M. (2004). Fault-based testing of e-commerce applications. In J. Augusto & U. Ultes-Nitsche (Eds.), ICEIS2004 (pp. 12-16).
Porto, Portugal: INSTICC Press.
Sánchez, M., & Felder, M. (2001). Slicing of statecharts. In G. Fernandez & C. Pons (Eds.), ASSE2001 (pp.177-190). SADIO.
Sánchez, M., & Felder, M. (2003). A systematic approach to generate test cases based on faults. In V. Braberman & A. Mendarouzqueta (Eds.), Proceedings ASSE2003, Buenos Aires, Argentina.
Sloane, A., & Holdsworth, J. (1996). Beyond traditional program slicing. In Proceedings of the International Symposium on Software Testing and Analysis (pp. 180-186).
San Diego, CA: ACM Press.
Voas, J. (1992). PIE: A dynamic failure-based technique. IEEE Transactions on Software Engineering, 18(8), 717-727.
Voas, J., & Miller, K. (1995). Software testability: The new verification. IEEE Software, 12(3), 17-28.
Voas, J., Morell, L., & Miller, K. (1991). Predicting where faults can hide from testing. IEEE Software, 8(2), 41-48.
Weiser, M. (1984). Program slicing. IEEE Transactions on Software Engineering, 10(4), 352-357.
Weyuker, E.,& Ostrand, T. (1980). Theories of program testing and the application of revealing subdomains. IEEE Transactions on Software Engineering, 6(3), 236-246.
White, L., & Cohen, E. (1980). A domain strategy for computer program testing. IEEE Trans. on Software Eng., 6(3), 247-257.
Zhou, C., Hoare, C., & Ravn, A. (1991). A calculus of durations. Information Proc. Letters, 40(5), 269-276.