EFFECTIVE TEST CASE GENERATION
USING ANTIRANDOM SOFTWARE
TESTING
KULVINDER SINGH
University Institute of Engineering & Technology(UIET), Kurukshetra University,Kurukshetra-136118 Haryana(India)
http://<www.kuk.ac.in>
RAKESH KUMAR
Department of Computer Application(DCSA), Kurukshetra University,Kurukshetra-136118 Haryana(India)
http://<www.kuk.ac.in>
IQBAL KAUR
Department of Mathematics,
Govt. Post Graduate College, Sec-14, Karnal Haryana(India)
Abstract: Random Testing is a primary technique for the software testing. Antirandom Testing improves the
fault-detection capability of Random Testing by employing the location information of previously executed test cases. Antirandom testing selects test case such that it is as different as possible from all the previous executed test cases. The implementation is illustrated using basic examples. Moreover, compared with Random Testing, test cases generated in Antirandom Testing are more evenly spread across the input domain. Antirandom Testing has conventionally been applied to programs that have only numerical input types, because the distance between numerical inputs is readily measurable. The vast majority of research involves distance techniques for generating the antirandom test cases. Different from these techniques, we focus on the concrete values of program inputs by proposing a new method to generate the antirandom test cases. The proposed method enables Antirandom Testing to be applied to all kinds of programs regardless of their input types. Empirical studies are further conducted for comparison and evaluation of the effectiveness of these methods is also presented. Experimental results show that, compared with random and hamming distance techniques, the proposed method significantly reduces the number of test cases required to detect the first failure. Overall, proposed antirandom testing gives higher fault coverage than antirandom testing with hamming distance method, which gives higher fault coverage than pure random testing.
Keywords: software testing; antirandom testing; random testing, mutation testing,.
1. Introduction
Developed system should be reliable, safe, and secure. To achieve this objective a number of techniques are being used such as fault avoidance, fault tolerance, fault removal, and fault evasion etc. The purpose of software testing is to reveal software faults in order to avoid as runtime failures during program usage. A program’s execution is correct when its behavior matches the functional and non-functional requirements specified in specification of the software [1]. Apart from of the limitations, testing is a vital part in software development. Testing is usually performed for the following purposes: (a) to improve quality i.e. assuring conformance to the specified design requirement. Being correct is the minimum requirement of quality, (b) For Verification & Validation (V&V): Testing is used as a tool in the V&V process. Testers can make claims based on interpretations of the testing results whether the product works under certain situations or not. Hamlet and Morell have formally stated the goals of a software testing methodology and implicitly provided an understanding of the limitations of testing [2],[3]. Young and Taylor have also observed that due to presence of defects within a program, software-testing techniques must involve some tradeoff between accuracy and computational cost [4]. Software system can be verified by either software testing or by the process of assessing the functionality and correctness of a program through execution or analysis.
to perform testing in a more effective way at a lower cost. According to the Chen et al. [5] ,[6], Random Testing is a basic method for software testing and easy to implement, has been demonstrated to be effective in detecting failures, and is good at exercising systems when the source code and the specifications of the program under test are not available or incomplete, On the other hand, as the random testing does not use any available information for test case selection, it is often argued that random testing is inefficient. Moreover, for reducing the number of test cases required to detect a failure, Malaya [7] purposed an antirandom testing method, in which the first test case is selected randomly, and each following test case is selected by choosing the one whose total hamming distance to all the previously executed test cases is maximum. Antirandom testing has a limited degree of randomness: the first test case is selected randomly; whereas the sequence of all the following test cases is deterministic. In this research paper researcher use the antirandom test generation techniques.
2. Testing Disciplines
2.1. Specification based testing
The testing technique based on the specification of the software system depends upon the specification either written in the formal or informal language. Above, a comprehensive overview of software testing concepts, techniques, and processes has been presented. There are three general styles of specification [8]: Process algebras, which describe systems in terms of behavior and interaction of active agents; Algebraic specifications describe systems constructively in terms of applications of system operations. States are defined by construction from some defined basic structure, and passed as parameters; Model-based specifications construct explicit models of the system state and Show how the state can be changed by various operations. In specification-based testing, general form of the documentation is relative to program specifications. Early approaches looked at the Input/ Output relation of the program seen as a “black-box” and use the Equivalence classes partitioning, Boundary conditions, or Cause-effect graphs can be used to explore the systematic way to combine all the possible input conditions
2.2. Random Testing
Random Testing (RT) randomly selects test cases/sequences of events from the input domain [9],[10]. The advantages of RT include its low cost, ability to generate numerous test cases automatically, generation of test cases in the absence of the software specification and source code and apart from these; it brings randomness into the testing process. Such randomness’s can best reflect the chaos of system operational environment; as a result, RT can detect certain failures unable to be revealed by deterministic approaches. All these advantages make RT irreplaceable in industry for revealing software failures [11],[12],[[13],[14][15],[16],[17],[18],[19] This approach may yield a large number of event sequences that are not legal & hence not executable, wasting valuable resources. Moreover, the test designer has no control over choice of event sequences; they may not have acceptable test coverage. Random testing selects arbitrarily test data from the input domain & then these test data are applied to the program under test. The automatic production of random test data, drawn from uniform distribution, should be the default method by which other systems should be judged, [20]. The random generation of tests identifies members of the sub domains arbitrarily, with a homogeneous probability, which is related to the cardinality of the sub domains. Under these circumstances, the chances of testing a function, whose sub domain has a low cardinality with regard to the domain as a whole, is much reduced. A random number generator generates the test data with no use of feedback from previous tests. The tests are passed to the procedure under test, in the hope that all branches will be traversed [21].
2.3. Antirandom Testing
In Anti random testing the test cases should be selected to have maximum distance from each other. Parameters & interesting values of the test object are encoded using a binary vector such that each interesting value of every parameter is represented by one or more binary values. Test cases are then selected such that a new test case resides on maximum Hamming or Cartesian distance from the already selected test cases. Anti random testing can be used to select a subset of all possible test cases, while ensuring that they are as far apart as possible. Moreover, it has proved useful in a series of empirical evaluations. Unfortunately, this method essentially requires enumeration of the input space & computation of each input vector when used on an arbitrary set of existing test data. This avoids scale-up to large test sets and/or long input vectors.
2.4.Adaptive Random Testing
keeping test cases apart shall enhance the effectiveness of RT. Therefore, ART does not just randomly generate but also evenly spreads test cases or it generates fewer duplicate test cases. Different Studies [27],[28],[29],[30],[31]shows that ART can be very effective in detecting failures when there is a continuous failure region inside the input domain as compared to RT. Since ART is as simple as RT and preserves certain degree of randomness, ART could be an effective replacement of RT.
2.5.Mutation testing and completeness of test cases
The mutation method is a fault-based testing strategy that measures the quality/adequacy of testing by examining whether the test set used in testing can reveal certain types of faults. Mutation Testing is based on two basic Assumptions: (a) The Competent Programmer Hypothesis: In general programmers are competent i.e., the programs they write are nearly correct. The program differs from a correct version in only a few small ways. (b) The Coupling Effect Hypothesis: Large program faults of semantic nature are coupled with smaller syntactic faults can be detected with mutation testing.
The core of a mutation-based testing is a set of operators that modifies the source code to inject a fault. The modified program is known as a mutant. For example, a mutation method may replaces an arithmetic operator or it may introduce the change in the comparison operator or may be any logical operator change in the original program, which is anticipated to represent the a wrong operator. The many mutation operators suggested, out of those operators, researcher selected the following mutation operators that are applicable to our problem:
[1] Clause Reference Fault (CRF) – Clause ‘a’ is replaced with another clause ‘b’. For example, the specification (x >7) V (y < 3) is implemented as (x > 9) V (y < 3).
[2] Clause Negation Fault (CNF) - Clause is replaced by its negation .
[3] Clause Insertion Fault (CIF) - Clause b is inserted, for example clause is replaced by op where another clause, and op is either conjunction or disjunction. There exist two subclasses of this class.
[4] Clause Conjunction Fault (CCF) - Clause is replaced by . [5] Clause Disjunction Fault (CDF) - Clause is replaced by .
[6] Relational Operator Reference Fault (RRF) - Relational operator is replaced by any other relational operator. If relational operator is replaced with its opposite then this is the same as negating the relational expression.
[7] Off-By-1 Fault (OFF) - in a relational expression E1opE2, replace the arithmetic expression E2 with E2 + 1 or E2 − 1. [8] Stuck-At Fault (STF) - stuck-at-0 replaces a clause with 0, stuck-at-1 replaces it with 1.
[9] Expression Negation Fault (ENF) - replace an expression E by .
[10] Missing Expression Fault (MEF) - a predicate is omitted during implementation. Missing Expression Fault includes both where a clause is missing and where a compound predicate is missing.
[11] Logical Operator Reference Fault (LRF) - a Boolean operator is replaced by another operator, e.g., x y is replaced by x ^ y. [12] Associative Shift Fault (ASF) - change the associability of terms. For example, replace (ab) c with a (b c).
[13] Term Omission Fault (TOF): A particular term is omitted during the implementation. For example are wrongly implemented as .
[14] Literal Negation Fault (LNF): A literal in a particular term is wrongly implemented as its negation.
[15] Literal Omission Fault (LOF): A literal in a particular term is omitted during the implementation such as
being implemented as .
[16] Literal Insertion Fault (LIF): A literal not appearing in a particular term is inserted into that term. For example, is incorrectly implemented as .
A mutant said to killed relative to a test data set, if at least one test case generates different results between the mutant and the implementation. else, the mutant is live. If no test case can kill a mutant, then it is either equivalent of the original implementation or a new test case needs to be generated to kill the live mutant, a method of enhancing a test data set. The adequacy of a test data set is measured by a mutation score (MS), which is the percentage of non-equivalent mutants killed by the test data..
The mutation score for a set of test cases Mutation Score =
D = Dead mutants N = Number of mutants
E = Number of equivalent mutants
A set of test cases is mutation adequate if its mutation score is 100%.
3. Consideration used for test selection 3.1 Random testing
In the random testing, the test cases are pseudorandom numbers generated in the MATLAB. A random number generator generates the test data with no use of feedback from previous tests. The tests are passed to the procedure under test, in the hope that all branches will be traversed.
3.2 Antirandom testing
In the antirandom software testing, in the previous researches, the test cases are generated by using the hamming or Cartesian distance techniques. However, in this research paper researcher used the Gray code for generating the test cases. A antirandom number generator generates the test data with use of feedback from previous tests. The tests are passed to the procedure under test, in the hope that it detects the maximum errors.
3.2.1 Algorithm for Antirandom Test case Generation
Consider a n bit binary number Bin 1:0] with I representing the index of the binary number. Let Gray [n-1:0] be the equivalent Gray code.
1. For I=n-1,
Gray [n-1] = Bin [n-1]
[the most significant bit (MSB) of the Gray code is same as the MSB of original binary number.]
2. For I=n-2 to 0,
Gray[i]= Bin[i+1] XOR Bin[i]
[ Ith bit of the Gray code is the exclusive-OR (XOR) of Ith of the bit of the binary number and (I+1)th of the bit of the binary number.]
In fact, a "Gray code" almost always refers to a binary-reflected Gray code. On the other hand,
mathematicians have discovered other kinds of Gray codes. For example, n-ary Gray code, Balanced Gray code, Beckett–Gray code etc.,
3.2.2 n-ary Gray code
This is the specialized type of Gray codes other than the binary-reflected Gray code, also known as a non-Boolean Gray code. As the name entails this Gray code uses non-non-Boolean values in its encodings.
The (n, k)-Gray code is the n-ary Gray code with k digits.[12] For example, the sequence of elements in the (3, 2)-Gray code is: {00, 01, 02, 12, 10, 11, 21, 22, 20}. Moreover, (n, k)-Gray code may be constructed recursively, as the binary-reflected Gray code, or may be constructed iteratively.
3.2.2 Balanced Gray code
In balanced Gray code, the bit changes are distributed as equally as possible among the bit positions.
4. Results
The different applications are tested by the use of test case generated by different testing techniques such as pseudo-random, antirandom with hamming code, adaptive antirandom with gray code. Numbers of test cases are applied and test for detection of faults inserted by using mutation testing. Table shows the antirandom test cases that are generated using the different techniques. Graph shows the average fault detected by these testing techniques.
Table 1. Test case generation techniques.
Sr. No Hamming Code Gray Code
1 000000101011 000000000011
2 000000101001 000000000101
3 000000101000 000000001111
4 000000101100 000000010001
5 000000101101 000000110011
5. Discussions and Conclusion
Adaptive Random Testing uses the distance techniques for generating the input values which are limited to programs that accept prescribed types of inputs. This paper proposed a general method to generate the test cases based on antirandom methods. This kind of method enables ART to be applied to any programs for the specification based software testing. The proposed method directly uses the different Gray code techniques for generating test cases. As shown in the results that the proposed method has the higher mutation score as compared to the random and another distance techniques. Moreover, the proposed method reduces the software testing cost as the less number of test cases are required for testing. A future research topic is to investigate the feasibility of applying this technique to GA algorithms.
References
[1] Sommerville Ian. Software Engineering. Addison-Wesley, 6th edition, August 2000.
[2] Dick Hamlet. Foundations of software testing: dependability theory. In Proceedings of the 2nd ACM SIGSOFT Symposium on Foundations of Software Engineering, pages 128–139. ACM Press, 1994.
[3] Morell L. J. “A theory of fault-based testing”. IEEE Transactions on Software Engineering, 16(8):844–857, 1990.
[4] Michael Young and Richard N. Taylor. Rethinking the taxonomy of fault detection techniques. In Proceedings of the11th International Conference on Software Engineering, pages 53–62. ACM Press, 1989.
[5] Chen,T. Y., Kuo, F.-C. and Zhou, Z. Q., “On favorable conditions for adaptive random testing,” International Journal of Software Engineering and Knowledge Engineering, vol. 17,no. 6, pp. 805–825, 2007.
[6] Chen, T. Y., Kuo, F.-C, Merkel, R. G. and Tse, T. H. “Adaptive random testing: The ART of test case diversity,” Journal of Systems and Software, vol. 83, no. 1, pp. 60–66, 2010.
[7] Malaiya, Y. K. “Antirandom testing: Getting the most out of black-box testing,” in Proceedings of the 6th International Symposium on Software Reliability Engineering, 1995, pp. 86–95.
[8] R. Hierons, J. Derrick, (Eds) “Special Issue on Specification-based Testing” Soft. Testing, Verification and Reliability, vol. 10, 2000.
[9] R. Hamlet. Random testing. In J. Marciniak, editor, Encyclopedia of Software Engineering. John Wiley & Sons, second edition, 2002. [10] G. J. Myers. The Art of Software Testing. Wiley, New York, second edition, 1979.
[11] R. Cobb and H. D. Mills. Engineering software under statistical quality control. IEEE Software, 7(6):45–54, 1990.
[12] T. Dab’oczi, I. Koll’ar, G. Simon, and T. Megyeri. Automatic testing of graphical user interfaces. In Proceedings of the 20th IEEE Instrumentation and Measurement Technology Conference 2003 (IMTC ’03), pages 441–445, Vail, CO, USA, 2003.
[13] Forrester J. E. and B. P. Miller. An empirical study of the robustness of Windows NT applications using random testing. In Proceedings of the 4th USENIX Windows Systems Symposium, pages 59–68, Seattle, 2000.
[14] Miller,B. P. Fredriksen,L. and B. So. An empirical study of the reliability of UNIX utilities. Communications of the ACM, 33(12):32– 44, 1990.
[15] Miller,B. P. Koski, D. C. Lee, P. Maganty,V. Murthy,R. A. Natarajan, and J. Steidl. Fuzz revisited: A re-examination of the reliability of UNIX utilities and services. Technical Report CS-TR-1995-1268, University of Wisconsin, 1995.
[16] [MIL 2005] E. Miller. Website testing. http:// www.soft.com/ eValid/ Technology/ White. Chapters/ website.testing.html, Software Research, Inc., 2005.
[17] Sen,K. Marinov, D. and G. Agha. Cute: a concolic unit testing engine for c. In ESEC/FSE-13: Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundations of software engineering, pages 263–272, New York, NY, USA, 2005.ACM Press.
[19] Yoshikawa,T. Shimura,K. and T. Ozawa. Random program generator for Java JIT compiler test system. In Proceedings of the 3rd International Conference on Quality Software (QSIC 2003), pages 20–24. IEEE Computer Society Press, 2003.
[20] Ince, D.C.:’The automatic generation of test data’, The Computer Journal, Vol. 30, No. 1, pp. 63-69, 1987. [21] Watt D. A., Wichman B. A., & Sayward F. G., Findlay W.: ’ADA language and methodology’, 1987.
[22] I. K. Mak. On the effectiveness of random testing. Master’s thesis, Department of Computer Science, University of Melbourne, 1997. [23] Ammann P. E. and Knight.J. C. “Data diversity: an approach to software fault tolerance”. IEEE Transactions on Computers,
37(4):418–425, 1988.
[24] Bishop, P. G. “The variation of software survival times for different operational input profiles”. In Proceedings of the 23rd International Symposium on Fault-Tolerant Computing (FTCS-23), pages 98–107. IEEE Computer Society Press, 1993.
[25] Finelli G. B.. Nasa software failure characterization experiments. Reliability Engineering and System Safety, 32(1–2):155–169, 1991. [26] Chan, K. P., Chen, T. Y., and Towey, D. “Restricted random testing: Adaptive random testing by exclusion”. Accepted to appear in
International Journal of Software Engineering and Knowledge Engineering, 2006.
[27] Chen T. Eddy, Y., G. Merkel, R. G. and Wong, P. K. “Adaptive random testing through dynamic partitioning”. In Proceedings of the 4th International Conference on Quality Software (QSIC 04), pages 79–86, Braunschweig, Germany, 2004. IEEE Computer Society Press.
[28] Chen T. Y. and Huang. D. H. “Adaptive random testing by localization”. In Proceedings of the 11th Asia-Pacific Software Engineering
Conference (APSEC’04), pages 292–298. IEEE Computer Society, 2004.
[29] Chen T. Y., Kuo, F. C. and Zhou Z. Q.. “On the relationships between the distribution of failure-causing inputs and effectiveness of adaptive random testing”. In Proceedings of the 17th International Conference on Software Engineering and Knowledge Engineering
(SEKE 2005)), pages 306–311, Taipei, Taiwan, 2005.
