Concluding Remarks
6.3 Open Issues and Future Work
target final-product, in terms of behavioral test patterns. From use cases, they structure scenarios to produce reusable test patterns common to an entire product line and represented using the UML. When the final design is available and a product chosen as test target, test cases for that particular product are synthesized.
Despite the approaches aforementioned are representative in the field of SPL testing, they have limitations that do not allow one to fully apply it to an industrial context, since they are mainly concerned about generating test cases, usually from use case scenarios. They do not provide the elements of a software process, which contain information that make its practical implementation feasible. In addition, little is reported regarding the studies’ validation. (Reuys et al.,2006) is the only study from these which reported how the validation was performed.
Despite that, the way validation was reported does not enable its replication since no guidelines on how the experimental study was performed were reported. This indeed does not invalidate the proposed process, however it might be infeasible its implementation in other contexts than the one presented in the study. Hence, more evidence is required.
In our proposal, we go further. We adapt the test case generation methods described in these approaches, since we considered them as well defined practices, but we also give details that are not provided by them, in terms of how to perform testing in a SPL project.
6.3 Open Issues and Future Work
Besides the out-of-scope topics mentioned as early as in the Section1.3, some problems we judge as important were not covered by this approach.
So far, we have no evidence about how we can exploit the SPL model to generate tests that are effective at revealing faults. Data collected through the experiment showed that subjects advocate the use of the approach as a fault revealing effective strategy, but we need measurable evidences other than subjects’ feedback. We need practical evidence. Moreover, we did not analyze the impact of variant binding times in testing concerns. Variant binding much probably impacts on the testing effort, but we had no inputs to evaluate such aspect.
Our future research agenda includes investigation on the topics not covered by this study, as the ones aforementioned as well as others detailed below.
Meta Model for Variability Management. Our approach for variability management within test assets uses a meta model for trace variability among artifacts. But it needs to be evolved, since currently it is only an initial model. We have already defined the concepts and relations of the assets, but we need to formalize it by describing it using an standardized
6.3. OPEN ISSUES AND FUTURE WORK
language, such as Object Constraint Language (OCL), by OMG2.
Test Automation. We believe that automated support for the variability management process is an important issue to be investigated. It will make automated product configuration analysis possible, thus, providing the organizations with mechanisms to evaluate PL adoption and evolution. There is a Master Student in our Research Group who is investigating how the RiPLE-TE process can be extended to support automatic variability management an traceability.
Test Case Selection. Based on the RiPLE-TE Process, thus including the metamodel, the next step is to provide an effective way to select test cases considering variability concerns.
There is another Master Student in our Research Group who is working towards extending the RiPLE-TE to consider test case selection. The student is investigating suitable techniques for selecting test cases, using both automatic tools and manual approaches.
Experimental Software Engineering. A next step is to perform an experimental study in a broader context, before applying it in an industrial project. This will give us more evidences and consequently more confidence on the process effectiveness.
The RiPLE has currently being applied in a SPL industrial project, in the medical information management domain. It is a large project, including about 800 features. Based on this project we will apply, along the 2010/2 semester, the RiPLE-TE process in a part of the project, and collect data to improve the proposal. Then, we will have the opportunity to gather data in a set of aspects, such as: understand sizes - understand the impact of the amount of variability points and variants in a project. We have in mind that real SPLs may have hundred of VPs and several hundreds of variants, hence it is necessary to understand, with evidence, how a testing approach fits in different scenarios; quantify extent and complexity of constraints - understand the relationship of constraints among features, and its impact on the testing activities. As constraints grow in complexity and number, the difficulty of modeling and generating test suites increases;
effectiveness and feasibility of testing methods- we will try to apply different testing methods and analyze their suitability to the proposed approach.
We propose to additionally investigate how much additional testing effort a variant would cause. As the SPL grows, we must have testing mechanisms that do not exponentially increase the effort as several new variants are inserted.
Fault Model for SPL. The project above mentioned will also serve as an environment to gather issues from all SPL disciplines involved throughout its development, so that we will accomplish an issue library to be analyzed and become a fault model, that can aid future Testing and Inspection agendas, in the form of an error prediction approach based on evidences.
2
Bibliography
Afzal, W., Torkar, R., and Feldt, R. (2008). A systematic mapping study on non-functional search-based software testing. In SEKE, pages 488–493. Knowledge Systems Institute Graduate School.
Afzal, W., Torkar, R., and Feldt, R. (2009). A systematic review of search-based testing for non-functional system properties. Information and Software Technology, 51(6), 957–976.
Al-Dallal, J. and Sorenson, P. (2008). Testing software assets of framework-based product families during application engineering stage. Journal of Software, 3(5), 11–25.
Almeida, E. S., Alvaro, A., Lucrédio, D., Garcia, V. C., and Meira, S. R. L. (2004). Rise project:
Towards a robust framework for software reuse. In D. Zhang, É. Grégoire, and D. DeGroot, editors, IRI, pages 48–53. IEEE Systems, Man, and Cybernetics Society.
Almeida, E. S., Alvaro, A., Lucrédio, D., Garcia, V. C., and Meira, S. R. L. (2005). A survey on software reuse processes. In D. Zhang, T. M. Khoshgoftaar, and M.-L. Shyu, editors, IRI, pages 66–71. IEEE Systems, Man, and Cybernetics Society.
Almeida, E. S., Alvaro, A., Garcia, V. C., Jorge, Burégio, V. A., Nascimento, L. M., Lucrédio, D., and Silvio (2007). C.R.U.I.S.E: Component Reuse in Software Engineering. C.E.S.A.R e-book, Recife, 1st edition.
Alvaro, A., de Almeida, E. S., and de Lemos Meira, S. R. (2006). A software component quality model: A preliminary evaluation. In EUROMICRO-SEAA, pages 28–37. IEEE.
Ammann, P. and Offutt, J. (2008). Introduction to Software Testing. Cambridge University Press, 1st edition.
Bachmann, F. and Clements, P. C. (2005). Variability in software product lines. Technical Report CMU/SEI-2005-TR-012 ESC-TR-2005-012, CMU/SEI - Software Engineering Institute, Pittsburgh, PA.
Bailey, J., Budgen, D., Turner, M., Kitchenham, B., Brereton, P., and Linkman, S. G. (2007).
Evidence relating to object-oriented software design: A survey. In ESEM, pages 482–484.
IEEE Computer Society.
BIBLIOGRAPHY
Barros, J. L. and Marqués, J. M. (2006). Support to development-with-reuse in very small software developing companies. In M. Morisio, editor, ICSR, volume 4039 of Lecture Notes in Computer Science, pages 419–422. Springer.
Basili, V. R., Caldiera, G., and Rombach, H. D. (1994). Goal question matric paradim. In Encyclopaedia of Software Engineering, volume 2. John Wiley & Sons, Inc.
Beatriz Pérez Lamancha, Macario Polo Usaola, M. P. (2009). Towards an automated test-ing framework to manage variability ustest-ing the uml testtest-ing profile. In ICSE Workshop on Automation of Software Test (AST09).
Bertolino, A. (2007). Software testing research: Achievements, challenges, dreams. FOSE, pages 85–103.
Bertolino, A. and Gnesi, S. (2003a). Pluto: A test methodology for product families. In Software Product-Family Engineering, 5th International Workshop, PFE, Siena, Italy, pages 181–197.
Bertolino, A. and Gnesi, S. (2003b). Use case-based testing of product lines. ACM SIGSOFT Software Engineering Notes, 28(5), 355–358.
Bezerra, Y. M., Pereira, T. A. B., and da Silveira, G. E. (2009). A systematic review of software product lines applied to mobile middleware. In ITNG ’09: Proceedings of the 2009 Sixth International Conference on Information Technology: New Generations, pages 1024–1029, Washington, DC, USA. IEEE Computer Society.
Black, A. (2003). Critical Testing Process: Plan, Prepare, Perform, Perfect. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.
Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., and Khalil, M. (2007). Lessons from applying the systematic literature review process within the software engineering domain.
Journal of Systems and Software, 80(4), 571–583.
Brito, K. S., Garcia, V. C., Almeida, E. S., and Meira, S. R. L. (2008). Lift - a legacy information retrieval tool. J.UCS - Journal of Universal Computer Science, 14(8), 1256–1284.
Budgen, D., Turner, M., Brereton, P., and Kitchenham, B. (2008). Using Mapping Studies in Software Engineering. In Proceedings of PPIG 2008, pages 195–204. Lancaster University.
Burnstein, I. (2002). Practical Software Testing. Springer-Verlag New York, Inc., Secaucus, NJ, USA.
BIBLIOGRAPHY
Cavalcanti, Y. C. (2009). A Bug Report Analysis and Search Tool. Master’s thesis, UFPE -Federal University of Pernambuco.
Cavalcanti, Y. C., da Cunha, C. E. A., de Almeida, E. S., and de Lemos Meira, S. R. (2009).
Bast: A tool for bug report analysis and search. In SBES - 23rd Brazilian Symposium on Software Engineering, Tools Session, Fortaleza, CE, Brazil.
Chen, L., Babar, M. A., and Ali, N. (2009). Variability management in software product lines:
A systematic review. In SPLC 2009: 13th Software Product Line Conference, San Francisco, CA, USA.
Chernak, Y. (2001). Validating and improving test-case effectiveness. IEEE Software, 18(1), 81–86.
Clements, P. and Northrop, L. (2001). Software Product Lines: Practices and Patterns. Addison-Wesley, Boston, MA, USA.
Cohen, M. B., Dwyer, M. B., and Shi, J. (2006). Coverage and adequacy in software product line testing. In R. M. Hierons and H. Muccini, editors, ROSATEA, pages 53–63. ACM.
Condori-Fernández, N., Daneva, M., Sikkel, K., Wieringa, R., Tubío, Ó. D., and Pastor, O.
(2009). A systematic mapping study on empirical evaluation of software requirements specifications techniques. In ESEM, pages 502–505.
Condron, C. (2004). A domain approach to test automation of product lines. SPLiT - Workshop on Software Product Line Testing, pages 27–35.
Craig, R. D. and Jaskiel, S. P. (2002). Systematic Software Testing. Artech House, Inc., Norwood, MA, USA.
Crnkovic, I. (2002). Building Reliable Component-Based Software Systems. Artech House, Inc., Norwood, MA, USA.
Denger, C. and Kolb, R. (2006). Testing and inspecting reusable product line components:
first empirical results. In ISESE: Proceedings of the International Symposium on Empirical Software Engineering, pages 184–193, New York, NY, USA.
Durão, F. A. (2008). Semantic Layer Applied to a Source Code Search Engine. Master’s thesis, UFPE - Federal University of Pernambuco.
BIBLIOGRAPHY
Dybå, T. and Dings, T. (2008). Empirical studies of agile software development: A systematic review. Information and Software Technology, 50(9-10), 833–859.
Dybå, T. and Dingsøyr, T. (2008). Strength of evidence in systematic reviews in software engineering. In ESEM ’08: Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement, pages 178–187, New York, NY, USA.
ACM.
Edwin, O. O. (2007). Testing in Software Product Lines. Master’s thesis, School of Engineering at Blekinge Institute of Technology.
Engström, E., Skoglund, M., and Runeson, P. (2008). Empirical evaluations of regression test selection techniques: a systematic review. In ESEM ’08: Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement,
pages 22–31, New York, NY, USA. ACM.
Feng, Y., Liu, X., and Kerridge, J. (2007). A product line based aspect-oriented generative unit testing approach to building quality components. In COMPSAC - Proceedings of the 31st Annual International Computer Software and Applications Conference, pages 403–408, Washington, DC, USA.
Ganesan, D., Maurer, U., Ochs, M., Snoek, B., and Verlage, M. (2005). Towards testing response time of instances of a web-based product line. SPLiT - Workshop on Software Product Line Testing.
Garcia, V. C., Lisboa, L. B., de Lemos Meira, S. R., de Almeida, E. S., Lucrédio, D., and de Mattos Fortes, R. P. (2008). Towards an assessment method for software reuse capability (short paper). In H. Zhu, editor, QSIC - International Conference on Quality Software, pages 294–299. IEEE Computer Society.
Geppert, B., Li, J. J., Rößler, F., and Weiss, D. M. (2004). Towards generating acceptance tests for product lines. In ICSR - Proceedings of 8th International Conference on Software Reuse, Lecture Notes in Computer Science, pages 35–48.
Goldsmith, R. F. and Graham, D. (2002). The forgotten phase. Software Development Magazine, pages 45 – 47.
BIBLIOGRAPHY
Graves, T. L., Harrold, M. J., Kim, J.-M., Porter, A., and Rothermel, G. (2001). An empirical study of regression test selection techniques. ACM Transaction on Software Engineering Methodology, 10(2), 184–208.
Harrold, M. J. (1998). Architecture-based regression testing of evolving systems. In International Worshop on Role of Architecture in Testing and Analysis (ROSATEA 1998), pages 73–77,
Marsala, Sicily, Italy.
Harrold, M. J. (2000). Testing: a roadmap. In ICSE ’00: Proceedings of the Conference on The Future of Software Engineering, pages 61–72, New York, NY, USA. ACM.
Hartmann, J., Vieira, M., and Ruder, A. (2004). A UML-based approach for validating product lines. SPLiT - Workshop on Software Product Line Testing, pages 58–65.
Haumer, P. (2007). Eclipse process framework composer (part 1: Key concepts). Technical report, IBM Rational Software.
Hui Zeng, W. Z. and Rine, D. (2004). Analysis of testing effort by using core assets in software product line testing. SPLiT - Workshop on Software Product Line Testing, pages 1–6.
IEEE (1988). IEEE guide for the use of IEEE standard dictionary of measures to produce reliable software - 982.2-1998. IEEE Computer Society.
IEEE (1998). IEEE Standard for Software Test Documentation - 829-1998. IEEE Computer Society.
Jaring, M., Krikhaar, R. L., and Bosch, J. (2008). Modeling variability and testability inter-action in software product line engineering. In ICCBSS - 7th International Conference on Composition-Based Software Systems, pages 120–129.
Jedlitschka, A., Ciolkowski, M., and Pfahl, D. (2008). Reporting experiments in software engineering. In Guide to Advanced Empirical Software Engineering, chapter 8, pages 201–
228. Springer, Secaucus, NJ, USA.
Jin-hua, L., Qiong, L., and Jing, L. (2008). The w-model for testing software product lines.
International Symposium on Computer Science and Computational Technology, 1, 690–693.
Juan Jenny Li, Birgit Geppert, F. R. and Weiss, D. (2007). Reuse execution traces to reduce testing of product lines. SPLiT - Workshop on Software Product Line Testing.
BIBLIOGRAPHY
Juristo, N. and Moreno, A. M. (2006). Guest editors’ introduction: Software testing practices in industry. IEEE Software, 23(4), 19–21.
Juristo, N., Moreno, A. M., and Vegas, S. (2002). A survey on testing technique empirical studies: How limited is our knowledge. In ISESE ’02: Proceedings of the 2002 International Symposium on Empirical Software Engineering, page 161, Washington, DC, USA. IEEE Computer Society.
Juristo, N., Moreno, A. M., and Vegas, S. (2004). Reviewing 25 years of testing technique experiments. Empirical Software Engineering, 9(1-2), 7–44.
Juristo, N., Moreno, A. M., Vegas, S., and Solari, M. (2006). In search of what we experimentally know about unit testing. IEEE Software, 23(6), 72–80.
Käkölä, T. and Dueñas, J. C., editors (2006). Software Product Lines - Research Issues in Engineering and Management. Springer.
Kamsties, E., Pohl, K., Reis, S., and Reuys, A. (2003). Testing variabilities in use case models.
In Software Product-Family Engineering, 5th International Workshop, PFE, Siena, Italy, pages 6–18.
Kang, S., Lee, J., Kim, M., and Lee, W. (2007). Towards a formal framework for product line test development. In CIT ’07: Proceedings of the 7th IEEE International Conference on Computer and Information Technology, pages 921–926, Washington, DC, USA. IEEE Computer Society.
Kauppinen, R. (2003). Testing framework-based software product lines. Master’s thesis, University of Helsinki Department of Computer Science.
Kauppinen, R. and Taina, J. (2003). Rita environment for testing framework-based software product lines. In P. Kilpelinen and N. Pivinen, editors, SPLST, pages 58–69. University of Kuopio, Department of Computer Science.
Kauppinen, R., Taina, J., and Tevanlinna, A. (2004). Hook and template coverage criteria for testing framework-based software product families. SPLiT - Workshop on Software Product Line Testing, pages 7–12.
Kishi, T. and Noda, N. (2006). Formal verification and software product lines. Communications of the ACM, 49(12), 73–77.
BIBLIOGRAPHY
Kitchenham, B. (2010). What’s up with software metrics? - a preliminary mapping study.
Journal of Systems and Software, 83(1), 37–51.
Kitchenham, B. and Charters, S. (2007). Guidelines for performing Systematic Literature Reviews in Software Engineering. Technical Report EBSE 2007-001, Keele University and Durham University Joint Report.
Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin, D. C., Emam, K. E., and Rosenberg, J. (2002). Preliminary guidelines for empirical research in software engineering.
IEEE Transactions on Software Engineering, 28(8), 721–734.
Kitchenham, B. A., Dyba, T., and Jorgensen, M. (2004). Evidence-based software engineering.
In ICSE: Proceedings of the 26th International Conference on Software Engineering, pages 273–281, Washington, DC, USA.
Kitchenham, B. A., Mendes, E., and Travassos, G. H. (2007). Cross versus within-company cost estimation studies: A systematic review. IEEE Transactions on Software Engineering, 33(5), 316–329.
Klaus Pohl, G. B. and van der Linden, F., editors (2005). Software Product Line Engineering -Foundations, Principles, and Techniques. Springer.
Kolb, R. (2003). A riskdriven approach for efficiently testing software product lines. GPCE -5th Generative Programming and Component Engineering.
Kolb, R. and Muthig, D. (2003). Challenges in testing software product lines. CONQUEST -7th Conference on Quality Engineering in Software Technology, pages 81–95.
Kolb, R. and Muthig, D. (2006). Making testing product lines more efficient by improving the testability of product line architectures. In ROSATEA: Proceedings of the ISSTA workshop on Role of software architecture for testing and analysis, pages 22–27, New York, NY, USA.
Krueger, C. W. (2006). Introduction to the emerging practice software product line development.
Methods Tools, 14(3), 3–15.
Kshirasagar Naik, P. T. (2008). Software Testing and Quality Assurance: Theory and Practice.
John Wiley & Sons, Hoboken, New Jersey.
BIBLIOGRAPHY
Lamancha, B. P., Usaola, M. P., and Velthius, M. P. (2009). Software product line testing - a systematic review. In ICSOFT International Conference on Software and Data Technologies, pages 23–30. INSTICC Press.
Lewis, W. E. (2008). Software Testing and Continuous Quality Improvement, Third Edition.
Auerbach Publications, Boston, MA, USA.
Li, J. J., Weiss, D. M., and Slye, J. H. (2007). Automatic integration test generation from unit tests of exvantage product family. In 11th International Conference on Software Product Lines (Workshops), pages 73–80. Kindai Kagaku Sha Co. Ltd., Tokyo, Japan.
Linden, F. J. v. d., Schmid, K., and Rommes, E. (2007). Software Product Lines in Action: The Best Industrial Practice in Product Line Engineering. Springer.
Lisboa, L. B. (2008). ToolDAy - A Tool for Domain Analysis. Master’s thesis, UFPE - Federal University of Pernambuco.
Lisboa, L. B., Garcia, V. C., Almeida, E. S., and Silvio (2007). Toolday a process-centered domain analysis tool. In 21st Brazilian Symposium on Software Engineering, Tools Session, João Pessoa, PB, Brazil.
Lisboa, L. B., Garcia, V. C., Lucrédio, D., de Almeida, E. S., de Lemos Meira, S. R., and de Mattos Fortes, R. P. (2010). A systematic review of domain analysis tools. Information &
Software Technology, 52(1), 1–13.
Mansell, J. X. (2006). Experiences and expectations regarding the introduction of systematic reuse in small- and medium-sized companies. InKäkölä and Dueñas(2006), pages 91–124.
Martins, A. C., Garcia, V. C., Almeida, E. S., and Silvio (2008). Enhancing components search in a reuse environment using discovered knowledge techniques. In 2nd Brazilian Symposium on Software Components, Architectures, and Reuse (SBCARS), Porto Alegre, Brazil.
Mascena, J. C. C. P., de Lemos Meira, S. R., de Almeida, E. S., and Garcia, V. C. (2006).
Towards an effective integrated reuse environment. In S. Jarzabek, D. C. Schmidt, and T. L.
Veldhuizen, editors, GPCE, pages 95–100. ACM.
Mathur, A. P. (2009). Foundations of Software Testing - Fundamental Algorithms and Techniques.
Dorling Kindersley, India, 2nd edition.
BIBLIOGRAPHY
McGregor, J., Sodhani, P., and Madhavapeddi, S. (2004). Testing variability in a software product line. SPLiT - Workshop on Software Product Line Testing, pages 45–50.
McGregor, J. D. (2001a). Structuring test assets in a product line effort. In Proceedings of the 2nd International Workshop on Software Product Lines: Economics, Architectures, and Implications, pages 89–92.
McGregor, J. D. (2001b). Testing a software product line. Technical report, CMU/SEI - Software Engineering Institute.
McGregor, J. D. (2002). Building reusable test assets for a product line. In ICSR - Proceedings of 7th International Conference on Software Reuse, pages 345–346.
McGregor, J. D., Northrop, L. M., Jarrad, S., and Pohl, K. (2002). Guest editors’ introduction:
Initiating software product lines. IEEE Software, 19(4), 24–27.
Medeiros, F. M., de Almeida, E. S., and de Lemos Meira, S. R. (2009). Towards an approach for service-oriented product line architectures. In 3rd Workshop on Service-Oriented Architectures and Software Product Lines (SOAPL) - Enhancing Variation, in conjuction with the 13th International Software Product Line Conference (SPLC), San Francisco, CA, USA.
Melo, C. A., Burégio, V. A. A., Almeida, E. S., and Meira, S. R. L. (2008). A reuse repository system: The core system. In 10th International Conference on Software Reuse (ICSR), Tools Demonstration, Beijing, China.
Mendes, R. C. (2008). Search and Retrieval of Reusable Source Code using Faceted Classifica-tion Approach. Master’s thesis, UFPE - Federal University of Pernambuco.
Mili, H., Mili, A., Yacoub, S., and Addy, E. (2001). Reuse-based software engineering:
techniques, organization, and controls. Wiley-Interscience, New York, NY, USA.
Moraes, M. B. S. (2010). A scoping approach for software product lines.
Moraes, M. B. S., Almeida, E. S., and de Lemos Meira, S. R. (2009). A systematic review on software product lines scoping. In ESELAW 2009: VI Experimental Software Engineering Latin American Workshop, So Carlos-SP, Brazil.
Muccini, H. and van der Hoek, A. (2003). Towards testing product line architectures. Electronic Notes in Theoretical Computer Science, 82(6).
BIBLIOGRAPHY
Muccini, H., Dias, M. S., and Richardson, D. J. (2005). Towards software architectubased re-gression testing. In WADS ’05: Proceedings of the 2005 workshop on Architecting dependable systems, pages 1–7, New York, NY, USA. ACM.
Muccini, H., Dias, M. S., and Richardson, D. J. (2006). Software architecture-based regression testing. Journal of Systems and Software, 79(10), 1379–1396.
Myers, G. J. and Sandler, C. (2004). The Art of Software Testing. John Wiley & Sons.
Nebut, C., Pickin, S., Traon, Y. L., and marc Jézéquel, J. (2002). Reusable test requirements for uml-modeled product lines. In In Proceedings of the Workshop on Requirements Engineering for Product Lines (REPL’02, pages 51–56.
Nebut, C., Pickin, S., Traon, Y. L., and marc Jézéquel, J. (2002). Reusable test requirements for uml-modeled product lines. In In Proceedings of the Workshop on Requirements Engineering for Product Lines (REPL’02, pages 51–56.