The previous sections 3.2 and 3.3 introduced current testing methodologies as well as related projects in the field of functional testing. All of these approaches include their relative strengths and weaknesses that have been mentioned at the end of each section. The target of this chapter is to derive requirements for a new optimised solution in order to do functional testing of value-added services. This means that these requirements will also be the basis for the proposed Test Creation Framework (TCF) which will be presented in chapter 4.
• Test Execution – The fully automatic execution of tests is one major requirement
that has to be met by any test framework. The test developer does not have to do any manual actions.
• Test Report – The test execution environment shall deliver a thorough test report
that a test developer can interpret easily.
Regarding the related projects, the TT-Medal Test Platform as well as the ComGeneration approach use the TTworkbench, a TTCN-3-based test execution environment in order to
3.4 Requirements for a New Optimised Solution for Functional Testing of Value-Added Services
execute tests. The TTworkbench also supports the generation of report results. The Telling TestStories project includes a proprietary approach to execute tests based on CORBA and RMI. U2TP does not support test execution and Fokus!MBT requires an external tool that is not specified.
A further criterion which could be derived from agile approaches such as ATDD and BDD is that the needs of the service customer should always be the centre of attention in the testing and development process. Based on this aspect, the following requirements can be determined by means of keywords:
• Collaboration and support for agile principles – An optimised solution is required
to integrate all stakeholders in the test process, such as service developers, test developers and service customers.
• Comprehension – All stakeholders shall always have the chance to get an
overview of the project progress (both testing and development), especially the service customer if he is interested.
None of the current related projects directly support the collaboration or comprehension. The ComGeneration approach lets the service customer participate in the compilation of the Service Description, a contract document between the service customer and the service provider. The Telling TestStories project includes the compilation of test stories which can be compared to approaches in ATDD and BDD. So, a minimal support for agile principles can de identified.
The next set of requirements refers to the usability of a test framework. Here, the following keywords have been defined:
• Manageability and time exposure – It is important that the framework concepts
and methodologies do not overburden the stakeholders and are quite easily manageable in a reasonable timeframe.
• Tool support – The framework shall provide tools especially for the test developer
to maintain the test process.
U2TP is very well documented as it is also a test specification standard and directly connected to UML. There is also a tool that uses components of U2TP (e.g. Eclipse Test & Performance Tools Platform Project) (Eclipse TPTP, 2015). For the ComGeneration approach, an Eclipse Modeling Framework (EMF)-based solution exists, but it lacks relevant documentation. The use of the tool is manageable, but not straightforward. Fokus!MBT is very complex, as it involves many types of applications depending on the functionality to apply (e.g. one tool for test data generation). For test modelling, U2TP is used. The Telling TestStories approach provides a good documentation and a tool is shipped as a bundle of Eclipse plugins (Telling TestStories, 2015). It is quite easily manageable, but the functionality is also very limited. For the TT-Medal project, there is no existing tool that can be used.
• Traceability of requirements – It shall be possible to detect the specified
requirements throughout the whole testing process.
The traceability of requirements is supported by the Fokus!MBT and Telling TestStories approaches. The other related projects do not mention the support.
The upcoming set of requirements is directly derived from the aims and objectives of this research:
3.4 Requirements for a New Optimised Solution for Functional Testing of Value-Added Services
• Reusability – It shall be possible to reuse certain aspects or components within
the test process in order to save time in future projects.
• NGN-compliance or support for general SIP-based IP networks – The framework
either shall consider the NGN-related artefacts such as possible SCEs, SDPs and SIP AS or standard SIP-based IP networks.
• Verification and Validation – The framework shall provide both verification and
validation through test processes. Especially the validation of a value-added service requires an intense involvement of the service customer in the test process.
Regarding the reusability, the ComGeneration approach defines reusable EFSMs that describe common behaviour. Fokus!MBT and U2TP specify reusable test patterns which refer to recurring test architectures, but no recurring behaviour is specified. Regarding the NGN-compliance or the support for SIP-based IP networks, ComGeneration is the only project to support this. As all approaches are MBT-based, the verification should also be supported by them. Because of the missing involvement of the service customer in the processes of the related projects, the validation is not supported by any mentioned project.
• Effectivity and efficiency of generated test cases – The framework shall generate
an amount of test cases that is feasible. Furthermore, these test cases shall be sufficient enough to prove that the SUT has been implemented completely towards the specified requirements.
Fokus!MBT and U2TP both apply the test generation methods mostly based on UML sequence diagrams. This is a rather efficient method, because the amount of test cases is manageable. However, it does not prove that the test cases are sufficient enough. The ComGeneration approach is not efficient as it includes the well-known state explosion
problem, but it covers all possible behaviours that might occur in value-added service consumption. The authors of Telling TestStories claim that their approach is efficient because the test can be defined on an abstract visual level with tool support (Felderer et
al., 2011). There is no explicit information given regarding the effectivity.
• Expandability – It shall be possible to expand the functionality of the framework
or rather to widen the support for further technologies (such as further protocols that can be tested).
U2TP is based on UML and because of the object-oriented concept of modularity and code reuse, this concept should also be provided by concepts that are based on U2TP (so, also for Fokus!MBT). Principally, this is also possible for ComGeneration, because for further support of technologies, new modular finite state machines have to be defined. Telling TestStories includes expandability through the possibility of automatically generating adapters for the communication with the SUT.
The following Table 3.2 illustrates a list of the related projects with the evaluation regarding the derived requirements.
3.4 Requirements for a New Optimised Solution for Functional Testing of Value-Added Services Table 3.2: Evaluation of related projects based on derived requirements
Requirements Current related test projects
U2TP TT-Medal Fokus!MBT ComGeneration Telling TestStories
Test Execution - + o + +
Test Report - + o + +
Collaboration and support for agile principles - - - o o Comprehension - - - - - Manageability and time exposure o - - o + Tool support + - + o o Traceability of requirements - - + - + Reusability o - o + - NGN-compliance or support for general SIP-based IP networks - - - + - Verification + + + + + Validation - - - - - Effectivity and efficiency of generated test cases
o - o o o
Expandability + - + + +
Considering the specified requirements, a novel framework for functional testing of value-added services will be proposed in chapter 4 and the underlying concept will then be explained in the upcoming chapters 5, 6 and 7.
3.5 Conclusion
This chapter introduced the fundamentals of testing and functional testing and its application for value-added telecommunication services. In addition, the difference between verification and validation of a system (or service) was discussed (section 3.1).
Section 3.2 introduced the state-of-the-art testing methodologies, especially agile concepts. The advantages and limitations of the approaches were discussed and it was concluded, that the development and the testing processes have to be performed by different persons (service developers and test developers). Mentionable is also that most agile concepts focus on a close collaboration between developers, testers and customers of a system or service.
Section 3.3 described related testing approaches, tools and methodologies. Most of them refer to some kind of MBT approach either focussing on enhancing system models with test-related parameters by using U2TP in order to automatically generate TTCN-3 test cases or by supporting a tester to create models from which the tests are directly derived. In principle, the approaches in literature lack the definition of a proper testing methodology from the definition of the requirements of a value-added service until the generation and subsequent evaluation of functional tests.
The main outcome of this chapter is the evaluation of the current related projects based on requirements. The requirements have been derived from the weaknesses and strengths of the testing methodologies and related projects and represent the major criterion for the proposed novel framework for testing of value-added services described in the upcoming chapter.
4 Proposed Framework for Testing of
Value-Added Services
This thesis proposes a novel framework that fulfils the requirements stated in section 3.4 and fills the gap of a thorough solution for service providers to provide well-tested value- added telecommunication services to their service customers. This chapter begins by defining the preconditions and tasks to be considered when the novel framework is established (section 4.1), followed by the introduction of the overall novel methodology to enable a more service customer-centric approach (section 4.2). Subsequently, section 4.3 will describe the framework architecture and its components.