Automated Module Testing of Embedded Software Systems

183  Download (0)

Full text


Automated Module Testing of Embedded Software Systems

Master’s Thesis Fredrik Olsson Henrik Lundberg

Supervisors Thomas Thelin, LTH Michael Rosenberg, EMP Nicklas Olofsson, EMP



When designing and implementing large-scale software one of the most important issues in order to reduce the cost and improve the quality is testing. In this master thesis a thorough background to testing with the focus on low-level testing is given as an introduction. Ideas and theory for different kinds of testing as well as available tools are described. The thesis focus on low-level testing and this is important because it saves both time and costs in the later phases of software development. Today’s development teams become more and more aware of the importance of low-level testing.

Unfortunately low-level testing is often inefficiently deployed as it is poorly structured and there are very few tools.

To facilitate and raise the quality of the low-level testing this thesis presents a process for module testing as well as describes the development of an automated testing tool. The process should ensure that all module tests follow the same structure and that all of the modules get the same type of low-level testing. The automated testing tool is a prototype that makes the procedure automated and saves time. The thesis also shows that the developed process and tool greatly enhanced and improved the software quality in the projects using it.



First of all we would like to thank our supervisors Nicklas Olofsson and Michael Rosenberg at Ericsson Mobile Platforms AB for helping us through this master thesis. We would also like to thank our supervisor Thomas Thelin at the Department of Communication Systems at Lund Institute of Technology for the help of giving our report a professional and correct appearance.

Finally, we would like to thank everybody at Ericsson Mobile Platforms AB that helped us by answering our questions and being supportive and encouraging throughout this thesis.

Henrik Lundberg & Fredrik Olsson Lund, November 2003


Table of Contents

Table of contents






2.2.1 Software Specification ... 4

2.2.2 Software Development ... 5

2.2.3 Software Validation and Software Verification ... 5

2.2.4 Software Evolution ... 6


2.3.1 Waterfall Model... 6

2.3.2 Evolutionary Model ... 7


2.4.1 SRS – Software/System Requirements Specification... 8

2.4.2 SDD – Software Design Description ... 9

2.4.3 SVVS – Software Verification and Validation Specification... 9

2.4.4 SVVI – Software Verification and Validation Instruction... 9

2.4.5 Other Documents... 9




3.2.1 Planning ... 12

3.2.2 The Plan... 12

3.2.3 Test Report... 13


3.3.1 Validation ... 14

3.3.2 Verification ... 15

3.3.3 IV & V - Independent Verification and Validation ... 16


3.4.1 Static and Dynamic Testing ... 16

3.4.2 Black-Box Testing... 17

3.4.3 White-Box Testing... 18

3.4.4 Regression Testing... 20


3.5.1 Low-Level Testing... 21 Unit Testing... 21

3.5.2 High-Level Testing ... 23 Integration Testing ... 23 Incremental Testing – “Top - Down” ... 24 Incremental Testing – “Bottom - Up” ... 25 Big Bang... 26 System Testing ... 27 Load Test... 28 Acceptance Testing ... 29



3.7.1 When to Use Testing Tools ... 31

3.7.2 Tools for Inspections and Reviews... 32


3.7.3 Test Execution and Evaluation Tools ... 33 Capture/Playback... 33 Simulators ... 35 Coverage Analysis... 35 Memory Testing ... 35 Load Tests ... 36











5.1.1 Terminology... 42


5.2.1 System Architecture ... 42

5.2.2 Detailed High Level Design PC ... 44

5.2.3 Detailed High Level Design Phone ... 44


5.3.1 System Architecture ... 45

5.3.2 Detailed High Level Design PC ... 46 Detailed High Level Design Module Tester ... 46

5.3.3 Detailed High Level Design Phone ... 47 Detailed High Level Design Host... 47 Detailed High Level Design DSP ... 47









8. REFERENCES ... 70









Table of Contents


1. Introduction

When testing software today most of the resources available to the companies are spent on system testing in which the entire product is tested to see if the product meets the requirements. Companies have developed large- scale strategies for this type of testing as well as clearly designed processes for facilitating the work with the tests. As the demands on reliability and fault-free software increases more and more people are assigned to testing.

However as the demands on the product increases so does the costs for the product. More testing requires more resources and resources cost money.

When a product is released there are three main concerns, the cost of the product, the quality of the product and the time in which the product was produced i.e. is the product still desired on the market? Unfortunately most of the times this creates a paradox as these three demands cannot be combined. If the product should have a high quality it has to be tested properly to remove all faults and this type of extensive testing cost money and take time which leads to a higher cost for the customer as well as a longer development time. Of course the developing companies want to keep the costs as low as possible and at the same time release the product as quickly as possible. But to be able to sell the product, it has to have quality that requires testing. Most companies have found some compromise in which they balance the quality, the costs and the time in which the product is released.

The first step in improving the quality is often to start a special testing group that designs and performs the tests. This means that the tests are done more professionally and more independently which increases the quality of the product. However as the software gets more and more extensive the workload on the testing group and the time spent on testing increase and it will soon become both too expensive and too time consuming to find and correct all of the faults. This is where the next step comes in. To reduce the time and resources spent by the testing group on system-testing low-level testing is given more focus.

Low-level testing confirms the quality in small parts (modules) of the software before they are put together to a complete software product. This type of testing is often not as well structured as it could have been and there are only few commercial tools to help. However if this type of testing is performed on all of the modules in a product the time and resources spent on system-testing are greatly reduced and in large software systems this means that the quality of the product increases at the same time as the time and money spent on the product decreases. This might seem like a paradox but in the studies made on this area it has clearly been showed that for large scale software development low-level testing in a structured and automated fashion greatly enhances the quality of the product. At the same time it saves time in the system-testing as the product have fewer faults.

Ericsson Mobile Platforms has realised this and in their large-scale software development they wish to enhance their already extensive low-level testing for mainly three reasons. They want to unify the testing to make sure that all of the modules in a product receive the same type of low-level testing and


Chapter 1 • Introduction

thereby have the same quality which makes it easier to plan the work with the product. They also want to try to refine the quality in their already high quality products which is done by more extensive and thorough testing.

Finally they want to reduce the time spent on resource demanding rework in late phases. This work is greatly reduced if low-level testing is performed properly as fewer faults are due to faults in the logic of the module code.

To be able to introduce low-level testing into the development phases at Ericsson Mobile Platforms two main things have to be done. Firstly a new process for module testing has to be developed. This process should contain careful instruction for how to design and perform module tests. This process has to be designed on the processes currently used to facilitate the incorporation. The process should also guarantee that all module tests are performed accordingly to the same standards which give all the modules roughly the same quality. The process also helps when the module tests are monitored. It is much easier to verify that the module has been tested and that enough testing has been performed. This verification can be done relatively easy and can be understood by all engineers and not only the developers. Secondly a tool to facilitate and automate module testing has to be developed. The tool must provide the assistance necessary for the testers to be able to perform unified module testing on as many of the available modules as possible. The tool also has to be very easy to use and understand to be accepted by the personnel and to facilitate the incorporation at Ericsson Mobile Platforms. The process and the tool should be developed in unison to provide a complete testing environment for low-level testing.

This master thesis describes the work with the process and the prototype of the automated testing tool. In chapter two a brief description of software development is given to get a better understanding of how the prototype was developed and how software in general is designed. To be able to understand why and how the tool was constructed and on which principles the process is built chapter three gives a background description on testing in general in which low-level testing in particular are discussed before the description of the development of the process and the tool are discussed. As the process and the tool introduces a new way of performing and looking at low-level testing among a lot of engineers it is of the uttermost importance that the background to the process and the tools are given a lot of time in this thesis.

Therefore not only a theoretical background is given but also a more practical background as a lot of the process and tool is built on the requirements at Ericsson Mobile Platforms and the processes available today. Chapter four describes some of currently available testing methods at Ericsson Mobile Platforms. It also contains a discussion of how to improve the module testing by keeping the best parts of the currently available techniques. In chapter five the implementation of the automated testing tool is described. The description continues in appendix where a more thorough design is presented. In chapter six different evaluations of the developed process and tool are described. In this evaluation engineers at Ericsson Mobile Platforms have tested the product in the sharp environment and faults and improvements on the process and tool are discussed. In a lot of cases the improvements is also incorporated into the prototype or the process but in other cases only solutions to the improvements are discussed as the time for this master thesis did not include these parts of the product. In the last


chapter the work is summarized and the result and experiences from the master thesis are discussed.


Chapter 2 • Software Engineering

2 Software Engineering 2.1 Introduction

In the early time of software development, the programmers got a specification from the customer and directly started to produce the code.

This approach was good enough in the beginning but when the software systems became larger and more complex it wasn’t possible anymore. [19]

In the middle of the 1970s the maintenance cost of the software was higher than the developing costs and it was rising. Another trend was that the hardware costs decreased in opposite to the software costs that continued to increase. The available software developing techniques did not comply with the rising demands. This is what was called the “software crisis”. [3]

The solution to the problem was Software Engineering. Applying an engineering approach during all phases of the software development made it possible to create larger and more complex software systems. Every phase, from the requirement specification to maintenance of the system, should be subject of thorough and systematic engineering work. That includes not only using existing theories and methods but also inventing new ones wherever it can support the development process. With the help of software engineering it is possible to produce high-quality software and adherent documentation.

The definition of high-quality software differs from program to program and reflects the behaviour of the software during executing rather than what the software does. For example it could be the response time or the reliability of the software. [7]

2.2 Software Process

A software process is needed during development. A process consists of a step-by-step description of how to manage development activities in every phase. A high-class software process is a pre-requisite for high-class software. [7]

The software process activities differ from project to project but mainly consist of the following four activities:

2.2.1 Software Specification

During this phase the definition and all the constraints of the system is written on paper. It is important that everything is thought of since correction in a later phase is much more expensive considering both time and money.

Not just the developers are in need for a good software specification, but also the salesmen and lawyers of the company since the contracts with customers often rely upon the software specification. The developers usually have a more detailed specification than the customers.

The activities in this phase should be described in a SRS, Software Requirements Specification (see 2.4.1). This document defines what the


customer and supplier have agreed on and because of that it is the most important document of all.

2.2.2 Software Development

The software development phase consists of a design part and an implementation part. During the design phase all the major issues are decided, e.g. system structure, data structures and interfaces between parts of the system. Sometimes what algorithms to use is decided during this part and sometimes that is decided during the implementation part.

The design is developed in an iterate manner. It starts on a low level of details and gradually transforms into a higher level of details. If an evolutionary process is used the software development phase could also include returning to the previous phase, to change the specification.

The documents needed in the software development phase could be created next after the specification phase but before this phase. They consist of the planning and design of the development.

The development should be planned and described in a SDD, Software Design Description (see 2.4.2).

Sometimes the SDD is replaced with two documents. The first is the STLDD, Software Top Level Design Document (see 2.4.2), which describes the design in a higher level, e.g. what data structures to use, and the second is the SDDD, Software Detailed Design Document (see 2.4.2), which describes the design in a level closer to the implementation.

2.2.3 Software Validation and Software Verification

Software validation is when the software is compared with the customer’s requirements and demands. This is a very important activity since if the customer is not satisfied with the result the product cannot be considered correct (see 3.3.1 Validation).

Software verification is when the software is compared with the requirements on the software. This is also an important activity since a product that does not comply with the requirements can neither be considered to be correct. The verification of the code is usually known as testing (see 3.3.2 Verification).

The validation and verification activities should be performed during and after the implementation in every phase.

When testing extensive software systems it is important to test the different parts isolated to make clear that the tested part is correct and works as its interface to the environment and the rest of the system specifies. This isolated testing is performed at different levels (see Figure 2.1: The testing process and 3.5 Low-level and high-level testing).


Chapter 2 • Software Engineering

Figure 2.1 The testing process

The documents in this phase are created even before the code writing has started. The first activities consist of creating a SVVS (see 2.4.3), which e.g.

specifies what type of tests to perform when the code is written and the actual testing starts. The details of the testing are described in the SVVI (see 2.4.4). That document shows among other things what to test and what the outcome of the tests is supposed to be.

2.2.4 Software Evolution

The software evolution is the correcting of the system after that the product is considered done. This includes both updates and expansion of the functionality in the developed software. That is usually many times more expensive than the changes being made during the development phase.

The software evolution costs are steadily increasing and are often a part of the total work. The systems delivered today are hence not as static as earlier and changes can be made due to changing requirements and customer needs.

2.3 Software Process Model

A software process model is an abstraction of a software process. This means that a software model is a condensed software process enlightened from a specific point of view. [7]

There are a number of different software process models. Two of the most common models are the waterfall model and the evolutionary model; these are described in more detail below.

2.3.1 Waterfall Model

The waterfall model was first presented in 1970. Every phase shown in figure below is only active one time during every iteration. When the activities in the iteration are finished the work continuous to the next phase and the work will not return to the previous phase during that iteration. In reality the work starts with the next phase before the previous one is finished. After a small number of iterations a part of the process is


considered finished for the moment. The next iteration then starts at the next phase in the process. All the problems that may arise are left for a later solution. [7]

The result of every phase is one or more document. Each one of these should be approved.

Figure 2.2 The waterfall model

Using the waterfall method (see Figure 2.2: The waterfall model), the design and functionality analysis are made in the beginning of the development. At that time the engineers have little knowledge about the project and its constraints. They do not know which part that is hard to implement and test.

This can have impact on the development in a later phase. [6]

2.3.2 Evolutionary Model

Using the evolutionary model a first version of the program is developed, and then given to the customer for comments. Depending on the comments, the program is changed and new features are added. This continues until the program is considered finished. Using this model the development phases is carried out concurrently and not one after another as with the waterfall model (see above). This makes it feasible to produce the specification during the development. [7]

One of the risks with this model is when not enough resources are assigned for testing in an early stage. In the waterfall model testing is done in the late part of development, but in the evolutionary model testing starts in the beginning and continues through the entire project. The testing costs sometimes seem to be large in the beginning but that is evened up later, as not so much testing has to be done in the end. [6]

2.4 Documents

It is important to produce documentation, during development, which describes the software system. The different parts of the development have their own documentation, i.e. the specification is thoroughly described in a Software Requirements Specification document.


Chapter 2 • Software Engineering

2.4.1 SRS – Software/System Requirements Specification

This is the most important document and the specification should be designed very carefully. The document defines the criteria that the customer and supplier have agreed on. It is therefore important to involve the customer in this document or at least to get his approval of the finished document. The SRS is the base for both the design and the testing and therefore faults in the document are very costly, especially if they are found after the approval of the document. [2, 3]

The SRS should involve a product overview that puts the product into perspective with regard to other products. If the product have any dependencies with other products it is also important to clearly state it in this document. The product should also be briefly described and the functions should be categorised to get a better understanding of the product to develop.

Specific features should also be included and a plan for when each feature can be released. [3]

All limitations to the product must be described. This includes for instance hardware limitations, interface requirements, communication protocols that must be supported. All assumptions made must be clearly stated so that the customer can see how and where the assumptions impact the product. [3]

The user interface must be described in detail. This includes pictures of all expected screens and careful descriptions of all user interactions. This is often a very important part for the customer and to get satisfied customers it is important that there is as little room as possible for interpretations so that the customer really gets what he wants.

The last and largest part includes the specific requirements. The demands should be divided into sections for an easy overview. Dividing the demands into functional and non-functional demands often do this. The functional demands are related to in- and output. Non-functional demands deal with limitations to the system, for instance memory, reliability and efficiency.

The functional demands are often split further into general demands that are valid for all parts of the product, specific demands that deals specifically with one part of the program and interaction demands that deals with combinations of the different parts in the system. [3]

When the demands are written it is important to check that the demands have a few important properties,

- clear - no room for interpretations

- consistent - prioritise between conflicting requirements - reasonable - all demands must be able to be fulfilled - measurable - all demands must be testable

- modifiable - all demands should be revisable

Make sure that all of the demands from customer have been covered in the SRS. If something is missed it costs a lot to correct in the later stages.

[2, 3]

When the SRS is developed it is important to include the testers. The testers make sure that all the demands included in the SRS are testable. It also


enables the testers to start their work with designing the tests earlier as they do not have to wait until the SRS is completely finished. [2]

2.4.2 SDD – Software Design Description

This document is a description of the design for the software. Sometimes this document is divided into two smaller document, STLDD – Software Top Level Design Document and SDDD – Software Detailed Design Document.

STLDD includes high-level design and global data that is shared between different components in the system. SDDD is a more detailed description in which often the implementation itself is included. [2, 3]

2.4.3 SVVS – Software Verification and Validation Specification

This is the first test related document that is developed. SVVS is developed in connection with SRS either at the same time or directly after. The document is developed by the testers and includes three main parts. The first part is inspections. All inspections planned for the project are listed and described. The second part is the tests. The team decides which types of tests that is performed and designs specifications for functional tests, system tests and regression tests. The third part is a quality evaluation in which goals are set up to meet the quality aspect of the project. [2]

2.4.4 SVVI – Software Verification and Validation Instruction

As soon as the SVVS is finished the test team should start designing the tests in more detail. The tests should contain a short introduction that describes what is tested, how the test is performed and what the expected outcome of the test is. Every instruction should only contain one test case so that faults could be easier tracked. All these instructions are collected in the SVVI. The SVVI is also known as “Test Procedure”. [2]

2.4.5 Other Documents

Of course there are many other documents that can be used to structure the work. These include Software Configuration Management Plan (SCMP) that specifies methods to identify software products, control and implement changes. This document is normally only written in really large projects that involve many engineers. Another important document is the Software Release Plan (SRP) that defines when and how a product should be released.

This process involves questions like how do we create baselines, who is responsible for releases and requirements for creating baselines and releases.

However it is the documents described in the chapters above that are the most important and the ones that should get the most attention from a testing point of view. [3]


Chapter 3 • Testing – A General Background

3 Testing – A General Background 3.1 Introduction

All of the programs today are written by human developers and these humans even though they do their very best they inevitable make mistakes.

The mistakes can be both costly and cause large-scale problems. The programs are tested to eliminate these problems. To find all of the faults an enormous amount of time would have to be spent and the cost would be much more than any customer could pay. Instead a good test planning is developed in which the most critical code is covered as well as possible.

Twenty percent of the code contains eighty percent of the faults so a big part of testing is to find out which code to test, the so-called critical sections. [5]

To get the best result from testing it is always better to test the code when it is being developed than wait until the product is finished. If the fault is found and corrected in an early stage the cost and impact of the fault could be minimised. If the fault was left for later testing the developers would have to go back in the development process and rewrite the code or perhaps even rewrite the design when the fault finally is found. Often the mistake had an influence on other parts of the code that also have to be rewritten, this take time and cost money. Besides if you look at the defect distribution you see that most of the faults are found in the early stages of development, up to sixty percent of the faults are found here. So a good rule of thumb is to test as soon as possible to prevent a long and costly rewriting process. [5]

When deciding who should perform the tests it is important to keep development and testing somewhat apart so that independent testing of the product is achieved. If a developer test his own code it is not very likely that he really finds all the faults because he is reluctant to find faults in his own work. It is often better that an independent tester performs the testing. This person could then attack the product with the goal to find all faults and all bugs. To get a good atmosphere between developer and tester it is though important that the developer understands and appreciates the work of the tester. To reach this, testers have to attack the product and not the developer in the testing process. [5]

The goals of the product decide when it is time to stop the testing process.

Almost none of the commercial products on the market today are flawless.

Instead the goal is to get the product as good as possible and the faults left should not create failures to the entire product and should not happen frequently. It is very hard to say exactly when to stop testing, on one hand you want a good product and on the other hand you want it on the market as soon as possible. You have to weigh these two against each other and reach a conclusion on when to stop the testing. [3]

From this we reach the conclusion that if you should do effective testing there are a few important principles to follow:

- Complete testing is not possible – try to find the critical sections - Testing is difficult – try to assign the best people to the testing


- Testing must be planned – try to cover as much as possible

- Testing requires independence – developers should not test their own code

There are great economical benefits in testing. If the testing is done properly the product has higher quality and the customers are more likely to be satisfied. The customers are not satisfied with a product that contains a lot of faults and they will not pay for such a product. This means there will not be any revenue and no profit. So we need the testing to help build a product that we can sell. If the product is tested and contains very few faults the customer is satisfied and pay the price we put on the product. [4, 5]

Another factor is the time spent on developing the product. It has been argued that testing actually can save time. The testing process takes time itself, but this is often saved in later stages which means time is actually saved. For instance if good testing is performed in the beginning of the development then there will be much fewer faults later in the development process. Faults in the later stages can be hard to trace and take a lot of time, so often you actually save time by doing intense testing during the development. [3]

Finally we have to take into account the resources and money spent on fixing the product after it has been released to the customers. If the product contains a lot of undetected faults much time and money have to be spent on writing patches to the product. The cost of this has to be taken into account when you calculate the total cost of the development. But with intense testing you can eliminate most of the faults and you have to spend much less resources on patching the product. [5]

Testing can be seen as a tool for achieving quality in the product, but quality is a difficult term to define and it is not certain that more testing gives the product a higher quality. When you measure quality it is important to realise that the customer is the only one who can determine if the product has the required quality or not. Testing can remove faults from the program and make the program run smoothly. Since all the tests are built on the specification of the product it is not certain for instance if the specification is very weak that testing gives the product a higher quality. It is equally important that the specification actually reflects what the customer wants. If the specification is a good reflection of what the customer wants then testing actually gives the product a higher quality as it turns the product towards the required behaviour. [6]

When quality is measured it is important to know that there are two basics for this. Of course the functionality of the program must be correct. The program must be able to do what the customers want it to do. But there is also the question of reliability. There is no use for a product that meets all of the requirements of the customer if it crashes all the time. So testing also has to make sure that the program is reliable and does not keep the user wondering if and when it actually works. [6]

An example of this according to us is “Windows 95” that had all the functionality but unfortunately the customers were not very satisfied with this product because it kept on crashing all the time. Microsoft had to try to


Chapter 3 • Testing – A General Background

fix this and came up with “Windows 98” which was slightly better. But it is not until recently when Microsoft developed “Windows XP” that they really focused on making the product reliably and this is also reflected in the customers who are more and more positive towards the product. The part of making the product reliable is part of the testing process and has to be tested just as thoroughly as the functionality.

3.2 Planning and Execution

3.2.1 Planning

In small projects with only limited software development planning of the tests is often not that important, but as soon as the project becomes medium- sized or even worse large, one of the most important points of the testing procedure is planning. Without planning the testing of large projects is very difficult. Planning greatly improves the effect of testing and at the same time minimise the time and money spent on testing. Another aspect is the fact that when the testing is done in a team every team member can express his views of the testing and finally all can agree and work after a plan. The workload can also easier be distributed between the members of the team. [6]

3.2.2 The Plan

When you plan the testing there are several important issues, which have to be taken into account. However first you have to have the specification for the project. All tests are built on the specification. The planning should start as soon as the specification is ready and preferably before the development starts. The reason for this is that the designing of tests often help to find faults in the specification and the specification can easily be revised if the development has not yet started.

When testing the first aspect to decide is what to actually test. It is important to decide what we are going to test and what we are not going to test. The test team have to evaluate every aspect and risk associated with not testing certain areas. The fact that some areas are not tested is often due to the fact that the test team has to prioritise because they do not have the resources to test it all. This aspect is often the most important and should be done as thoroughly as possible.

The second aspect is the time spent on testing. Important things to decide are: the amount of tests that should be done, the amount of time spent on each test and the amount of time spent on writing and correcting the tests. It is important that one try to evaluate at an early stage how much time should be spent on each of these things. Otherwise it is easy to get stuck on something while the time goes by. Especially important is the writing of the tests and the tests of these. If the tests are not ready the development is halted and this cost a lot. The test cases have to be ready when the developer is ready with a part to be tested. [3]

The third aspect is the test development, the distribution of the work tasks.

This is especially important if the team is large and there is a structure in the group. A project is often divided into smaller groups and each of these


groups implements some functionality; these parts of the program are called modules. It is important to get the testers to work close with the developers to achieve testing in an early stage in each module so that the integration of the project runs as smoothly as possible. When the test team is scattered it is harder to communicate, then it is much more important with the plan to avoid testing the same thing twice or miss testing something. [3]

The last aspect is when to stop testing. There are several issues to take into account when deciding that the testing of the product should come to an end.

Often the decision depends not only on the product but also on the time and cost. The testers have to ask themselves, what do we want to achieve? Can we achieve this within the given time and to a reasonable cost? Knowing this, criteria for when the tests are complete have to be decided.

The planning should result in a written test plan that all the testers agree on.

How extensive this report should be depends on whom the report is for.

Most of the times this report is an internal document used solely by the testers and for this purpose the report does not have to be that extensive.

Sometimes the report as well as the tests is shipped with the product to the customer and then the report must be very extensive and contain details of all the aspects mentioned above. [6]

A good approach is to let someone else look at the document before it is approved to see that all needs are covered.

There are no right or wrong on how much should be included in the test plan. Some testers decide just to include the planning as mentioned above while others like to extend the test plan to cover design, specifications, test bugs and more. The important issue however is to work out a common view on how the process should be developed so that all testers agree and work toward the same goal.

3.2.3 Test Report

To get a structured and effective response to tests it is important to make test reports that can be understood and maybe even appreciated by the developers. To do this there are certain standards that one should follow when writing a test report. Normally every company has made up own standards to follow but most of them include the simple rules described further in the next sections. [8]

A brief description of the system and the particular software that is tested should be included. The purpose of the document and contents of the report should also be covered. This part of the document is often very short and is only made to give the reader a general view of the system, software and tests. [8]

An overview of the testing itself also needs to be included. A short evaluation of how the test went should initiate this part. Then the discrepancies from the expected result and the limitations of the test should be shown. The overview also includes a part that describes what could not be tested and why. For every discrepancy three main things should be evaluated, the effect on the system, how much work there is to correct the


Chapter 3 • Testing – A General Background

discrepancy and a recommendation if the problem needs to be solved or more correctly the severity of the problem. [8]

A more detailed test result also needs to be included in order for the developers to be able to follow the tests when they are to correct the faults.

This part also gives managers a better chance to follow the results of the test.

The test log should be included in this material. A test log shows the test in chronological order and also where the test was performed. A description of what hardware and software were used and who performed the test should also be included to better be able to deal with the problem and recreate the faults at a later time. [8]

3.3 Validation and Verification

When a component or a product is ready it has to be checked to make sure that it works the way it is supposed to. The product has to be tested. There are two important terms to remember, validation and verification.

- Validation – are we building the right product?

- Verification – are we building the product right?

Boehm, 1979

3.3.1 Validation

Validation shows that the software does what the customer expects as distinct from what has been specified. This means that the purpose of validation besides satisfying the customer is to bring the SRS as close as possible to the customer’s expectations so that the developed product needs less revising. [7]

A good way to do this is to include the customer as early as possible. The customer could for instance revise and approve the SRS to get it as close as possible to what the customer wants. But it is also equally important to keep the customer updated during the development to see that the implementation of the demands in the SRS fulfils the customer’s expectations.

Sometimes it is not possible to include the customer, either because there is no specific customer due to the fact that the product is not developed under contract or because the customer does not want or has time to be included in the actual development. In the first case it could be a good idea to select a fictional customer, someone that can act as the customer; this is ideally one of the persons that decided on the making of the product though he knows what the product should do. In the second case it is important to make it clear to the customer that if the product should be as good as possible he has to participate in some extent during the development. Otherwise the product might not be what he expects and a lot of time has to be spent on revising which delays the delivery of the product. [5]

The last part of the validation is often done when the product is ready for shipping in the so-called acceptance test. This test is covered in more detail later but it is practically a test of the product preferably done by the customer before the product is released or shipped to see if the developed product


complies with the expectations of the customer. It is vital to get the best possible evaluation of the product that this test is done by the customer.

Always remember it is the customer that will use the product. If the customer is not satisfied then the product is not ready for shipping even if it according to the developers complies with all of the demands in the SRS.

This type of conflict is often due to different interpretations of the specification and can often be cleared by involving the customer in an early stage. [5]

3.3.2 Verification

The developers build the product based on the demands stated in the SRS.

To really know that the product complies with all of the demands in the SRS one have to verify this throughout all of the development process. Often the project is divided into phases and after each phase it is important to verify that the product met the demands set up in the SRS. The SRS should state what to be achieved in each phase and the verification after each phase should verify that this has been met and constitute the base for the next phase. It is very important that the verification work is continuous otherwise it is easy to continue the development on false grounds and correcting this at a later stage is very expensive. [4]

The verifications can be done in many different ways. Two of the more common are reviews and inspections. Methods for reviews are covered later and we focus a little more on inspections here. The developing company itself often performs verifications. Sometimes the company has their own test department to help them with the task and sometimes not. To improve the product it is though important to include both developers and independent people in the inspections, these people can either be test personnel or developers that developed another function.

Many different strategies for inspections are available. Below three of the available strategies are described. The first one is a formal inspection that is led by someone else than the producer of the document. This approach requires all of the participants to prepare by reading and reviewing the document. At the meeting all of the defects found during the preparation are discussed and the faults that are important are put into a report that forms the basis for the developers rework. No solutions to the problems are discussed on the meeting. The second one is walkthroughs that are led by the producer.

This form requires no preparation from the participants. At the meeting the entire document is walked through by the producer and during this the participants discuss and search for faults in the document. This often proves to be a very good approach and surprisingly many faults are found during walkthroughs. The third one is “Buddy checks” which is the least formal of the methods. In this method the document is handed to someone other than the producer and this person reads the document thoroughly. “Buddy checks” are the easiest approach but also the one that finds the fewest faults.

But it is still better than no checking at all. [5]


Chapter 3 • Testing – A General Background

3.3.3 IV & V - Independent Verification and Validation

Independent verification and validation includes a third party besides the customer and the producer, a testing company. This testing company performs all of the validation and verification. The testing company reports only to the customer and this gives the customer a great advantage in finding both faults and specification faults.

There are several benefits in using an independent testing company. The testing is often more thoroughly performed as the testing company is completely independent from the developing company. This gives increased protection from faults, identifies risks associated with the software and permits fewer latent faults. Unfortunately independent testing companies cannot be involved in all projects as they often are very expensive. The companies that perform this type of testing are few and the workload is often heavy so they charge a lot for their services. Companies often only use this for very critical programs as for instance flight control software for space shuttles used by NASA. [3, 10]

3.4 Testing Methods

3.4.1 Static and Dynamic Testing

In testing there are two different approaches when looking at the code. Static testing is done without executing the code. Instead you go through the code manually to find faults. Dynamic testing is done by actually executing the code and looking for faults. [6]

In static testing more code can be covered as you actually check the code without execution. This means that for instance syntax faults are eliminated.

Compilers can help a great deal with this part of the testing as they analyse the code without executing it. Static tests can also be performed by human testers in the form of inspections, walkthroughs and “Buddy checks” as described in the previous chapter. When this is done it is important that there is a discussion between the participants to find as many faults as possible. A problem with this type of testing is that it can be really hard to check the actual demands in the SRS when testing, instead you find logical faults. Due to this static testing must be followed by dynamic testing to actually test the demands. Another problem is to test complex programs using this approach, often you can find the basic faults but it can be very hard to find code and logical faults in the more complex parts of the solution. [1, 5]

In dynamic testing you actually execute the code as you test the program.

This means that you can design the test to cover all of the demands stated in the SRS. This is a formal review of the product and one of the most powerful tools in finding discrepancies between the product and the customer’s demands. There are different approaches to dynamic testing. The tests can be written either without looking at the code, called “Black-box” or by analysing the code called “White-box”. Dynamic testing can find logical faults much better than static testing. But it also has clear disadvantages. One problem is that it might not cover all of the code, how do we know that all branches, statements and cases have been executed when we only test the


demands (Black-box)? This approach might generate faults at a later stage because some of the code was not covered in the tests. In a complex program it can be very difficult to write tests that actually covers all of the code.

Some dynamical approaches actually test all the code but these methods often do not cover the demands but rather only the code (White-box). Much of this testing can be automated with different programs that executes test and generate test reports, so this approach might actually require less work than static testing. [1]

To get a good and thorough testing it is important to use both statically and dynamical testing even if this can take extra time. As described above both statically and dynamical testing have drawbacks but at different points. If you combine static and dynamic testing you often get the best result. A combination of the two can find both logical and syntactical faults and eliminate far more than any of the two can on its own. So a recommended approach is to try to find time for both methods in the test phases to be able to find as many faults as possible and get the best possible product.

3.4.2 Black-Box Testing

One method for testing is black-box testing. Black-box tests the specification without any knowledge of the implementation. This means that the only criterion for success in the testing are if the result is what is should be according to the SRS. The input is chosen very carefully to get the desired result. For each demand a test is designed and the output is compared with the expected one. If there are no discrepancies then the product is considered to be correct. [1]

Various flaws can arise when using this method. There is no way to be sure that all of the code is executed and that all of the cases in the code really is tested. This means that faults can arise at a later stage when the same demand is tried but under different conditions.

Another problem is that it can be very difficult to choose the correct input for the tests. There are several methods to facilitate this as for instance equivalence partitioning and boundary value testing. But even with the help of these it is not easy to write the input as good as possible without any knowledge of the code. A good input has to test as much as possible without actually writing test for all inputs. For instance in a program that is designed to only accept letters as input, one does not want to test all letters and all non-letters as inputs. [1]

Equivalence partitioning is one way to find good inputs. The method is based on dividing inputs into groups so called equivalence classes were all the inputs make the program behave in a comparable way. The groups can be chosen from the specification. When the groups are ready the tester chooses a few inputs from each group and tests these inputs. If these inputs give the correct output then all of the values in the group are considered to give the correct output and they are not actually tested. The chosen inputs should preferably be in the middle and on the boundaries of the classes. This method dramatically reduces the number of tests that needs to be performed.






Related subjects :