Software Testing

(1)

CHAPTER-1

INTRODUCTION TO

SOFTWARE TESTING

(2)

INSIDE THIS CHAPTER

 IntroductionIntroduction

 The Testing ProcessThe Testing Process

 What is Software Testing?What is Software Testing?

 Why Should We Test? What is the Purpose?Why Should We Test? What is the Purpose?  Who Should Do Testing?Who Should Do Testing?

 What Should We Test?What Should We Test?

 Selection of Good Test CasesSelection of Good Test Cases

 Measurement of the Progress of TestingMeasurement of the Progress of Testing  Incremental Testing ApproachIncremental Testing Approach

 Basic Terminology Related to Software TestingBasic Terminology Related to Software Testing  Testing Life CycleTesting Life Cycle

 When to Stop Testing?When to Stop Testing?  Principles of TestingPrinciples of Testing  Limitations of TestingLimitations of Testing

(3)

INTRODUCTION

Testing is the processes of executing the program Testing is the processes of executing the program with the intent of finding faults. Who should do with the intent of finding faults. Who should do this testing and when should it start are very this testing and when should it start are very important questions that are answered in the important questions that are answered in the text. As we know that software testing is the text. As we know that software testing is the fourth phase of Software Development Life Cycle fourth phase of Software Development Life Cycle (SDLC). About 70% of development time is spent (SDLC). About 70% of development time is spent on testing. We explore this and many other on testing. We explore this and many other

interesting concepts in this chapter. interesting concepts in this chapter.

(4)

THE TESTING PROCESS



Please note that testing starts from

requirements analysis phase only and

goes till the last maintenance phase.



Static Testing wherein the SRS is tested

to check whether it is as per user

requirements or not.



Dynamic Testing starts when the code is

ready or even a unit (or module) is

ready.

(5)

WHAT IS SOFTWARE TESTING?

 The concept of software testing has evolved from simple program The concept of software testing has evolved from simple program

“check-out” to a broad set of activities that cover the entire software

life-cycle.

 There are five distinct levels of testing that are given below:There are five distinct levels of testing that are given below:  Debug: Debug: It is defined as the successful correction of a failure.It is defined as the successful correction of a failure.

 Demonstrate:Demonstrate: The process of showing that major features work with The process of showing that major features work with

typical input.

 Verify: Verify: The process of finding as many faults in the application under The process of finding as many faults in the application under

test(AUT) as possible.

 Validate: Validate: The process of finding as many faults in requirements, The process of finding as many faults in requirements,

design and AUT.

 Prevent: Prevent: To avoid errors in development of requirements, design and To avoid errors in development of requirements, design and

implementation by self-checking techniques, including “test before

design”.

(6)



“

Testing is the process of exercising or

evaluating a system or system

component or automated means to

verify that it satisfies specified

requirements”.



“

Software testing is the process of

executing a program or system with

intent of finding errors.”



“

It involves any activity aimed at

evaluating an attribute or capability of

a program or system and determining

that it meets its required results.”

(7)

“

Testing Is Basically A Task Of

_{Testing Is Basically A Task Of}

Locating Errors”



It may be:



Positive

Testing:

Operate

application it should be operated.



Negative

Testing:

Testing

for

abnormal operations.



Positive View of Negative Testing:

The job of testing is to discover errors

before the user does.

(8)

SOFTWARE VERIFICATION

 ““It is the process of evaluating a system or It is the process of evaluating a system or

component to determine whether the products component to determine whether the products of a given development phase satisfy the of a given development phase satisfy the

conditions imposed at the start of that phase.” conditions imposed at the start of that phase.”

 ““It is the process of evaluating, reviewing, It is the process of evaluating, reviewing,

inspecting and doing desk checks of work inspecting and doing desk checks of work products such as requirement specifications, products such as requirement specifications,

design specifications and code.” design specifications and code.”

 ““It is human testing activity as it involves It is human testing activity as it involves

looking at the documents on paper.” looking at the documents on paper.”

(9)

SOFTWARE VALIDATION



“

It is the process of evaluating a

system or component during or at

the end of development process to

determine whether it satisfies the

specified requirements. It involves

executing the actual software. It is

a computer based testing process.”

(10)

LEAP YEAR FUNCTION

 CREATE FUCTION f_is_leap_year (@ ai_year small CREATE FUCTION f_is_leap_year (@ ai_year small

int)

RETURNS small int

AS

BEGIN

-- if year is illegal(null or –ve), return -1

IF (@ ai_year IS NULL) or IF (@ ai_year IS NULL) or (@ ai_year <= 0) RETURN -1 (@ ai_year <= 0) RETURN -1 IF (((@ ai_year % 4) = 0) AND IF (((@ ai_year % 4) = 0) AND ((@ ai_year %100)< > 0)) OR ((@ ai_year %100)< > 0)) OR ((@ ai_year %400) = 0) ((@ ai_year %400) = 0)

RETURN 1 –leap year

RETURN 0 – Not a leap year

END

(11)

DATABASE TABLE: TEST_LEAP_YEAR

Serial No.

Serial No. Year(year to Year(year to test)

test) Expected resultExpected result Observed resultObserved result MatchMatch

1 1 -1-1 -1-1 -1-1 YesYes 2 2 -400-400 -1-1 -1-1 YesYes 3 3 100100 00 00 YesYes 4 4 10001000 00 00 YesYes 5 5 18001800 00 00 YesYes 6 6 19001900 00 00 YesYes 7 7 20102010 00 00 YesYes 8 8 400400 11 11 YesYes 9 9 16001600 11 11 YesYes 10 10 20002000 11 11 YesYes 11 11 24002400 11 11 YesYes 12 12 44 11 11 YesYes 13 13 12041204 11 11 YesYes 14 14 19961996 11 11 YesYes 15 15 20042004 11 11 YesYes

(12)

WHY SHOULD WE TEST? WHAT

IS THE PURPOSE?



The Technical Case



The Business Case



The Professional Case



The Economics Case



To Improve Quality



For Verification and Validation (V&V)



For Reliability Estimation

(13)

WHO SHOULD DO TESTING?

 Testing starts right from the very beginning.Testing starts right from the very beginning.  This implies that testing is everyone’s This implies that testing is everyone’s

responsibility. responsibility.

 It is a It is a Team EffortTeam Effort..

 Even Developers are responsible.Even Developers are responsible.

 They build the code but do not indicate any They build the code but do not indicate any

errors as they have written their own code. errors as they have written their own code.

(14)

WHAT SHOULD WE TEST?

 Consider that there is a while loop that has Consider that there is a while loop that has three paths. If this loop is executed twice, three paths. If this loop is executed twice, we have (3*3) paths and so on. So, the total we have (3*3) paths and so on. So, the total number of paths through such a code will number of paths through such a code will

be: be:

 = 1+3+(3*3)+(3*3*3)+….= 1+3+(3*3)+(3*3*3)+…. = 1+∑3= 1+∑3n_n

 This means an infinite number of test cases. This means an infinite number of test cases. Thus, testing is not 100% exhaustive.

(15)

SELECTION OF GOOD TEST

CASES

 According to Brian Marick, ”A test idea is a According to Brian Marick, ”A test idea is a brief statement of something that should be brief statement of something that should be

tested.” tested.”

 Cem Kaner said-”The best cases are the one Cem Kaner said-”The best cases are the one that find bugs.”

that find bugs.”

 A test case is a question that you ask of the A test case is a question that you ask of the program. The point of running the test is to program. The point of running the test is to gain information like whether the program will gain information like whether the program will

pass or fail the test. pass or fail the test.

(16)

MEASUREMENT OF TESTING

 A good project manager(PM) wants that worse A good project manager(PM) wants that worse

conditions should occur in the very beginning of

the project only than in the later phases.

 There is no standard to measure our testing There is no standard to measure our testing

process.

 But metrics can be computed at the organizational, But metrics can be computed at the organizational,

process project and product levels.

 Metrics is assisted by four core components- Metrics is assisted by four core components-

schedule, quality, resources and size.

(17)

INCREMENTAL TESTING APPROACH

 Stage 1:Stage 1: Exploration. Exploration.

 Purpose:Purpose: To gain familiarity with the application. To gain familiarity with the application.  Stage 2:Stage 2: Baseline test. Baseline test.

 Purpose:Purpose: To devise and execute a simple test case. To devise and execute a simple test case.  Stage 3:Stage 3: Trends analysis. Trends analysis.

 Purpose: Purpose: To evaluate whether the application performs as expected when To evaluate whether the application performs as expected when

actual output can be predetermined. actual output can be predetermined.

 Stage 4: Stage 4: Inventory.Inventory.

 Purpose:Purpose: To identify the different categories of data and create a test for To identify the different categories of data and create a test for

each category item. each category item.

 Stage 5:Stage 5: Inventory combinations. Inventory combinations.

 Purpose:Purpose: To combine different input data. To combine different input data.  Stage 6: Stage 6: Push the boundaries.Push the boundaries.

 Purpose: Purpose: To evaluate application behavior at data boundaries.To evaluate application behavior at data boundaries.  Stage 7:Stage 7: Devious data. Devious data.

 Purpose: Purpose: To evaluate system response when specifying bad data.To evaluate system response when specifying bad data.  Stage 8: Stage 8: Stress the environmentStress the environment..

(18)

BASIC TERMINOLOGY RELATED TO SOFTWARE

TESTING

 Error(or Mistake or Bugs): Error(or Mistake or Bugs): When people make mistakes while coding, we call When people make mistakes while coding, we call

these mistakes bugs.

 Fault (or Defect): Fault (or Defect): A missing or incorrect statement(s) in a program resulting A missing or incorrect statement(s) in a program resulting

from an error is a fault.

 Failure: Failure: A failure occurs when a fault executes.A failure occurs when a fault executes.

 Incident: Incident: An incident is a symptom associated with a failure that alerts the user An incident is a symptom associated with a failure that alerts the user

to the occurrence of a failure.

 Test: Test: A test is the act of exercising software with test cases.A test is the act of exercising software with test cases.

 Test Case: Test Case: The essence of software testing is to determine a set of test cases The essence of software testing is to determine a set of test cases

for the item to be tested.

 Test Suite: Test Suite: A collection of test scripts or test cases that is used for validating A collection of test scripts or test cases that is used for validating

bug fixes within a logical or physical area of a product.

 Test Script: Test Script: The step-by-step instructions that describe how a test case is to be The step-by-step instructions that describe how a test case is to be

executed.

 Test Ware: Test Ware: It includes all testing documentation created during testing process.It includes all testing documentation created during testing process.  Test Oracle: Test Oracle: Any means used to predict the outcome of a test.Any means used to predict the outcome of a test.

 Test Log:Test Log: A chronological record of all relevant details about the execution of a A chronological record of all relevant details about the execution of a

test.

 Test Report:Test Report: A document describing the conduct and results of testing carried A document describing the conduct and results of testing carried

out for a system.

(19)

Requirements Specification Design Coding Testing Fault Resolution Fault Isolation Fault Classification Error Error Error Error Fault Fault Fault Incident Fix TESTING LIFE CYCLE

(20)

WHEN TO STOP TESTING?



The

pessimistic approach

to stop

testing is whenever some or any of the

allocated resources- time, budget or test

cases are exhausted.



The

optimistic stopping rule

is to stop

testing when either reliability meets the

requirement,

or

the

benefit

from

requirement,

or

the

benefit

from

continuing testing cannot justify the

testing cost.

(21)

PRINCIPLES OF TESTING

 Testing should be based on User Requirements.Testing should be based on User Requirements.  Testing Time and Resources are Limited.Testing Time and Resources are Limited.

 Exhaustive Testing is impossible.Exhaustive Testing is impossible.  Use Effective Resources to Test.Use Effective Resources to Test.

 Test Planning Should be Done Early.Test Planning Should be Done Early.

 Testing should begin “in small” and Progress Testing should begin “in small” and Progress

Toward Testing “in large”. Toward Testing “in large”.

 All tests should be traceable to customer All tests should be traceable to customer

requirements. requirements.

 Prepare test reports including test cases and Prepare test reports including test cases and

test results to summarize the results of testing. test results to summarize the results of testing.

(22)

LIMITATIONS OF TESTING

 Testing can show presence of errors-not their absence.Testing can show presence of errors-not their absence.  No matter how hard you try, you would never find the No matter how hard you try, you would never find the

last bug in an application.

 The domain of possible inputs is too large to test.The domain of possible inputs is too large to test.

 There are too many possible paths through the There are too many possible paths through the programs to test.

programs to test.

 In short, maximum coverage through minimum test-In short, maximum coverage through minimum test-cases. That is the challenge of testing.

cases. That is the challenge of testing.

 Various testing techniques are complementary in Various testing techniques are complementary in nature and it is only through their combined use that

nature and it is only through their combined use that

one can hope to detect most errors.

(23)

AVAILABLE TESTING TOOLS,

TECHNIQUES AND METRICS

 Mothora: Mothora: It is an automated mutation testing tool-It is an automated mutation testing tool-set developed at Purdue university. The tester can

set developed at Purdue university. The tester can

create and execute test cases, measure test case

adequacy.

 NuMega’s Bounds Checker, Rational’s Purify: NuMega’s Bounds Checker, Rational’s Purify: They are run-time checking and debugging aids.

They are run-time checking and debugging aids.

 Ballista COTS Software Robustness Testing Ballista COTS Software Robustness Testing

Harness(Ballista):

Harness(Ballista): It is full-scale automated It is full-scale automated robustness testing tool. The goal is to automatically

robustness testing tool. The goal is to automatically

test and harden commercial off-the-shalf (COTS)

software against robustness failures.

(24)

SUMMARY

 Software testing is an art. Most of the testing methods Software testing is an art. Most of the testing methods

and practices are not very different from 20 years ago.

Good Testing also requires a tester’s creativity,

experience and intuition together with proper

techniques.

 Testing is more than just debugging. It is also used in Testing is more than just debugging. It is also used in

validation, verification process and reliability

measurement.

 Testing is expensive Automation is a good way to act Testing is expensive Automation is a good way to act

down cost and time.

 Complete testing is infeasible. Complexity is the root of Complete testing is infeasible. Complexity is the root of

the problem.

 Testing may not be the most effective method to Testing may not be the most effective method to

improve software quality.

(25)

CHAPTER-2

SOFTWARE VERIFICATION AND

VALIDATION

(26)

INSIDE THIS CHAPTER

 IntroductionIntroduction

 Differences Between Verification and ValidationDifferences Between Verification and Validation  Differences between QA and QC?Differences between QA and QC?

 Evolving Nature of AreaEvolving Nature of Area  V&V LimitationsV&V Limitations

 Categorizing V&V TechniquesCategorizing V&V Techniques

 Role of V&V in SDLC- Tabular FormRole of V&V in SDLC- Tabular Form

 Proof of Correctness (Formal Verification)Proof of Correctness (Formal Verification)  Simulation and PrototypingSimulation and Prototyping

 Requirements TracingRequirements Tracing

 Software V&V Planning (SVVP)Software V&V Planning (SVVP)  Software Technical Reviews (STRs)Software Technical Reviews (STRs)  Independent V&V Contractor (IV & V)Independent V&V Contractor (IV & V)

(27)

INTRODUCTION

 The evolution of software that satisfies its user The evolution of software that satisfies its user

expectations is a necessary goal of a successful expectations is a necessary goal of a successful

software development organization. software development organization.

 To achieve this goal, software engineering To achieve this goal, software engineering

practices must be applied throughout the practices must be applied throughout the

evolution of the software product. evolution of the software product.

 Most of these practices attempt to create and Most of these practices attempt to create and

modify software in a manner that maximizes modify software in a manner that maximizes the probability of satisfying its user the probability of satisfying its user

expectations. expectations.

(28)

DIFFERENCES BETWEEN

VERIFICATION AND

VALIDATION



Software V&V is “a systems engineering

process

employing

a

rigorous

process

employing

a

rigorous

methodology

for

evaluating

the

methodology

for

evaluating

the

correctness and quality of software

product through the software life cycle”.

(29)

Verification

Verification ValidationValidation It is a static process of verifying

It is a static process of verifying

documents, design and code.

documents, design and code. It is a dynamic process of validating/testing the actual project.It is a dynamic process of validating/testing the actual project. It does not involve executing the

It does not involve executing the

code.

code. It involves executing the code.It involves executing the code.

It is human based checking of

documents/files.

documents/files. It is computer based execution of program.It is computer based execution of program. Target is requirements specification,

Target is requirements specification,

application architecture, high level

and detailed design, database

design.

Target is actual product- a unit, a

module, a set of integral modules,

final product.

It uses methods like inspections,

walk throughs, Desk-checking etc.

walk throughs, Desk-checking etc. It uses methods like black box, gray box, white box testing etc.It uses methods like black box, gray box, white box testing etc. It, generally, comes first- done

It, generally, comes first- done

before validation.

before validation. It generally follows verification.It generally follows verification.

It answers to the question- Are we

building the product right?

building the product right? It answers to the question- Are we building the right product?It answers to the question- Are we building the right product? It can catch errors that validation

It can catch errors that validation

cannot catch.

(30)

DIFFERENCES BETWEEN

QA & QC?



Quality Assurance:

The planned and

systematic activities, implemented in a

quality system so that the quality

requirements for a product or service will

be fulfilled, is known as quality assurance.



Quality

Control:

The

observation

techniques and activities used to fulfill

requirements for quality is known as

quality control.

(31)

Quality Assurance

(QA)

(QA) Quality Control (QC)Quality Control (QC)

It is process related.

It is process related. It is product related. It is product related. It focuses on the It focuses on the process use d to process use d to develop a product. develop a product. It focuses on testing of It focuses on testing of a product developed or a product developed or a product under a product under development. development. It involves the quality

It involves the quality of the processes

of the processes It involves the quality It involves the quality of the products

of the products It is a preventive

It is a preventive control

control It is a detective control.

It is a detective control. Allegiance is to

Allegiance is to development

(32)

Evolving Nature of Area

 As the complexity and diversity of software As the complexity and diversity of software

products continue to increase, the challenge products continue to increase, the challenge to develop new and more effective V&V to develop new and more effective V&V strategies continues. The V&V approaches strategies continues. The V&V approaches that were reasonably effective on small that were reasonably effective on small batch – oriented products are not sufficient batch – oriented products are not sufficient for concurrent, distributed or embedded for concurrent, distributed or embedded products. Thus, this area will continue to products. Thus, this area will continue to evolve as new research results emerged in evolve as new research results emerged in

response to new V&V challenges. response to new V&V challenges.

(33)

V&V Limitations



Theoretical foundations



Impracticality of testing all data



Impracticality of testing all paths



No absolute proof of correctness.

(34)

Categorizing V&V Technique

(35)

Role of V&V in SDLC

 Traceability analysisTraceability analysis

 Interface analysisInterface analysis  Criticality analysis Criticality analysis

 Step 1: Construct a block diagram or control flow diagram Step 1: Construct a block diagram or control flow diagram (CFD) of the system and its elements. Each block will

(CFD) of the system and its elements. Each block will

represent one software function (or module) only.

 Step 2: Trace each critical function or quality requirement Step 2: Trace each critical function or quality requirement through CFD.

through CFD.

 Step 3:Classify all traced software functions as critical toStep 3:Classify all traced software functions as critical to  Proper execution of critical software functions.Proper execution of critical software functions.

 Proper execution of critical quality requirements.Proper execution of critical quality requirements.

 Step 4: Focus additional analysis on these traced critical Step 4: Focus additional analysis on these traced critical software functions.

software functions.

 Step 5: Repeat criticality analysis for each lifecycle Step 5: Repeat criticality analysis for each lifecycle process to determine whether the implementation details

process to determine whether the implementation details

shift the emphasis of the criticality.

(36)

Hazard and Risk Analysis

V&V V&V Activity

Activity V&V Tasks V&V Tasks Key IssuesKey Issues

Requirements

V&V

 Traceability analysisTraceability analysis  Software requirements Software requirements

evaluation.

 Interface analysisInterface analysis  Criticality analysis Criticality analysis  System V&V test plan System V&V test plan

generation.

 Acceptance V&V test Acceptance V&V test Plan generation Plan generation  Configuration Configuration management management assessment assessment

 Hazard analysis Hazard analysis  Risk Analysis Risk Analysis

 Evaluates Evaluates the the correctness, correctness, completeness, accuracy,

completeness, accuracy,

constituency, testability and

readability of software

requirements.

 Evaluates Evaluates the the software software interfaces

interfaces

 Identifies the criticality of each Identifies the criticality of each software function.

software function.

 Initiates the V&V test planning Initiates the V&V test planning for V &V system test.

for V &V system test.

 Ensures Ensures completeness completeness and and adequacy of S CM process.

adequacy of S CM process.

 identifies potential hazards, identifies potential hazards, based on the product data

based on the product data

during the specified

development activity.

 Identifies potential risks, based Identifies potential risks, based on the product data during the

on the product data during the

specified development activity.

(37)

Design V&V

Design V&V  Traceability analysisTraceability analysis  Software design Software design

evaluation

 Interface analysis.Interface analysis.  Criticality analysisCriticality analysis

 Component V&V test Component V&V test plan generation and

plan generation and

verification.

 Integration V&V test Integration V&V test plan generation and

plan generation and

verification

 Hazard analysis Hazard analysis

 Risk analysis Risk analysis

 Evaluates Evaluates software software design design modules for correctness,

modules for correctness,

completeness, accuracy,

consistency, testability and

readability.

 Initiates the V&V test planning Initiates the V&V test planning for V&V component test.

for V&V component test.

 Initiates the V&V test planning Initiates the V&V test planning for V&V integration test.

for V&V integration test.

Implementati

on V&V

 Traceability analysisTraceability analysis

 Source code and source Source code and source code documentation

code documentation

evaluation

 Interface analysisInterface analysis  Criticality analysisCriticality analysis  V&V test case V&V test case

generation and

verification

 V&V test procedure V&V test procedure generation and

generation and

verification.

 Component V&V test Component V&V test execution and

execution and

verification.

 Hazard analysisHazard analysis  Risk analysisRisk analysis

 Verifies Verifies the the correctness, correctness, completeness, consistency,

completeness, consistency,

accuracy, testability and

readability of source code.

(38)

Test V&V

Test V&V  Traceability analysis Traceability analysis  Acceptance V&V test Acceptance V&V test

procedure

 Integration V&V Test Integration V&V Test execution and

execution and

verification

 System V&V test System V&V test execution and execution and verification verification Maintenance Maintenance V&V V&V  SVVP (software SVVP (software verification and verification and

validation plan) revision

 Propose3d change Propose3d change assessment

assessment

 Anomaly evaluationAnomaly evaluation  Criticality analysis Criticality analysis  Migration assessmentMigration assessment  Retirement assessmentRetirement assessment  Hazard analysisHazard analysis

 Risk analysis Risk analysis

 Modifies the SVVP. Modifies the SVVP.

 Evaluates the effect of software Evaluates the effect of software of operation anomalies

of operation anomalies

 Verifies the correctness of Verifies the correctness of software when migrated to a

software when migrated to a

different operational

environment.

 Ensures Ensures that that the the existing existing system continues to function

system continues to function

correctly when specific

software elements are retired.

(39)

Proof of Correctness (Formal

Verification)

 A proof of correctness is a mathematical proof that a computer A proof of correctness is a mathematical proof that a computer

program or a part thereof will, when executed, yield correct

results i.e., results fulfilling specific requirements.

 Hypothesis: The hypothesis of such a correctness theorem is Hypothesis: The hypothesis of such a correctness theorem is

typically a condition that the relevant program variables must

satisfy immediately before the program is executed. This

condition is called the

condition is called the preconditionprecondition. .

 Thesis: The thesis of the correctness theorem is typically a Thesis: The thesis of the correctness theorem is typically a

condition that the relevant programme variables must satisfy

immediately after execution of the program. This latter

condition is called the

condition is called the post conditionpost condition. .

 So the correctness theorem is stated as follows: So the correctness theorem is stated as follows:

 If the condition, V, is true before execution of the program, S, If the condition, V, is true before execution of the program, S,

then the condition, P, will be true after execution of S”.

 Notation: such a correctness theorem is usually written as Notation: such a correctness theorem is usually written as

{V}S{P}, where V, S and P have been explained above.

(40)

Simulation and Prototyping

 Simulation and prototyping are techniques for Simulation and prototyping are techniques for

analyzing the expected behavior of a product. There

are many approaches to constructing simulations and

prototypes that are well documented in the literature.

 For V&V purposes, simulations and prototypes are For V&V purposes, simulations and prototypes are

normally used to analyze requirements and

specifications to insure that they reflect the user’s

needs. Since they are executable, they offer additional

insight into the completeness and correctness of these

documents.

 Simulations and prototypes can also be used to analyze Simulations and prototypes can also be used to analyze

predicted product performance, especially for

candidate product designs, to insure that they conform

to requirements.

(41)

Requirements Tracing

 It is a technique for insuring that the product as well It is a technique for insuring that the product as well as the testing of the product, addresses each of its

as the testing of the product, addresses each of its

requirements.

 One type of matrix maps requirements to One type of matrix maps requirements to

software modules.

 Another type of matrix maps requirements to test Another type of matrix maps requirements to test

cases. Construction and analysis of this matrix

can help insure that all requirements are properly

tested.

 A third type of matrix maps requirements to their A third type of matrix maps requirements to their

evaluation approach. The evaluation approaches

may consist of various levels of testing, reviews,

simulations etc.

(42)

Software V&V Planning

 Step 1: Identification of V&V GoalsStep 1: Identification of V&V Goals  Step 2: Selection of V&V techniques Step 2: Selection of V&V techniques

 During requirements phase: The applicable techniques for During requirements phase: The applicable techniques for

accomplishing the V&V objectives for requirements are –

technical reviews, prototyping and simulations. The review

process is often called as a System Requirement Review (SRR).

 During Specifications Phase: the applicable techniques for this During Specifications Phase: the applicable techniques for this

phase are technical reviews, requirements tracing, prototyping

and simulations. The requirements must be traced to the

specifications.

 During Design Phase: The techniques for accomplishing the V&V During Design Phase: The techniques for accomplishing the V&V

objectives for designs are technical reviews, requirements

tracing, prototyping, simulation and proof of correctness. We can

go for 2 types of design reviews:

 High level designs High level designs  Detailed designsDetailed designs

 During implementation phaseDuring implementation phase  During maintenance phase During maintenance phase

(43)

 Step 3: Organizational responsibilitiesStep 3: Organizational responsibilities

 Developmental OrganizationDevelopmental Organization

 Independent Test Organization (ITO)Independent Test Organization (ITO)

 Software Software quality quality assurance assurance (SQA) (SQA)

organizations organizations

 Independent V&V ContractorIndependent V&V Contractor

 Step 4: integrating V&V ApproachesStep 4: integrating V&V Approaches  Step 5: Problem TrackingStep 5: Problem Tracking

 Step 6: Tracking Test Activities Step 6: Tracking Test Activities  Step 7: Assessment Step 7: Assessment

(44)

Software Technical

Reviews



Error prone software development and

maintenance process



Inability to test all software



Reviews are a form of testing



Reviews are away of tracking a project



Reviews provide feedback

(45)

Types of STRs

Informal Reviews

Informal Reviews Formal Reviews Formal Reviews It is a type of review that typically

It is a type of review that typically

occurs spontaneously among peers

occurs spontaneously among peers It is a planned meeting It is a planned meeting

Reviewers have no responsibility

Reviewers have no responsibility Reviewers are held accountable for Reviewers are held accountable for their participation in the review.

their participation in the review.

No review reports are generated

No review reports are generated Review reports containing action Review reports containing action items is generated and acted upon.

(46)

Review Methodologies



There are three approaches to

reviews



Walk Through



Inspection (or work product reviews),

and

(47)

Independent V&V Contractor

 Technical Independence: Technical Independence: It requires that members It requires that members of the IV & V team (organization or group) may not

of the IV & V team (organization or group) may not

be personnel involved in the development of the

software.

 Managerial Independence: Managerial Independence: It means that the It means that the responsibility for IV & V belongs to an organization

responsibility for IV & V belongs to an organization

outside the contractor and program organizations

that develop the software.

 Financial Independence: Financial Independence: It means that control of It means that control of the IV & V budget is retained in an organization

the IV & V budget is retained in an organization

outside the contractor and program organization

that develop the software.

(48)

Positive Effects of Software V&V on

Projects

 Better quality of software. This includes factors like Better quality of software. This includes factors like completeness, consistency, readability and testability

completeness, consistency, readability and testability

of the software.

 More stable requirementsMore stable requirements

 More rigorous development planning, at least to More rigorous development planning, at least to interface with the software V&V organization.

interface with the software V&V organization.

 Better adherence by the development organization to Better adherence by the development organization to programming language and development standards

programming language and development standards

and configuration management practices.

 Early error detection and reduced false starts. Early error detection and reduced false starts.

 Better Better schedule schedule compliance compliance and and progress progress monitoring.

monitoring.

 Greater project management visibility into interim Greater project management visibility into interim technical quality and progress.

technical quality and progress.

 Better criteria and results for decision making at Better criteria and results for decision making at formal reviews and audits.

(49)

Negative Effects of Software V&V

on Projects

 Additional project cost of software V&V (10-30% extra) Additional project cost of software V&V (10-30% extra)

 Additional interface involving the development team, user Additional interface involving the development team, user and software V&V organization.

and software V&V organization.

 Additional documentation requirements, beyond the Additional documentation requirements, beyond the deliverable products, if software V&V is receiving

deliverable products, if software V&V is receiving

incremental program and documentation releases.

 Need to share computing facilities with and to provide Need to share computing facilities with and to provide access to, classified data for the software V&V organization.

access to, classified data for the software V&V organization.

 Lower development staff productivity if programmers and Lower development staff productivity if programmers and engineers spend time explaining spend time explaining the

engineers spend time explaining spend time explaining the

system to software V&V analysis, especially if explanations

are not documented.

 Increased paper work to provide written responses to Increased paper work to provide written responses to software V&V error reports and other V&V data

software V&V error reports and other V&V data

requirements.

(50)

Standard for Software Test

Documentation (IEEE 829)

 The IEEE829 standard for software test The IEEE829 standard for software test

documentation describes a set of basic documentation describes a set of basic software test documents. It defines the content software test documents. It defines the content

and form of each test document. and form of each test document.

 In this addendum we give a summary of the In this addendum we give a summary of the

structure of the most important IEEE829 structure of the most important IEEE829

defined test documents. defined test documents.

 This addendum is based on the course This addendum is based on the course

materials by Jukka Paakki (and the IEEE829 materials by Jukka Paakki (and the IEEE829

standard) standard)

(51)

TEST PLAN

 Test plan identifierTest plan identifier  Introduction_Introduction

 Test itemsTest items

 Features to be testedFeatures to be tested  Features not to be testedFeatures not to be tested  Approach Approach

 Item pass / fail criteriaItem pass / fail criteria

 Suspension criteria and resumptionSuspension criteria and resumption  Test deliverables_{Test deliverables}

 Testing tasksTesting tasks

 Environmental needsEnvironmental needs  ResponsibilitiesResponsibilities

 Staffing and training needs_{Staffing and training needs}  ScheduleSchedule

 Risks and contingencies Risks and contingencies  Approvals _Approvals

(52)

Test Case Specification



Test Case Specification Identifier



Test Items



Input Specifications



Output Specifications



Environmental Needs



Special Procedural requirements



Intercase Dependencies

(53)

Test Incident Report (Bug

Report)

 Bug report identifierBug report identifier  SummarySummary

 Bug description Bug description

 Inputs Inputs

 Expected results Expected results  Actual resultsActual results  Data and timeData and time

 Test procedure step Test procedure step  EnvironmentEnvironment

 Repeatability Repeatability  TestersTesters

 Other observers Other observers

 Additional information Additional information

(54)

Test Summary Report



Test Summary Report Identifier



Summary



Variances



Comprehensiveness Assessment



Summary of Result



Evaluation



Summary of Activities



Approvals

(55)

CHAPTER-3

BLACK BOX

TESTING TECHNIQUES

(56)

INSIDE THIS CHAPTER

 Introduction to Black Box (or functional testing)Introduction to Black Box (or functional testing)  Boundary Value analysis (BVA)Boundary Value analysis (BVA)

 Equivalence class testingEquivalence class testing

 Decision table based testingDecision table based testing

 Cause effect graphing techniqueCause effect graphing technique

 Comparison on black box (for functional) Comparison on black box (for functional) testing techniques

testing techniques  Kiviat Charts Kiviat Charts

(57)

INTRODUCTION



The term black box refers to the software

which is treated as a black box.



The system or source code is not checked

at all. It is done from customer’s

viewpoint.



The test engineer engaged in black box

testing only knows the set of inputs and

expected outputs and is unaware of how

those inputs are transformed into outputs

by the software.

(58)

Boundary Value Analysis (BVA)

 It is a black box testing technique that believes and extends It is a black box testing technique that believes and extends

the concept that the density defect is more towards the

boundaries. This is done to the following reasons:

 Programmers usually are not bale to decide whether they Programmers usually are not bale to decide whether they

have to use <= operator or < operator when trying to make

comparisons.

 Different terminating conditions of for-loops, while loops and Different terminating conditions of for-loops, while loops and

repeat loops may cause defects to move around the

boundary conditions.

 The requirements themselves may not be clearly The requirements themselves may not be clearly

understood, especially around the boundaries thus causing

even the correctly coded program to not perform the correct

way.

(59)

What is BVA?

 The basic idea of BVA is to use input variable values at The basic idea of BVA is to use input variable values at

their minimum, just above the minimum, a nominal

value, just below their maximum and at their

maximum.

 BVA is based upon a critical assumption that is known BVA is based upon a critical assumption that is known

as single fault assumption theory.

 We derive the test cases on the basis of the fact that We derive the test cases on the basis of the fact that

failures are not due to simultaneous occurrence of two

faults. So, we derive test case by holding the values of

all but one variable at their nominal values and letting

that variable assume its extreme values.

(60)

Limitations of BVA



Boolean and logical variables present a

problem for Boundary value analysis.



BVA assumes the variables to be truly

independent which is not always possible.



BVA test cases have been found to be

rudimentary because they are obtained

with very little insight and imagination.

(61)

Robustness Testing

 Another variant to BVA is robustness testing. In BVA, we Another variant to BVA is robustness testing. In BVA, we

are within the legitimate boundary of our range. That

is, we consider the following values for testing:

{min, min+, nom, max-, max, max+}

 Again, with robustness testing, we can focus on Again, with robustness testing, we can focus on

exception handling. With strongly typed languages,

robustness testing may be very awkward. For example

in, pascal, if a variable is defined to be within a certain

range, values outside that range result in run time

errors that abort normal execution.

(62)

WORST CASE TESTING

 We reject our basic assumption of single fault We reject our basic assumption of single fault

assumption theory and focus on what assumption theory and focus on what happens when we reject this theory – it happens when we reject this theory – it simply means that we want to see that what simply means that we want to see that what happens when more than one variable has an happens when more than one variable has an

extreme value. extreme value.

 This is a multiple path assumption theory. In This is a multiple path assumption theory. In

electronic circuit analysis, this is called as electronic circuit analysis, this is called as

worst case analysis. worst case analysis.

(63)

Test case for the triangle

problem

 Before we generate the test case, firstly we need Before we generate the test case, firstly we need

to give the problem domain: to give the problem domain:

 Problem Domain: the triangle program accepts Problem Domain: the triangle program accepts

three integers, a, b and c as input. These are three integers, a, b and c as input. These are taken to be the sides of a triangle. The integers taken to be the sides of a triangle. The integers

a, b and c must satisfy the following conditions: a, b and c must satisfy the following conditions:

 CC

1

1 : 1 : 1 << a a << 200 200CC44 : a < b + c : a < b + c

 CC₂₂ : 1 : 1 << b b << 200 200CC₅₅ : b < a + c : b < a + c  CC₃₃ : 1 : 1 << c c << 200 200CC₆₆ : c < a + b : c < a + b

 The output of the program may be: Equilateral, The output of the program may be: Equilateral,

Isosceles, Scalene or “not a triangle” Isosceles, Scalene or “not a triangle”

(64)

How to generate BVA Test

Cases?

 We know that our range is [1,200] where 1 is We know that our range is [1,200] where 1 is

the lower bound and 200 being the upper the lower bound and 200 being the upper bound also, we find that this program has bound also, we find that this program has

three inputs – a, b and c. So, for our case three inputs – a, b and c. So, for our case

 BVA yields (4n+1) test cases, so we can say BVA yields (4n+1) test cases, so we can say

that the total number of test cases will be that the total number of test cases will be

(4=3+1) = 12 + 1 = 13. (4=3+1) = 12 + 1 = 13.

 We draw the table now which shows those 13 We draw the table now which shows those 13

test-cases. test-cases.

(65)

BVA test cases for triangle

problem

Case Id

Case Id aa bb cc Expected Expected output output 1 1 100100 100100 11 Isosceles Isosceles 2 2 100100 100100 22 IsoscelesIsosceles 3 3 100100 100100 100100 EquilateralEquilateral 4 4 100100 100100 199199 IsoscelesIsosceles 5

5 100100 100100 200200 Not a triangleNot a triangle 6 6 100100 100100 100100 IsoscelesIsosceles 7 7 100100 11 100100 IsoscelesIsosceles 8 8 100100 22 100100 EquilateralEquilateral 9 9 100100 100100 100100 IsoscelesIsosceles 10

10 100100 199199 100100 Not a triangleNot a triangle 11 11 11 200200 100100 IsoscelesIsosceles 12 12 22 100100 100100 IsoscelesIsosceles 13 13 100100 100100 100100 EquilateralEquilateral 14 14 199199 100100 100100 IsoscelesIsosceles 15

(66)

Guidelines for BVA

 the normal versus robust values and the single the normal versus robust values and the single

fault versus the multiple-fault assumption fault versus the multiple-fault assumption theory result in better testing. These methods theory result in better testing. These methods can be applied to both input and output domain can be applied to both input and output domain

of any program. of any program.

 Robustness testing is a good choice for testing Robustness testing is a good choice for testing

internal variables. internal variables.

 Keep in mind that you can create extreme Keep in mind that you can create extreme

boundary results from non-extreme input boundary results from non-extreme input

values. values.