Software Quality Assurance: Introduction

Software Engineering Group

Software Quality

Assurance:

Introduction

Dr. Holger Giese

Dr. Holger Giese

Software Engineering Group

Software Engineering Group

Room E 3.165

Room E 3.165

Tel. 60

Tel. 60

-

-

3321

3321

Email:

Outline

I

Introduction

II

Software Life Cycle

III Quality Control

IV Infrastructure

V Management

VI Standards

I Introduction

I.1 Motivation

I.2 Definitions & Terminology

I.3 Quality Management

I.4 Components of a SQA System

I.5 Organizing for SQA

I.6 Discussion & Summary

I.7 Bibliography

I.1 Motivation

The Software Crisis:

„ IBM Consulting group estimates that 55% of large

distributed systems projects cost more than expected, 68% overrun their schedules, and 88% require redesign.

„ The Standish group estimated the cost of ‘bad software’ for US businesses at $85 billion for 1998.

„ The Y2K problem was estimated to cost $1 to $2 trillion.

„ W.W Gibbs, in “Software's Chronic Crisis” in the Scientific American, September 1994 estimates that the average software project overshoots its schedule by half.

Software's Chronic Crisis

Denver's new international air port was to be the pride of the Rockies, a wonder of modern engineering. Twice the size of Manhattan, 10 times the breadth of Heathrow, the airport is big enough to land three jets simultaneously-in bad weather. Even more impressive than its girth is the airport's subterranean baggage-handling system. Tearing like intelligent coal-mine cars along 21 miles of steel track, 4,000 independent "telecars" route and deliver luggage between the counters, gates and claim areas of 20 different airlines. A central nervous system of some 100 computers networked to one another and to 5,000 electric eyes, 400 radio receivers and 56 bar-code scanners orchestrates the safe and timely arrival of every valise and ski bag. At least that is the plan. For nine months, this Gulliver has been held captive by Lilliputians-errors in the software that controls its automated baggage system. Scheduled for takeoff by last Halloween, the airport's grand opening was postponed until December to allow BAE Automated Systems time to flush the gremlins out of its $193-million system. December yielded to March. March slipped to May. In June the airport's planners, their bond rating demoted to junk and their budget hemorrhaging red ink at the rate of $1.1 million a day in interest and operating costs, conceded that they could not predict when the baggage system would stabilize enough for the airport to open.

…

W. W. Gibbs. Software's chronic crisis. Scientific American, 271(3):72--81, September 1994.

Denver International Airport Baggage System

The Denver International Airport has a modern,

automated baggage-handling system designed by BAE

Automated Systems, Inc. (In June, 2003 G & T Conveyor Company, Inc. acquired BAE)

See:

Schloh, Michael. Analysis of the Denver International Airport baggage system

http://www.csc.calpoly.edu/~dstearns/SchlohProjec t/csc463.html

Eipe, Rohit. The importance of software architecture: Denver International Airport's automated baggage handling system: A report

http://wiki.cs.uiuc.edu/cs427/Essay+by+Rohit+Eipe :+Denver+Intnl+Airport:+Baggage+System

Nice, Karim. How Baggage Handling Works, HowBIZWorks

http://biz.howstuffworks.com/baggage-handling.htm

See:

Schloh, Michael. Analysis of the Denver International Airport baggage system

http://www.csc.calpoly.edu/~dstearns/SchlohProjec t/csc463.html

Eipe, Rohit. The importance of software architecture: Denver International Airport's automated baggage handling system: A report

http://wiki.cs.uiuc.edu/cs427/Essay+by+Rohit+Eipe :+Denver+Intnl+Airport:+Baggage+System

Nice, Karim. How Baggage Handling Works, HowBIZWorks

Automated Baggage-Handling System (1/3)

„ The baggage handling system at an airport plays a crucial role. It makes the difference in an airport's ability to attract or keep a major airline hub ("an airport that serves as a central connecting point)

„ A baggage-handling system has three main jobs:

† Move bags from the check-in area to the departure gate

† Move bags from one gate to another during transfers

† Move bags from the arrival gate to the baggage-claim area

„ Conveyors equipped with

junctions and sorting

machines automatically route the bags to the gate.

conveyer belt

check-in plane unloading

Automated Baggage-Handling System (2/3)

„

Destination-coded

vehicles

(DCVs),

unmanned carts

(also named

telecars)

† propelled by linear induction motors mounted to the tracks

† can load bags

(without stopping)

† can unload bags (without stopping)

unload DCV tunel system

a DCV (telecar)

Automated Baggage-Handling System (3/3)

DIA has:

„ More than 5 miles (8 km) of conveyor belts

„ More than 19 miles (30 km) of DCV tracks

„ About 4,000 DCVs

Ö enabling it to handle over 1,000 bags per minute.

load plane baggage-claim

Observation: First System Test (March 1994)

„ Telecars jumped tracks, crashed into each other; pieces of baggage were flung off telecars, in some cases chewed to pieces.

„ Baggage was sent to the wrong places.

„ Line balancing algorithm problems arose, where there were bags lined up waiting for telecars to come by, and yet empty telecars would just go by without picking up these bags.

„ In peak conditions, the system quickly became overloaded with tracking all the telecars in transit.

„ Head of line blocking occurred at popular telecar intersections in the

tunnel network, and rerouting of approaching telecars didn’t always take place.

„ Poorly printed baggage tags led to the laser scanners at one point rejecting 70% of all baggage tags, sending them to manual baggage stations.

Remark 1:This famous demo of the system in which hundreds of bags were destroyed was not done by BAE, but was done by the City of Denver when BAE told them not to do it because the system was not working. BAE claimed that the city violated the contract many times, making it nearly impossible for them to finish the job. Eventually the city quit managing the project and United Airlines took over, and BAE was able to finish the project.

Observation: Timeline (1/2)

„ The project was begun in the mid 80s by US Transportation Secretary Fredrico Pena, the previous Denver mayor.

„ Construction was supposed to begin in September 1989 with an opening date set for October 31st 1993.

„ On March 2nd, 1993 the first delay was announced, changing the date to December 19th, 1993.

„ October that year was the second delay.

„ March 1994 was the third.

„ After this Logplan was brought in, yet the fourth delay was announced on May 2nd, 1994.

Observation: Timeline (2/2)

„ DIA finally began operations sixteen months behind schedule, with a reduced number of gates operational,

concourse A being postponed indefinitely, the only having been implemented in concourse B, an inability to handle transfer flights, and with a capacity for handling only 30 bags a minute.

„ As of November 1996, DIA’s automatic baggage handling system was still not working. The airport was not even

increasing its air traffic; in 1995 the number of passengers dropped by 2 million from when Stapleton was running, in 1994. High fees associated with Denver charging each airline a $20 fee per passenger, which must have been passed on to the customer, are supposedly the reason for this disappointing performance.

Observation: Financial Damages

Automated baggage handling system:

„ Originally planed budget was $193 million

„ the amount finally spent was close to $311 million (included the cost of installing a manual system)

DIA

„ planed DIA costs were $1.7 billion

„ the final cost was more than 2.5 times that, at $4.5 billion Delay costs:

„ estimated costs for every month that the DIA remained closed was about $33.3 million

Cause: Lack of Management

„ The baggage handling system has been implemented almost as an afterthought! Thus the system of tunnels was not designed with a

particular baggage handling system in mind, and as it turned out, the small tunnels and many sharp turns made BAE’s job rather difficult.

„ The airport designers didn’t even design their system with a baggage handling system in mind, so BAE had to work around all kinds of

problems.

„ This was the reason that the management team never saw the baggage handling system, of all things, to be so important that it could have such a profound economic impact on the project.

„ The planning of the system was done in a completely haphazard way. Little communication took place between DIA’s airport designers, the city officials, the airlines, and BAE.

„ The planning instead was driven by higher level management with little or no understanding of the complexity of the system, and by consultants contracted to develop a specification of the system, but who had no

direct responsibility to the team(s) that would actually develop the system.

Cause: Impossible Schedule/Plan

„ One ‘reason’, if it might be called that, that BAE didn’t perform a review of their

own design was that they were being made to work under an impossible tight schedule.

„ BAE claim that they protested at the outset that the timeline Denver city officials proposed for the completion of the project was utterly unrealistic.

„ Denver on the other hand claims that BAE committed to delivering the system within a certain period of time, and should have delivered a bug free system in that time.

„ Whoever the culprit, one thing is for sure, the time required to fully understand and do justice to a system of this size and complexity was grossly

underestimated, and this had direct bearing on the efficiency of the design that was put forth, and ultimately implemented.

„ identify problems with the design, and take decisions about them early, instead of reacting to those problems after they were uncovered in testing

„ BAE and Denver didn’t heed the advice of others, and grossly underestimated the time for developing and testing the system (The Munich airport, on which DIA was modeled, spent two years testing the system and six months of running the system 24x7 before opening)

Cause: Complexity Problems

„ As it turned out, when a change needed to be made to one part of the system, it was not clearly understood how that change would impact the rest of the system. When the system was eventually test-run, BAE found it incredibly difficult to even understand why things were going wrong.

„ BAE’s design was such that the number of components working together was very high, and they were very tightly coupled, and there was little redundancy.

„ sheer size of the system made it so complex that not even the people who designed it were able to understand it very well, leave alone a third party

„ to weed out problems at the design stage than to implement a design that had rather glaring faults, and then try to

Cause: Many Interfaces

„

One extremely difficult problem that BAE

faced was the task of conversing with United

Airlines’Apollo reservation computers.

„

BAE’s computers had to speak the same

software language as each of the airlines

„

The process of language translation that was

necessitated was painful and bug-filled at

best.

DIA Conclusions

„

Missing communication between the stakeholders

„

The schedule didn’t allow enough time to fully

design the system before jumping into

implementation.

„

The design of the baggage handling system was

not integrated into the design of the airport.

Not entirely inappropriately, the DIA was coined as

being DOA – dead on arrival!

How can we circumvent such

disasters?

Observation:

„ Quality problems prevent the system from functioning

„ Management decisions results in wrong reactions

† Missing design reviews

† Fixing rather than analyzing

† Erratic debugging rather than systematic testing Therefore even larger delays result

Systematic approach for the development of high quality software is required

I.2 Definitions & Terminology

„

Software

† Definition & characterization

† Why do the classical approach to QA not apply?

„

Software quality

† Definition

† Quality models

† Quality attributes

„

Measure quality

† Reliability, Availability, safety, maintainability,

† Defect counts

† Software errors, faults and failures

Basic Terminology

Software is:

Computer programs, procedures, and

possibly associated documentation and data

pertaining to the operation of a computer

system.

What is special about Software?

„

Invisibility of the product

„

Limited opportunities to detect defects

(“bugs”)

„

Often new demanding functionality has to be

realized

„

Often software has to realize extraordinary

Classical Quality Assurance (Hardware)

„ Requirements: complete set of external quality characteristics Öcomplete set of internal quality characteristics and a detailed and complete set of requirements and specifications

„ Design:

† Design that satisfies the requirements and specifications

† design for reliability (wear out)

† design for manufacturability

† design for maintainability (e.g., self-diagnosis)

„ Manufacturing: statistical production process control with acceptance sampling

„ Operation: collect failure data for continuous improvement and predictions (intelligent maintenance)

Software Quality Assurance?

„

Requirements:

† Completeness is hard to achieve (complexity)

„

Design:

† design for reliability only in special cases

† design for manufacturability is not required

† design for maintainability in special cases

† Focal area of software quality assurance!

„

Manufacturing

Ö

Implementation:

† limited success with statistical process control (metrics)

„

Operation:

What is quality?

„

Quality, simplistically, means that a product

should meet its specification.

„

This is problematical for software systems

†

There is a tension between customer quality

requirements (efficiency, reliability, etc.) and

developer quality requirements (maintainability,

reusability, etc.);

†

Some quality requirements are difficult to specify

in an unambiguous way;

†

Software specifications are usually incomplete

and often inconsistent.

Approaches to Tackle Quality

„

Transcendental view

: quality is universally

identifiable, absolute, unique and perfect

„

Product view

: the quality of a product is

measurable in an objective manner

„

User view

: quality is fitness for use

„

Manufacturing view

: quality is the result of

the right development of the product

„

Value-based view (Economic)

: quality is a

Software Quality – IEEE View

Software quality is:

(1)

The degree to which a system, component,

or process meets specified requirements.

(2)

The degree to which a system, component,

or process meets customer or user needs

or expectations.

Software Quality

Software quality is :

Conformance to explicitly stated functional

and performance requirements, explicitly

documented development standards, and

implicit characteristics that are expected of

all professionally developed software.

Software Quality

Quality – the degree of excellence of something. We measure the excellence of software via a set of attributes.

[Glass1992] (≠ “satisfying requirements”)!

„ Quality is absolute or at least independent of the requirements underlying the product.

„ It is part of the software development to “get the right requirements”.

(≠ “user satisfaction”)!

Quality Models

Two main approaches:

„ Standard Models:

† McCall

† ISO/IEC 9126

„ Application or company specific quality models

† FURPS

† GQM Approach

„ Such general definitions of software quality are not

sufficient in practice

„ Thus, software quality is described by specific

quality models

„ „causal relationship from intangible quality views to tangible software

measures“

Factor-Criteria-Metrics-Model

Classification into :

„ Factors (to specify): They describe the external view of the software, as viewed by the users.

„ Criteria (to build): They describe the internal view of the software, as seen by the developer.

„ Metrics (to control): They are defined and used to provide a scale and

method for measurement.

Software Quality Quality criterion 1 Quality criterion 2 … Quality criterion 3 Quality factor 1 Quality factor 2 … Quality factor 3

[MaCall+1977] [Galin2004]

McCall’s Factor Model Tree

„ A quality factor represents a behavioral characteristic of the system. † Operation † Revision † Transition „ A quality criterion is an attribute of a quality factor that is related to software production and design.

„ A quality metric is a

measure that captures some aspect of a quality criterion.

The Six Quality Characteristics of

a Software (ISO/IEC 9126)

„ Software quality: The totality of

features and characteristics of a software product that bear on its ability to satisfy stated or implied needs. (ISO 9126: 1991, 3.11)

„ Software quality characteristics:

A set of attributes of a software product by which its quality is described and evaluated. A

software quality characteristic may be refined into multiple levels of sub-characteristics. (ISO 9126: 1991, 3.13)

„ Each characteristic is refined to a set of sub-characteristics

„ Each sub-characteristic is evaluated by a set of metrics.

„ Some metrics are common to several sub-characteristics.

Attributes of software that bear on the users’effort for operation and operation control. Operability

Attributes of software that bear on the users’effort for learning its application. Learnability

Attributes of software that bear on the users’ effort for recognizing the logical concept and its applicability.

Understandability Usability

Attributes of software that bear on the capability to re-establish its level of performance and recover the data directly affected in case of a failure and on the time and effort needed for it.

Recoverability

Attributes of software that bear on its ability to maintain a specified level of performance in case of software faults or of infringement of its specified interface. Fault tolerance

Attributes of software that bear on the frequency of failure by faults in the software. Maturity

Reliability

Attributes of software that bear on its ability to prevent unauthorized access, whether accidental or deliberate, to programs or data.

Security

Attributes of software that make the software adhere to application related standards or conventions or regulations in laws and similar prescriptions.

Compliance

Attributes of software that bear on its ability to interact with specified systems. Interoperability

Attributes of software that bear on the provision of right or agreed results or effects. Accurateness

Attributes of software that bear on the presence and appropriateness of a set of functions for specified tasks.

Suitability

Functionality

Definitions Subcharacteristics

Attributes of software that bear on opportunity and effort using it in the place of specified other software in the environment of that software.

Replaceability

Attributes of software that make the software adhere to standards or conventions relating to portability.

Conformance

Attributes of software that bear on the effort needed to install the software in a specified environment.

Installability

Attributes of software that bear on the opportunity for its adaptation to different

specified environments without applying other actions or means than those provided for this purpose for the software considered.

Adaptability

Portability

Attributes of software that bear on the effort needed for validating the modified software.

Testability

Attributes of software that bear on the risk of unexpected effect of modifications. Stability

Attributes of software that bear on the effort needed for modification, fault removal or for environmental change.

Changeability

Attributes of software that bear on the effort needed for diagnosis of deficiencies or causes of failures, or for identification of parts to be modified.

Analyzability Maintainability

Attributes of software that bear on the amount of resource used and the duration of such use in performing its function.

Resource behavior

Attributes of software that bear on response and processing times and on throughput rates in performances its function.

Time behaviour Efficiency

Definitions Subcharacteristics

Hewlett Packard: F.U.R.P.S. (1/2)

„

Result of a statistical project survey at Hewlett

Packard 1987 to improve its products:

Factors:

† Functionality: functions it performs, their generality and security

† Usability: aesthetics, consistency, documentation

† Reliability: frequency and severity of failure, accuracy of output

† Performance: response time, resource consumption

† Supportability: can it be extended, adapted, corrected?

„

FURPS is originally a company specific quality

model

Hewlett Packard: F.U.R.P.S. (2/2)

Quality Supportabilit y Performance Reliability Usability Functionality Testability Extensibility Adaptability Maintainability Compatibility Configurability Serviceability Installability Localizability Speed Efficiency Rsc. consumption Thruput Response time Frequ./severity of fail. Recoverability Predictability Accuracy Mean time to failure Human factors Aesthetics Consistency Documentation Feature set Capabilities Generality Security

GQM: Goal-Question-Metric

„

A measurement program can be more successful

if designed with the goals in mind.

„

GQM approach provides a framework with 3

steps:

1. List the major goals of the development/maintenance

project

2. Derive from each goal the questions that must be

answered to determine if the goals are being met

3. Decide what must be measured to answer the

questions adequately

GOAL: Evaluate effectiveness of coding standard

QUESTIONS: Who is using Standard?

What is coder

productivity? What is code _quality?

METRICS: Proportion of Coders -using standard -using language Experience of Coders -with standard -with language -with environment etc Code size Effort Errors…

Example

GQM Discussion

„ Benefits :

„ generates only those measures relevant to the goal

„ Several measurements may be needed to answer a single question.

„ A single measurement may apply to more than one question.

„ The goal provides the purpose for collecting the data.

„ Not evident from the GQM

„ The model needed to combine the measurement in a sensible way so that the question can be answered.

„ It must be supplemented by one or more models that express the relationship among the metrics. (equation definition is not clear)

„ Disadvantages:

„ Additional efforts to derive the goals and metrics

Measuring Quality?

„

User‘s view:

†

Reliability (number of failures over time)

†

Availability

†

Usability etc.

†

Often directly measurable

„

Manufacturer‘s view:

†

Defect counts

Reliability

Reliability

is the probability of a component, or

system, functioning correctly over a given period of

time under a given set of operating conditions.

Quantitative:

Reliability R

(

t

) is the probability that the system

will conform to its specification throughout a period

of duration

t

.

„

Note that t is important

„

If a system only needs to operate for ten hours at a

time, then that is the reliability target.

Availability

The

availability

of a system is the probability that

the system will be functioning correctly at any given

time.

Quantitative:

Availability A (

or

V)

is the percentage of time for

which the system will conform to its specification.

Safety

„

Safety

is a property of a system that it will

not endanger human life or the environment.

„

A

safety-related system

(

safety-critical

system

) in one by which the safety of

equipment or plant is assured.

Maintainability

„

Maintenance

is the action taken to retain a

system in, or return a system to, its designed

operating conditions.

Maintainability

is the

ability of a system to be maintained (ability to

undergo repairs and modifications).

„

Mean time to repair (MTTR)

Defect Counts

„ „software errors are the cause of poor software quality“

[Galin2004]

„ In software, higher quality (in the form of lower defect rates) and reduced development time go hand in hand.

[McConnell 1996]

„ Unlike the low-defect kind of quality, attention to this kind of quality tends to lengthen the development schedule.

[McConnell 1996]

„ Intuitively, the occurrence of defects is negatively related to functionality and reliability.

Defects also interfere, to some degree, with other dimensions of quality.

[Pressman2004]

Faults, Errors and Failures

„ A fault is a defect within the system.

(Error cause, German: Fehlerursache)

„ An error is a derivation of the required operation of the system or subsystem.

(Manifestation of a fault in a system, German: Fehlzustand)

„ software errors are a human action which results in software containing a fault

„ A system failure occurs when the system fails to perform its required function.

(Transition to incorrect service delivery, German: Ausfall)

„ If the distinction between fault and failure is not critical, defect can be used as a generic term.

Recursive Nature of Faults

User Operator Physical process/system _System System (server) component (server) component (server) component (server) component (server) component (server)

?

?

?

?

?

„

A fault in a component can “infect” all other

components which depend on it.

Fault/Failure Chain

Fault Ö error

„ a fault which has not been activated by the computation process is

dormant

„ a fault is active when it produces an error Error Ö failure

„ an error is latent when it has not been recognized

„ an error is detected by a detection algorithm/mechanism Failure Ö fault

„ a failure occurs when an error “passes through” and affects the service delivered

„ a failure results in a fault for the system which contains or interacts with the component

Fault Error Failure Fault Error Failure

Examples for Fault/Failure Chain

Program error (software):

„ a dormant fault in the written software (instruction or data)

„ upon activation the fault becomes active and produces an

error (system state)

„ if the erroneous data affects the delivered service, a failure

occurs

Electromagnetic interference (hardware):

„ leads to faulty input value (either digital or analog)

„ by reading the input the fault becomes active and produces an error

„ if the erroneous input value is processed and becomes visible at the interface a failure occurs

Fault Classification

Nature (Critical distinction):

„

random/hardware faults

„

logical/systematic/design faults

E.g., Degradation (wear-out) vs. design faults

Duration:

„

permanent, transient, intermittent

Extent:

Nature: Degradation Faults

„

The system used to work but now it does not

„

Something broke for some reason:

† Infant mortality

† End of useful life—wear out

† Physical damage—vibration, humidity, temperature

Examples:

„

Light bulb burns out

„

Disk head crashes

„

Power supply fails

Nature: Failure “Bathtub Curve”

Infant mortality

Nature: Design Faults

„

The system never worked

„

Nothing broke

„

The design is flawed

Examples:

„

Incorrect electrical wiring

Dealing With Faults

Fault Prevention:

„

Development techniques that prevent the

introduction of faults

Fault Elimination:

„

Development techniques that find and remove

faults already introduced

Fault Forecasting:

„

Estimate current number, future incidence and

likely consequences

Fault Tolerance (not considered here):

„

Execution-time techniques that cope with the

Fault Avoidance

Activities:

„ Preventing the introduction or occurrence of faults

(fault prevention):

† SWT, formal methods, high level languages, CASE tools, …

„ Reducing the number and seriousness of faults

(fault removal):

† Analysis Ö diagnosis Ö correction

„ Evaluating the present number future incidents, and severity of faults (fault forecasting):

† Statistics & management int i; while(i<10){ if (i>3) { } … } Fault removal Fault prevention Fault forecasting

[Galin2004]

Main causes for software faults:

1.

Faulty requirements definition

2.

Client-developer communication failures

3.

Deliberate deviations from software requirements

4.

Logical design errors

5.

Coding errors

6.

Non-compliance with documentation and coding

instructions

7.

Shortcomings of the testing process

8.

User interface and procedure errors

I.3 Quality Management

Quality Management System [ISO 9000]:

The organizational structure, responsibilities,

procedures, processes and resources for

implementing quality management

„

Concerned with ensuring that the required level of

quality is achieved in a software product.

„

Involves defining appropriate quality standards

and procedures and ensuring that these are

followed.

„

Should aim to develop a ‘quality culture’ where

quality is seen as everyone’s responsibility.

Environment Characteristics

„ Being contracted

„ Subjection to customer-supplier relationship

„ Requirement for teamwork

„ Need for cooperation and coordination with other development teams

„ Need for interfaces with other software systems

„ Need to continue carrying out a project while the team changes

„ Need to continue

maintaining the software system for years

The Cost of Quality

„ Cost of Quality includes all costs incurred in the pursuit of quality or in performing quality related activities such as appraisal costs, failure costs and external failure costs.

[Pressman2004]

The Quality Compromise

:

„ We cannot wait for specifications to improve before paying attention to quality management.

„ We must put quality management procedures into place to improve quality in spite of imperfect specification.

Scope of Quality Management

„

Quality management is particularly

important for large, complex systems. The

quality documentation is a record of

progress and supports continuity of

development as the development team

changes.

„

For smaller systems, quality management

needs less documentation and should focus

on establishing a quality culture.

Quality Management and

Software Development

Software development process Quality management process D1 D2 D3 D4 D5 Standards and

Quality Management Activities

(1)

Quality assurance

† Establish organisational procedures and standards for quality.

(2)

Quality planning

† Select applicable procedures and standards for a particular project and modify these as required.

(3)

Quality control

† Ensure that procedures and standards are followed by the software development team.

„

Quality management should be

separate

from

project management to ensure independence.

(1)

Quality Assurance

Quality Assurance [ISO 9000]:

All those planned and systematic actions necessary to

provide adequate confidence that a product or service will satisfy requirements for quality

Software quality assurance [IEEE]:

1. A planned and systematic pattern of all actions necessary to provide adequate confidence that an item or product conforms to established technical requirements.

2. A set of activities designed to evaluate the process by which the products are developed or manufactured. Contrast with: quality control.

Quality Assurance

Software quality assurance is [Galin2004] :

A systematic, planned set of actions necessary to provide adequate confidence that the software development

process or the maintenance process of a software system product conforms to established functional technical

requirements as well as with the managerial requirements of keeping the schedule and operating within the budgetary confines.

„ Quality assurance consists of the auditing and reporting functions of management [Pressman2004]

(2)

Quality Planning

„

Quality planning

is the process of

assessing the requirements of the

procedure and of the product and the

context in which these must be observed.

„

Quality assurance plan

is the central aid

for planning and checking the quality

assurance.

Quality Assurance Plan

„

A quality assurance plan sets out the desired

product qualities and how these are assessed and

defines the most significant quality attributes.

„

The quality assurance plan should define the

quality assessment process.

„

It should set out which organisational standards

should be applied and, where necessary, define

new standards to be used.

Quality Assurance Plans

„

Quality assurance plan structure:

†

Product introduction

†

Product plans

†

Process descriptions

†

Quality goals

†

Risks and risk management

„

Quality assurance plans should be

†

short,

†

succinct

„ Purpose of Plan

„ References

„ Management

† organization structure, SQA tasks, their placement in the process

† roles and responsibilities related to product quality

„ Documentation

† project documents, models, technical documents, user documents.

„ Standards, Practices and Conventions

„ Reviews and Audits

„ Test

† test plan and procedure

„ Problem Reporting and Corrective action

„

Tools, Techniques and

Methodologies

„

Code Control

„

Media Control

„

Supplier control

„

Records Collection,

Maintenance and

Retention

„

Training

„

Risk Management

[IEEE_Std_730-1998, Pressman2004]

(3)

Quality Control

Quality Control [ISO 9000]:

The operational techniques and activities

that are used to fulfil requirements for quality

„

Quality Control

is the series of inspections, reviews

and tests used throughout the development cycle

to ensure that each work product meets the

requirements placed upon it.

Quality Control

„

This involves checking the software

development process to ensure that

procedures and standards are being

followed.

„

There are two approaches to quality control

†

Quality reviews;

†

Automated software assessment and software

measurement.

Quality Control

Objective:

„

minimize the produced defects

„

increase the product quality

Implementation approaches:

„

Fully automated

„

Entirely manual

„

Combination of automated tools and human

Quality Control

Quality control includes a feedback loop to the process:

„ provide management with the necessary data about product quality.

„ gain the insight and confidence of product quality Two types of quality control:

„ Quality design: the characteristics that designers specify for an item (includes: requirements, specifications, and the design of the system).

„ Quality of conformance: the degree to which the design specification are followed. It focuses on implementation based on the design.

Quality Assurance System

„

Quality assurance system

is the organizational

structure, responsibilities, procedures, processes

and resources for implementing quality

management.

„

The quality of a developed product is influenced

by the quality of the production process.

„

This is important in software development as

some product quality attributes are hard to

assess.

„

However, there is a very complex and poorly

understood relationship between software

processes and product quality.

Process and Product Quality

Process-based Quality

„

There is a straightforward link between process

and product in manufactured goods.

„

More complex for software because:

† The application of individual skills and experience is particularly important in software development;

† External factors such as the novelty of an application or the need for an accelerated development schedule may impair product quality.

„

Care must be taken not to impose inappropriate

process standards - these could reduce rather

than improve the product quality.

„

Define process standards such as how

reviews should be conducted, configuration

management, etc.

„

Monitor the development process to ensure

that standards are being followed.

„

Report on the process to project

management and software procurer.

„

Don’t use inappropriate practices simply

because standards have been established.

I.4 Components of a SQA System

(1)

Pre-project components

(2)

Software project life cycle components

(3)

Infrastructure components for error prevention

and improvements

(4)

Management SQA components

(5)

SQA standards, system certification and

assessment components

(6)

Organizing for SQA – the human components

and considerations guiding construction of

(1)

Pre-project Components

Pre-project

„

Contract reviews

„

Development and quality plans

(2)

Project Life Cycle Components

Development

„

Reviews

„

Expert opinions

„

Software testing

„

Assurance of the quality of external participants’

work

Maintenance

„

Software maintenance components

(see Chapter II and III)

(3)

Infrastructure Components

„

Procedures and work instruction

„

Templates and checklists

„

Staff training, retraining and certification

„

Preventive and corrective actions

„

Configuration management

„

Documentation control

(see Chapter IV)

(4)

Management SQA Components

„

Project progress control

„

Software quality metrics

„

Software quality costs

(5)

Standards, Certification, Assessment

„

Project process standards

„

Quality management standards

Objectives:

„

Utilization of international professional knowledge

„

Improvement of coordination with other

organizations’ quality systems

„

Objective professional evaluation and

measurement of the organization’s SQA

achievement

(6)

Organizing for SQA

„

Management’s role in SQA

„

The SQA unit

„

SQA trusties

„

SQA committees

„

SQA forums

Project

Development plan and Quality Plan

(2) Project Life Cycle SQA components

Fo rmal Desig n Revie w s Ex pe rt s O pi ni on Pe er R ev ie w s SQ A o f E xt er na l P ar tic ip an ts So ftw ar e M ai nt en an ce So ftw ar e Te stin g

(3) Quality Infrastructure components

Procedures Supporting _{Devices Instruction}Training Preventive_Actions ConfigurationManagement

Document-ation Control (4) Quality Management Project Progress Control Software Quality Metrics Software Quality Costs Quality Management Standards (5) Standards Project Process Standards (6) Organizational Base – Human components

Management SQA Unit SQA Trustees SQA Committees SQA Forums Contract review

(1) Pre-pro

ject SQA c

omponents

Project

Development plan and Quality Plan

(2) Project Life Cycle SQA components

Fo rm al D es ig n R ev ie w s Ex pe rt s O pi ni on Pe er R ev ie w s SQ A o f E xt er na l P ar tic ip an ts So ftw ar e M ai nt en an ce So ftw ar e Te stin g

(3) Quality Infrastructure components

Procedures Supporting _{Devices Instruction}Training Preventive_Actions ConfigurationManagement

Document-ation Control (4) Quality Management Project Progress Control Software Quality Metrics Software Quality Costs Quality Management Standards (5) Standards Project Process Standards (6) Organizational Base – Human components

Management SQA Unit SQA Trustees SQA Committees SQA Forums Contract review

(1) Pre-pro

ject SQA c

omponents

[Galin2004]

I.5 Organizing for SQA

a) Management b) SQA Unit c) SQA Trustees d) SQA Committees e) SQA Forums a) Management b) SQA Unit c) SQA Trustees d) SQA Committees e) SQA Forums

[Galin2004] Management SQA unit Software Development and Maintenance Department Other Departments SQA Trustees SQA Committees Legend Line of authority line for SQA issues Flow of Forum ’s recommendations line Executive responsible for software quality Exec. Exec . Exec. Software Development Teams Software Testing Department Software Testing Teams SQA Forums

The SQA Framework: Participants

Managers

„ Top management executives, especially the executive in charge of SQA

„ Software development and maintenance department managers

„ Software testing department managers

„ Project managers and team leaders of development and maintenance projects

„ Leaders of software testing teams

Testers

„ Members of software testing teams

SQA professionals and interested practitioners

„ SQA trustees

„ SQA committee members

„ SQA forum members

a)

Management

Overview:

„

Top management’s quality assurance activities

„

Software quality policy

„

The executive in charge of software quality

„

Management review

„

Department management responsibilities for

quality assurance processes

„

Project management responsibilities for quality

assurance

TOP Management Responsibilities

„ Assure the quality of the Company’s software products and software maintenance services.

„ Communicate the importance of product and service

quality in addition to customer satisfaction to employees.

„ Assure full compliance with customer requirements.

„ Ensure that SQA objectives are established and accomplished.

„ Initiate planning and oversee implementation of changes to adapt the SQA system to changes related to the

organization's clientele, competition and technology.

„ Intervene directly to resolve of crisis situations and minimize damages.

„ Ensure availability of resources required by SQA systems.

SQ Policy Requirements

„ Quality policy refers to the basic aims and objectives of an organization regarding quality as stipulated by the management.

[Pressman2004]

Conformity to the organization purpose and goals Commitment to:

„ General software quality assurance concepts

„ The quality standards adopted by the organization

„ Allocate adequate resources for software quality assurance

„ Continuous improvement of the organizations quality and productivity

Responsibilities (Executive in Charge)

„

Responsibility

for preparation of an annual

SQA activities program and budget

„

Responsibility

for preparation of SQA system

development plans

„

Overall control

of implementation of the

annual SQA regular activities program and

planned SQA development projects

„

Presentation

and advocacy of SQA issues to

executive management

Management Reviews

Def.: Management review is the name given to the periodic meeting convened to allow executives to obtain an overview of their

organization’s software quality issues. Typical items:

„ Periodic performance reports, including quality metrics

„ Customer satisfaction feedback

„ Follow up reports for SQA annual regular activity program and SQA development projects

„ Summary of special quality events related to customers, suppliers, subcontractors, etc.

„ Review of significant findings of internal and external quality audits as well as special surveys

„ Identification of new software quality risks and unsolved pre-existing risks

„ Recommendations for software quality management improvements.

Management Reviews: Objectives

„

Assess

achievement of quality objectives set

for the organization’s software quality

management system

„

Initiate

updates and improvements of the

software quality management system and its

objectives

„

Outline directions

for remedying major SQA

deficiencies and software quality management

problems.

„

Allocate

additional resources to the software

Department Responsibilities (1/2)

The quality system-related responsibilities:

„

Preparation

of the department’s annual SQA

activities program and budget, based on

recommended SQA unit program.

„

Preparation

of the department’s SQA systems

development plans, based on recommended

SQA unit plan.

„

Control

of performance of the department’s

annual SQA activities program and development

projects

„

Presentation

of the department's SQA issues to

Department Responsibilities (2/2)

Project-related responsibilities

„ Control of compliance to quality assurance procedures in the department's units

„ Detailed follow up of contract review results and proposal approvals

„ Review of unit performance of planned review activities; approval of project documents and project phase completion

„ Follow up of software tests; approval of project’s software products.

„ Follow up of progress of software development project schedules and budget deviations. Advise and support project mangers in

resolving difficulties.

„ Follow up of quality of maintenance services

„ Detailed follow up of project risks and their solutions

„ Follow up of project's compliance with customer requirements and customers satisfaction.

„ Approval of large software change orders and significant deviations from project specifications.

Project Management Responsibilities

Professional hands-on tasks:

„ Preparation of project and quality plans and their updates.

„ Participation in joint customer-supplier committee

„ Close follow up of project team staffing, including recruitment, training and instruction.

Management tasks The follow up issues:

„ Performance of review activities and the consequent corrections, including participating in some reviews.

„ Software development and maintenance units’ performance with respect to development, integration and system test activities, corrections and regression tests and acceptance tests

„ Software installation in customer sites and the running-in of the software system by the customer

„ SQA training and instruction of project team members

„ Schedules and resources allocated to project activities.

„ Customer requests and satisfaction

„ Evolving project development risks, application of solutions and control of results.

b)

The SQA Unit

Overview:

„ Activities

„ Responsibilities

„ Tasks performed by the head of the SQA unit

„ SQA sub-unit tasks related to the project life cycle

„ SQA sub-unit infrastructure operations tasks

„ SQA sub-unit audit and certification tasks

„ SQA sub-unit support tasks

„ SQA sub-unit standards and procedures: Development and maintenance tasks

SQA Unit Tasks

„ Quality assurance planning oversight, record keeping, analysis and reporting

„ Participates in the development of the projects software process

„ Reviews software engineering activities to verify compliance with the defined software process.

„ Audits designated software work products to verify

compliance with those defined as part of the software process.

„ Ensures that deviations in software work and work products are documented and handled according to a document

procedure.

„ Records any noncompliance and reports to senior management.

Unit: Organizational Structure

[Galin2004] Head SQA Unit SQA Development and Maintenance SQA Operations SQA Standards and Procedures SQA Information Systems SQA Engineering Internal and Certification SQA Audits Project Life Cycle SQA SQA Support SQA Infrastructure Operations

SQA Unit Head Tasks (1/2)

Planning tasks

„ Preparation of proposals for the Unit’s annual activity program and budget

„ Planning and updating the organization’s software quality

management system and recommended annual SQA activities programs for the software development and maintenance

departments.

„ Preparation of recommended SQA systems development plans for the software development and maintenance departments.

Management tasks

„ Management of SQA team's activities

„ Monitoring implementation of the SQA activity program

„ Nomination of team members, SQA committee members and SQA trustees

„ Preparation of special and periodic status and performance reports.

SQA Unit Head Tasks (2/2)

Contacts with customers and other external bodies and the executive in charge of software quality

„ Serving as the customer’s address for software quality issues of software products and services supplied

„ Representation of the organization before external bodies regarding software quality issues

„ Drafting the management review reports

„ Raising SQA organizational issues and preparing requested material for top management's consideration

SQA professional activities

„ Participation in project joint committees

„ Participation in formal design reviews

„ Review and approval of deviations from specifications

„ Consultation to project managers and team leaders

„ Participation in SQA committees and forums

Life Cycle Tasks (Sub-Units)

Project life cycle control tasks

„ Follow up of development and maintenance teams’ compliance with SQA procedures and work instructions

„ Approval or recommendation of software products (design reports and code).

„ Monitoring delivery of software maintenance services to internal and external customers

„ Monitoring customer satisfaction (surveys, etc.) and maintaining contact with customer’s SQA representatives

Participation tasks participation in:

„ Contract reviews

„ Preparation and updating of project development and project quality plans

„ Formal design reviews

„ Subcontractors’ formal design reviews

„ Software testing, including customer acceptance tests

„ Software acceptance tests of subcontractors’ software products

„ Installation of new software products

Infrastructure Tasks (Sub-Units)

„ Publication of updated versions of procedures, work instructions, templates, checklists, etc., with their circulation.

„ Training and instruction to new and current staff and SQA trustees regarding SQA procedures, work instructions, new and revised procedures, development tools and methods, etc.

„ Monitoring and supporting implementation of new and revised SQA procedures

„ Follow up of staff certification activities

„ Proposal of subjects requiring preventive and corrective actions

„ Follow up of configuration management activities

„ Follow up of compliance with documentation procedures and work instructions

Types of Audits (in or by SW Org)

„

Internal audits

„

Audits of subcontractors and suppliers to

evaluate their SQA systems

„

External audits performed by certification

bodies

„

External audits performed by customers

who wish to evaluate the SQA system prior

to accepting the organization as a supplier

Audits and Certifications (Sub-Units)

„ Preparation of annual programs for SQA audits

„ Performance of SQA audits

„ Follow up of corrections

„ Preparation of periodic summary reports

„ Collection of data on the performance of the audited organization from internal and external sources

„ Periodic evaluation of the audited organization

„ Coordination of the external audit's contents and schedule

„ Preparation of documents as specified by external auditors

„ Instruction of the audited teams and performance of preparations for external audits

„ Participation in the audit

Support Tasks (Sub-Units)

„

Preparation of project development plans and

project quality plans

„

Staffing review teams

„

Choice of development methodologies and tools

that reflect the accumulated failure experience

„

Choice of measures to solve identified software

development risks

„

Choice of measures to solve schedule delays

and budget overruns

„

Choice of SQA metrics and software costs

components

„

Use of SQA information systems

Standard and Procedures (Sub-Units)

„ Prepare an annual program for development of new procedures and procedure updates

„ Responsibility for development of new procedures and procedure updates, including participation in appropriate committees and forums

„ Follow up of developments and changes in SQA and

software engineering standards; introduction of additional relevant procedures and changes

„ Initiation of updates and adaptations of procedures in

response to changes in professional standards, including adoption or deletion of standards applied by the

organization.

Engineering (Sub-Units)

„ Testing quality and productivity aspects with respect to new development tools and new versions of currently used development tools

„ Evaluation of quality and productivity of new and improved development and maintenance methods

„ Development of solutions to difficulties confronted in

application of currently used software development tools and methods

„ Development of methods for measuring software quality and team productivity

„ Provision of technological support to CAB committees during analysis of failures and formulation of solutions

Information Systems (Sub-Units)

„

Development of SQA information systems for

software development and maintenance units for:

†

Collection of activity data.

†

Processing of information delivered by the

units: periodic reports, lists, exception

reports, queries and estimates of software

quality metrics and software quality costs.

„

Updating of SQA information systems

„

Development and maintenance of the

organization's SQA Intranet/Internet site

SQA Unit Plan

„

Evaluations to be performed

„

Audits and reviews to be performed

„

Standards that are applicable to the project

„

Procedures for error reporting and tracking

„

Documents to be produced by the SQA group

„

Amount of feedback provided to software

c)

SQA Trustees

Unit-related tasks:

„ Support their colleagues' attempts to solve difficulties in the implementation of SQA procedures and work instructions

„ Help their unit manager in performing his or her SQA tasks Promote compliance and monitor implementation of SQA procedures and work instructions by colleagues

„ Report substantial and systematic non-compliance events to the SQA unit

„ Report severe software quality failures to the SQA unit

Organization-related tasks

„ Initiate changes and updates of organization-wide SQA procedures and work instructions

„ Initiate organization-wide improvements of development and

maintenance processes and applications for solutions to recurrent failures observed in their units

„ Identify organization-wide SQA training needs and propose an appropriate training or instruction program

d)

SQA Committees

Permanent committees commonly deal with:

„ SCC (software change control),

„ CA (corrective actions),

„ Procedures,

„ Development of method, tools and quality metrics.

Ad-hoc committees commonly deal with specific cases:

„ Updates of a specific procedure,

„ Analysis and solution of a software failure,

„ Elaboration of software metrics for a targeted process or product,

„ Updating software quality costs,

„ Data collection methods for a specific issue.

e)

SQA Forums

SQA forums typically focus on:

„ SQA procedures improvements and implementation

„ Quality metrics

„ Corrective actions – analysis of failure and success cases

„ Quality system issues – development and implementation of new tools

„ Quality line management problems – daily operational software quality problems

Members of an open forum may include:

„ SQA unit members

„ SQA trustees

„ Software development and maintenance staff

„ SQA and