Welcome!
Teaching Users to Test
Kari Fjällström
Advisor in Quality Assurance and Test Management
MSC
My background My great grandmother is Athabascan Indian from Holikachuk, Alaska! My great grandfather came to Alaska during
the beginning of the Gold Rush in 1884, a
Teaching User to Test - Agenda
Types of learners Why do we test?
Test, requirements and risks How to write a test case…
Combining test conditions into test scenarios How to document a test run…
How to report test progress…
Testing a new system – and reporting bugs How to report a bug…
Timeline (ST, AT, UAT, GoLive, Production) Stay with the process, follow-up!
Types of Learners
North
Air
Theoretic Learner
South
East
Fire
Activist Learner
West
Water
Reflective Learner
Learning Style Preferences
Theoretic Learner
• Models and rules
• Generalizations
• Theories
Pragmatic Learner
• Tips and tricks
• Advantages
Activist Learner
• Chance to try myself
• Opportunities to show/tell
• Demos
Reflective Learner
• Time to research
• Opportunities to review
• Repetions, walk throughs
Focus!
What do you as a test lead want users to
know about test?
–
how to write test cases
–
how to document test and test runs
–
how to report test status – tested ok, tested nok, not
tested, will not test
What is Testing?
• Monitor what is happening
• Measure quality
• Control outcome – ensure successful delivery
• Try to destroy system, identify its weaknesses
Some Goals of Testing
• See if the system works
• Learn how a system works – how to use the system
• Demonstrate that the system supports business processes
• Test a correction
• Make sure that what should not happen does not happen
• Clarify requirements
• Build confidence in the system
• Save money by finding faults early in the development
cycle
• Make sure that which isn’t changed, hasn’t been affected
by new functionality
ISTQB definition of what testing is
The process consisting of all life cycle activities,
both static and dynamic,
concerned with planning, preparation and execution
of the 1)
evaluation of software products
and related work products
to determine that they 2)
satisfy specified requirements
,
to demonstrate that they are
3)
fit for purpose (“good enough”)
Who is a “Tester”?
• the user in production (as long as the system is used) • the developer during development
• the tester or test analyst
• business application experts
• pilot customers test early versions of software • configuration managers test during installation
• auditors evaluate systems against legal requirements
Is the tester a person who is fault-finding and nit-picking, a person obsessed with details who can’t see the forest for the trees?
Acceptance Testing
Acceptance testing is most often the responsibility of the user or customer, although other stakeholders maybe involved as well. The execution of the acceptance test requires a test environment that is for most aspects, representative of the production environment (‘as-if
production’).
The goal of acceptance testing is to establish confidence in the system, part of the system or specific non-functional characteristics, e.g. usability, of the system.
Acceptance testing is most often focused on a validation type of
testing, whereby we are trying to determine whether the system is fit for purpose. Finding defects should not be the main focus in
acceptance testing. Although it assesses the system’s readiness for deployment and use, it is not necessarily the final level of testing.
Stakeholders the service order process
Who is responsible?
Who has a problem when things go
wrong?
Different stakeholders are responsible for different parts of the service order process!
Functionality Suitability Accuracy Interoperability Security Functionality Compliance Reliability Maturity Fault tolerance Recoverability Reliability Compliance Usability Understandability Learnabality Operability Attractiveness Usability Compliance Efficiency Time behaviour Resource utilizat. Efficiency- Compliance Maintainability Analysability Changeability Stability Testability Maintainability Compliance Quality attributes ISO 9126 Portability Installability Co-existence Replaceability Portability Compliance Adaptability Performance Load & Stress
Quality can be measured in different ways / dimensions.
What we
Stakeholders and Quality Attributes Information system Development Maintenance End user Audit Helpdesk Requirements on maintainability Requirements on Requirements on changeability Requirements on analysability Test Testability
Why do we test in real life?
Jag vill duscha…
Vattnet kan vara för kallt eller varmt…
We don’t test everything
But we do test when there’s a
Project and Product Risks
Project risks Product risks
IT Business
Project overrun in: Time
Money
Unsatisfactory quality Incorrect functionality
Not user-friendly
Bad test environment Insufficient resources Unfamiliarity with the testing
method
Difficult to maintain Low efficiency Difficult to install
Requirements, Risks and Test Cases
Mina krav
Risk
Testfall
Jag måste
kontakta
vänner…
Mobilen fungerar
inte pga batteriet
(stäng av mobilen
Kolla laddningen
tills det är dags att
ringa)
Jag vill gå over
gata till andra
sidan…
Jag kan bli påkörd
av en bil
Jag tittar till höger
och vänster, kolla
traffiken
Jag vill duscha… Vatten kan vara för
kallt eller varmt!
med min hand och
Jag känner vattnet
justerar
temperaturen innan
jag duschar
Behind every requirement there’s a risk! Requirement Quality Attribute Risk An invoice that is understandable and easy for the customer to interpret
Usability Kunden ringer för förklaring Support, Kundtjänst belastas Utbildning av Kundtjänst
personal A service can be
discontinued
Functionality Kunden får en tjänst som de inte betalar för eller betalar för en tjänst som de inte får. Förlorade intäkter och
RRBT: Risk and Requirement Based Testing
Requirements
Product Risks
Risk, no requirement: Add requirement (design fault found in early stage) Requirement, no risk: Leave out requirement (don’t develop more than necessary, no “frills”)Why do we prioritize tests?
Time and dependencies on deliveries, test data and
configuration!
What’s most important? What’s the biggest problem if it
doesn’t work?
→ Analyzing requirements and risks
We want to deal with the worst stuff first!
Other stuff we may have to live with in production for
some months.
Test the scariest stuff (biggest risk, most important) first! Analyse requirements Analyse risks Determine risk impact Prioritise Requirements using MoSCoW
Specify and prioritise test cases
Won’t have Could have Should have Must have Won’t test Could test Should test Must test
Combine Product Risks and
Finding errors later will cost you more
A reason to test more early on…
Definition Design Development Usage
How to structure test
Structuring tests of a system of systems
1. Identify systems - “test clusters”
2. Identify functions (processes, flows) - “test clusters”
3. What requirements/risks (“test conditions”) are associated with each function/process/flow?
4. Define test cases for each condition (there may be one or more) 5. A series of test cases and conditions can be combined in a “test
Test Analysis – summary of structure
Hierachy
cluster
test condition
test case
test line/steps
action word
parameters
A cluster is
product/function/
service
A test condition is
a requirement/risk
Test cases are
used to enter data
and check results
,Breaking a System into Clusters
Clusters are logical units
A function or process can also be a cluster.
Business System Customers and Sales
Accounts Receivable Assets Bookkeeping General Ledger Suppliers Accounts Payable
Breaking a System into Clusters
Clusters are logical units
For each condition we may have several test cases.
Setting up test cases
Conditions and test cases
•
Positive tests (can I do the flows that are
important to my job?)
•
Negative tests (can I enter invalid data into the
system? What happens when I do that?)
The basic Test Case
•
Preconditions (förutsättningar)
•
Input (steps or actions)
•
Expected results (Förväntat resultat)
•
Observations (Utfall)
A good title for the test case is important!
Each test case should have
How to write a test case
A test case:
1. execution preconditions 2. input values
3. expected results (output) 4. observations
Ett testfall:
1. Förutsättningar
2. Steg (inmatning av data) 3. Förväntat resultat
Scenario testing is most often used in User Acceptance Test
What is a scenario or test script?
Scenarios are used when testing that
the system meets the requirements
and that
the system useful in practice.
A
scenario
or test script is
a
connected group of test conditions and test cases
which follows the
business processes
The test cases in scenarios are often dependent (if one test fails, further testing may be blocked)
What do scenarios prove?
Scenarios based on business or accounting
flows:
• process bookkeeping orders
• process invoices
• process payments
Does the system meet the requirements?
Is the system useful and practical?
A scenario (from Italian, “that which is pinned to theater scenery”) is a concise description of a play as a series of actions and events.
Hur man prioriterar testfallen
6.1 Ny tillgång Must Test
6.2 Avskrivning Must Test
6.4 Utrangering (skruta något) Must Test 6.9 Aktivera (imateriel) tillgång med flera fakturor Must Test 6.10 Flytta (sälj) till annat koncernbolag som användare i E1 Must Test 6.11 Flytta till annat resultatställe inom samma bolag Must Test
6.12 Avstämning Must Test
6.13 Bokslutsrapport Must Test
6.5 Skriva ner Should Test
6.7 Minska avskrivningstid Should Test 6.8 Öka avskrivningstid Should Test
6.14 Delning Should Test
6.16 Lägga till faktura till befintlig anläggning utan avskrivning Should Test 6.17 Avskrivning utan att påverka Should Test 6.4b Utrangering med försäljning Could Test
6.6 Skriva upp Won't Test
6.15 Lägga till faktura till befintlig anläggning med avskrivning Won't Test
Soap Opera
A soap opera is an ongoing, episodic work of fiction, usually broadcast on television or radio.
Soap opera stories run concurrently, intersect, and lead into further developments. An individual episode of a
soap opera will generally switch between several different concurrent story threads that may at times
interconnect and affect one another, or may run entirely independent of each other.
Each episode may feature some of the show's current storylines but not always all of them.
Soap opera episodes typically end on some sort of
cliffhanger, a promise that the storyline is to be continued in another episode.
What is a soap scenario?
• A kind of scenario test, but exaggerated
o One test covers more than what happens in a lifetime
• Done by testers and end-users
• Write situations without worrying about the details of the system
o Think business, not technical
o What complications can happen? o What can go terribly wrong?
• Write down tests:
o Based on real life o Exaggerated
o Condensed into a limited number of
Useful tips when creating Soap scenarios
• Relate to business • Identify responsible persons
• Create drama over departmental /
organisational boundaries • What are the risks in the process, with the product? • Define coverage level (keep it intense but focused)
Soaps and risks: what is computer-style drama? Leap years (Feb 29,2008)
Large orders
Many users online at the same time
Long distances Long time periods Transport fees Discounts Many handovers Acute deliveries Backorders Cancelled orders/services Human error – are mistakes
easily corrected?
Merging incoming transactions from different sources (also diverging)
A test suite for one test level
“User not authorised” System hangs Intake test: major errors present. Does the system work at all? Repair and rerun End test: system performs as required. Re-test areas as needed Repair and rerun “Amount formatted incorrectly” Basic test: basic errors present. Run all positive/valid tests, sunshine scenarios Repair and rerun Complete test: further errors present. Add negative/invalid tests and soap scenarios
“Invalid input is allowed”
Test Sets
•
Does the system work at all?
•
Can we run basic flows?
•
Can we handle errors when they happens?
Some theory
We’ve talked about dividing a system into areas or applications Clusters can be applications, program, functions, flows…
What are test levels?
• V-model
Different tests for different test levels:
• Example ST – process incoming batch file, checking that it’s parsed (moved to fields in database) correctly.
• Example AT – Order flow – process an external order
• Process a credit order
Test Levels in the V-model Acceptance Test Component Integration Test System Integration Test System Test User needs, Requirements, Business processes System Specification Technical Design & Code
Test Levels and Scenarios Acceptance Test Component Integration Test Component Test System Integration Test System Test User needs, Requirements, Business processes System Specification Technical Design & Code E1: Bokföringsflödet Fakturering Rapporter Felsökning Inkommande filer: Bokföringsunderlag Fakturering underlag Felsökning
Create different tests for different test levels (SOA example)
Component Test System Integration Test Acceptance Test
1. generate price list based on customer’s contract and product purchase price
2. create XML files of current price lists per customer 3. send XML file from Server
1 (Supplier)
4. receive XML file in Server 2 (Message Transformation and Routing)
5. XML file is converted to EDIFACT format
6. EDIFACT file is sent from
Server 2
1. generate pricelist header and lines in database 2. check XML files for
proper formatting, data mapping and number of files
3. Transfer XML file from
Server 1 (Supplier) to Server 2 (Message Transformation and Routing) 4. Check transformation XML to EDIFACT (mapping of data) 5. Transfer EDIFACT file
1. generate and send price lists for current
period to all
Black box and white box testing – identifying errors
Component Test step Potential source of failure
1) generate pricelist based on customer’s contract and product purchase price
Price calculated incorrectly
Problem accessing correct contract
Problem accessing product purchase price
2) create XML files of current price lists per customer
Addressing information for envelope (partner agreement not found)
Selection of price lists (header and lines) for this period
3) send XML file from Server 1 (Supplier)
Configuring both servers, Server 1 (Supplier) and
Server 2 (Message Transformation and Routing)
Incorrect addressing (partner agreement)
Incorrect header 4) receive XML file in Server 2 (Message
Transformation and Routing)
5) XML file is converted to EDIFACT format
Incorrect mapping of data
Misspelling of tag names
Invalid format 6) EDIFACT file is sent from Server 2
(Message Transformation and Routing)
7) EDIFACT file is received by Server 3
Configuring both servers, Server 2 (Message Transformation and Routing) and Server 3 (Customer)
SOA development roles
Responsible Component Test step
Developer 1 (Oracle Database expert)
1) generate pricelist based on customer’s contract and product purchase price
Developer 2 (XML, Java expert) 2) create XML files of current price lists per customer
Configurator 1 3) send XML file from Server 1
(Supplier)
Configurator 2 4) receive XML file in Server 2
(Message Transformation and Routing)
Developer 3 (XML, EDIFACT) 5) XML file is converted to EDIFACT format
V- model and Quality Gates – three parties involved Acceptance Test Component Integration Test System Integration Test System Test User needs, Requirements, Business processes System Specification Technical Design & Code Quality Gate 3 Exit criteria Entry criteria User Organisation Maintenance Designers Developers Quality Gate 2
Quality Gate 1 Quality Gate 4
Quality Gate 5 Production
Exit and Entry Criteria
exit criteria: The set of generic and specific conditions, agreed upon with the stakeholders, for permitting a process to be officially
completed.
The purpose of exit criteria is to prevent a task from being
considered completed when there are still outstanding parts of the task which have not been finished. Exit criteria are used to report against and to plan when to stop testing. [After Gilb and Graham]
entry criteria: The set of generic and specific conditions for permitting a process to go forward with a defined task, e.g. test phase.
The purpose of entry criteria is to prevent a task from starting which would entail more (wasted) effort compared to the effort needed to remove the failed entry criteria. [Gilb and Graham]
Att testa ett system av system
Vi tester integration av ett antal system
Så vi tester batcher av poster –
kommer de fram?
har dem skickas redan?
Och hur ska hitta/rätta fel bland dessa poster?
fel i formattet
Risk Analysis and standard set of tests
It is important to do a risk analysis of the system as it is today and to specify a set of standard tests to run to determine the “health” of the system.
A risk analysis of the system is done in order to establish which
functions or programs constitute the greatest risk to the
operational services in the event of disaster.
Then we decide - with respect to the functions at risk — which (test) actions should be performed to detect/prevent a
Hur man rapporter testresultat
Good Vilka testfall Gick Bra?
Hur man rapporter testresultat
Green Good Vilka testfall Gick Bra?
Red Failed Vilka testfall Fel? inklusiv fel # eller beskriving Pale
yellow Selected (but not yet run) Vad finns kvar att testa?
Inte alla testfall kommer att köras! Det finns testfall kvar att köra!
Why is it important to document test cases?
Test cases can be re-used by others in the team (ST
or AT become RT)
Management wants evidence that we did test the
flows!
Personal reasons:
you may want to move on to other areas and new
assignments, documenting your work makes that
transition easier (delegation and handovers)
TFS – Test Cases, Bugs and Change Requests
Test Case
I den nya versionen av TFS heter Work Item-typen för testfall Test Case. Med Test Case är att man kan skriva testfallet i steg
och sedan rapportera av steg för steg när testfallet körs.
Bug Bug används för att rapportera fel och hantera arbetsflödet vid rättning.
Bug tillstånd: Proposed, Active, Resolved och Closed.
Change Request (CR)
Change Request - ändringsbegäran - används för att logga önskade förändringar av krav eller nya krav. Tillstånden är
TFS –Bug Severity
Allvarlighetsgrad
(Severity) Beskrivning
Systemstoppande Helt stoppande fel för test alternativt systemstoppande för produktion, t ex att
applikationen inte startar eller kraschar Allvarligt Allvarligt ur ett verksamhetsperspektiv,
måste vara rättat till produktion. Mindre Allvarligt Kan leva med i produktion
Irriterande Irriterande / kosmetiska fel
TFS – Bug and CR states TIPS: Vid byte av State, skriv alltid en kommentar i Symptom, börja med namn och datum
Scope: Who performs verification in Production Environment?
System Test Verification Test Live Proving
(Tjänsteprov) Development environment Regression Test Integration & Acceptance Test Functional Shakedown Test
Acceptance Test environment Production environment
Unit Test
IN SCOPE
OUT OF SCOPE OUT OF SCOPE
System Test Verification Test Live Proving
(Tjänsteprov) Development environment Regression Test Integration & Acceptance Test Functional Shakedown Test
Acceptance Test environment Production environment
Unit Test
IN SCOPE
OUT OF SCOPE OUT OF SCOPE
Verification in production environment a part of Acceptance Test Newly developed system merged with all that already is…
Some good advice to the test lead/teacher
Say what you want!
Demonstrate what you want!
Adjust to different learning styles!
Answer questions!
Listen and observe!
Offer suggestions for improvement!
Teaching Users to Test presented by Kari Fjällstöm
Do you
have
questions?
Types of learners Why do we test?Test, requirements and risks How to write a test case…
Combining test conditions into test scenarios How to document a test run…
How to report test progress…
Testing a new system – and reporting bugs How to report a bug…
Timeline (ST, AT, UAT, GoLive, Production) Stay with the process, follow-up
