Severity: With respect to functionality Priority: With respect to customer.
Severity: All defects are not with same severity. Priority: All defects are not with same priority. Severity: Seriousness of the defect.
Priority: Importance of the defect.
Severity: Project functionality point of view important. Priority: Customer point of view important.
Resolved Bug Severity
High Medium Less
All P0 All P1 Selected P2 All P0 Selected P1 Some P2 Some P0 Some P1 Some P2 On modified Build to ensure bug resolving
Defect Reporting and Tracking:
During comprehensive test execution, test engineers are reporting mismatches to development team as defect reports in IEEE format.
1. Defect Id: A unique number or name. 2. Defect Description: Summary of defect.
3. Build Version Id: Parent build version number. 4. Feature: Module / Functionality
5. Testcase name and Description: Failed testcase name with description 6. Reproducible: (Yes / No)
7. If yes, attach test procedure.
8. If No, attach snapshots and strong reasons 9. Severity: High / Medium / Low
10. Priority
11. Status: New / Reopen (after 3 times write new programs) 12. Reported by: Name of the test engineer
13. Reported on: Date of Submission 14. Suggested fix: optional
15. Assign to: Name of PM 16. Fixed by: PM or Team Lead
17. Resolved by: Name of the Developer 18. Resolved on: Date of solving
19. Resolution type:
20. Approved by: Signature of the PM
Defect Age: The time gap between resolved on and reported on. Defect Submission:
Fig: Large Scale Organizations. Test Manager Test Lead Test Engineer Project Manager Team Lead Developers QA Transmittal Reports
Defect Submission:
Fig: Small Scale Organizations. Defect Status Cycle:
Bug Life Cycle: Test Lead Test Engineer Project Manager Team Lead Developers Transmittal Reports New
Fixed (Open, Reject, Deferred)
Closed
Resolution Type:
There are 12 resolution types such as
1. Duplicate: Rejected due to defect like same as previous reported defect.
2. Enhancement: Rejected due to defect related to future requirement of the customer. 3. H/w Limitation: Raised due to limitations of hardware (Rejected)
4. S/w Limitation: Rejected due to limitation of s/w technology.
5. Functions as design: Rejected due to coding is correct with respect to design documents.
6. Not Applicable: Rejected due to lack of correctness in defect. 7. No plan to fix it: Postponed part timely (Not accepted and rejected)
8. Need for More Information: Developers want more information to fix. (Not accepted and rejected)
9. Not Reproducible: Developer want more information due to the problem is not reproducible. (Not accepted and rejected)
10. User misunderstanding: (Both argues you r thinking wrong) (Extra negotiation between tester and developer)
11. Fixed: Opened a bug to resolve (Accepted) 12. Fixed Indirectly: Differed to resolve (Accepted) Types of Bugs: Detect Defect Reproduce Defect Report Defect Fix Bug Resolve Bug Close Bug
Defect Report
Resolution Type
Testing
Development
UI bugs: (Low severity)
Spelling mistake: High Priority Wrong alignment: Low Priority
Input Domain bugs: (Medium severity)
Object not taking Expected values: High Priority Object taking Unexpected values: Low Priority Error Handling bugs: (Medium severity) Error message is not coming: High Priority
Error message is coming but not understandable: Low Priority Calculation bugs: (High severity)
Intermediate Results Failure: High Priority Final outputs are Wrong: Low Priority Service Levels bugs: (High severity) Deadlock: High Priority
Improper order of Services: Low Priority Load condition bugs: (High severity) Memory leakage under load: High Priority
Doesn't allows customer expected load: Low Priority Hardware bugs: (High severity)
Printer not connecting: High Priority Invalid printout: Low Priority
Boundary Related Bugs: (Medium Severity)
Id control bugs: (Medium severity) Wrong version no, Logo
Version Control bugs: (Medium severity) Difference between two consecutive versions Source bugs: (Medium severity) Mismatch in help documents
Test Closure:
After completion of all possible testcase execution and their defect reporting and tracking, test lead conduct test execution closure review along with test engineers.
In this review test lead depends on coverage analysis: • BRS based coverage
• UseCases based coverage (Modules) • Data Model based coverage (i/p and op) • UI based coverage (Rules and Regulations)
Analysis of the differed bugs:
Whither deferred bugs are postponable or not.
Testing team try to execute the high priority test cases once again to confirm correctness of master build.
Final Regression Process: Gather requirements
Effort estimation (Person/Hr) Plan Regression
Execute Regression Report Regression
Final Regression Testing:
User Acceptance Testing:
After completion of test execution closure review and final regression, our organization concentrates on UAT to collect feed back from customer / customer site like people.
There are two approaches: 1. Alpha testing 2. Beta testing SignOff:
After completion of UA and then modifications, test lead creates Test Summary Report (TSR). It is a part of s/w release note. This TSR consists of
1. Test Strategy / Methodology (what tests) 2. System Test Plan (schedule)
3. Traceability Matrix (mapping requirements and testcases) 4. Automated Test Scripts (TSL + GUI map entries)
5. Final Bug summary Report
Gather requirements Effort estimation Plan Regression Execute Regression Report Regression
Bug Id Description Found By Status(Closed /
Deferred) Severity Module / Functionality Comments
Case Study (Schedule for 5 Months):
Deliverable Responsibility Completion Time
TestCase Selection Test Engineer 20-30 days
TestCase Review Test Lead, Test Engineer 4-5 days
RVM / RTM Test Lead 1 day
Sanity & Test Automation Test Engineer 20-30 days
Test Execution as Batches Test Engineer 40-60 days
Test Reporting Test Engineer & Test Lead On going during test execution
Communication and Status Reporting
Everyone in testing team Weakly twice Final Regression Testing &
Closer Review
Test Engineer and Test Lead 4-5 days User Acceptance Testing Customer Site People
( Involvement of Testing Team)
5-10 days Test Summary Report
(Sign Off) Test Lead 1-2 days
Testing computer software – Cem Kamer
Effective methods for software testing – William E Perry Software Testing Tools – Dr. K.V.K.K. Prasad
[email protected] What u r doing?
What type of testing process going on ur company?
What type of test documentation prepared by ur organization? What type of test documentation u will prepare?
Whats ur involvement in that?
What are key components of ur company test plan? What type of format u prepare for test cases?
How ur pm selects what type of tests need for ur project? When u will go to automation?
What is regression testing? When u will do this? How u report defects to development team? How u know whither defect accepted or rejected? What u do when ur defect rejected?
How u will learn project with out documentation?
What is the difference between defect age and Build interval period? How u will do test without documents?
What do u mean by green box testing? Experience on winrunner
Auditing:
During testing and maintenance, testing team conducts audit meetings to estimate status and required improvements. In this auditing process they can use three types of measurements and metrics.
Quality Measurement Metrics:
These measurements are used by QA or PM to estimate achievement of quality in current project testing [monthly once]
Product Stability:
Sufficiency:
• Requirements Coverage
• Type – Trigger Analysis (Mapping between covered requirements and applied tests) Defect Severity Distribution Organization trend limit check:
• Organisation trend limit check Test Management Measurements:
These measurements are used by test lead during test execution of current project [weakly twice] Test Status • Executed tests • In progress • Yet to execute Delays in Delivery
• Defect Arrival Rate • Defect Resolution Rate • Defect Aging
Test Effort
• Cost of finding a defect (Ex: 4 defects / person day) N o . O f bu g s Duration 20% Testing – 80% Bugs 80% Testing – 20% Bugs
Process Capability Measurements:
These measurements are used by quality analyst and test management to improve the capability of testing process for upcoming projects testing. (It depends on old projects maintenance level feedback)
Test Efficiency
• Type-Trigger Analysis • Requirements Coverage Defect Escapes
• Type-Phase analysis.
(What type of defects my testing team missed in which phase of testing) Test Effort
• Cost of finding a defect (Ex: 4 defects / person day)
This topic looks at Static Testing techniques. These techniques are referred to as "static" because the software is not executed; rather the specifications, documentation and source code that comprise the software are examined in varying degrees of detail.
There are two basic types of static testing. One of these is people-based and the other is tool- based. People-based techniques are generally known as “reviews” but there are a variety of different ways in which reviews can be performed. The tool-based techniques examine source code and are known as "static analysis". Both of these basic types are described in separate sections below.
What are Reviews?
“Reviews” is the generic name given to people-based static techniques. More or less any activity that involves one or more people examining something could be called a review. There are a variety of different ways in which reviews are carried out across different organisations and in many cases within a single organisation. Some are very formal, some are very informal, and many lie somewhere between the two. The chances are that you have been involved in reviews of one form another.
One person can perform a review of his or her own work or of someone else’s work. However, it is generally recognised that reviews performed by only one person are not as effective as reviews conducted by a group of people all examining the same document (or whatever it is that is being reviewed).
Review techniques for individuals
Desk checking and proof reading are two techniques that can be used by individuals to review a document such as a specification or a piece of source code. They are basically the same processes: the reviewer double-checks the document or source code on their own. Data stepping is a slightly different process for reviewing source code: the reviewer follows a set of data values through the source code to ensure that the values are correct at each step of the processing.
Review techniques for groups
The static techniques that involve groups of people are generically referred to as reviews. Reviews can vary a lot from very informal to highly formal, as will be discussed in more
walkthrough is a form of review that is typically used to educate a group of people about a technical document. Typically the author "walks" the group through the ideas to explain them and so that the attendees understand the content. Inspection is the most formal of all the formal review techniques. Its main focus during the process is to find faults, and it is the most effective review technique in finding them (although the other types of review also find some faults). Inspection is discussed in more detail below.
Reviews and the test process Benefits of reviews
There are many benefits from reviews in general. They can improve software development productivity and reduce development timescales. They can also reduce testing time and cost. They can lead to lifetime cost reductions throughout the maintenance of a system over its useful life. All this is achieved (where it is achieved) by finding and fixing faults in the products of development phases before they are used in subsequent phases. In other words, reviews find faults in specifications and other documents (including source code) which can then be fixed before those specifications are used in the next phase of development.
Reviews generally reduce fault levels and lead to increased quality. This can also result in improved customer relations.
Reviews are cost-effective
There are a number of published figures to substantiate the cost-effectiveness of reviews. Freedman and Weinberg quote a ten times reduction in faults that come into testing with a 50% to 80% reduction in testing cost. Yourdon in his book on Structured Walkthroughs found that faults were reduced by a factor of ten. Gilb and Graham give a number of documented benefits for software Inspection, including 25% reduction in schedules, a 28 times reduction in maintenance cost, and finding 80% of defects in a single pass (with a mature Inspection process) and 95% in multiple passes.
What can be Inspected?
Anything written down can be Inspected. Many people have the impression that Inspection applies mainly to code (probably because Fagan's original article was on "Design and code inspection"). However, although Inspection can be performed on code, it gives more value if it is performed on more "upstream" documents in the software development process. It can be applied to contracts, budgets, and even marketing material, as well as to policies, strategies, business plans, user manuals, procedures and training material. Inspection also applies to all types of system development documentation, such as requirements, feasibility studies and designs. It is also very appropriate to apply to all types of test documentation such as test plans, test designs and test cases. In fact even with Fagan's original method, it was found to be very effective applied to testware.
What can be reviewed?
Anything that can be Inspected can also be reviewed, but reviews can apply to more things than just those ideas that are written down. Reviews can be done on visions, strategic plans and "big picture" ideas. Project progress can be reviewed to assess whether work is proceeding according to the plans. A review is also the place where major decisions may be made, for example about whether or not to develop a given feature.
Reviews and Inspections are complementary. Inspection excludes discussion and solution optimising, but these activities are often very important. Any type of review that tries to combine more than one objective tends not to work as well as those with a single focus. It
works better to use Inspection to find faults and to use reviews to discuss, come to a consensus and make decisions.
What to review / Inspect?
Looking at the ‘V’ life cycle diagram that was discussed in Session 2, reviews and Inspections apply to everything on the left-hand side of the V-model. Note that the reviews apply not only to the products of development but also to the test documentation that is produced early in the life cycle. We have found that reviewing the business needs alongside the Acceptance Tests works really well. It clarifies issues that might otherwise have been overlooked. This is yet another way to find faults as early as possible in the life cycle so that they can be removed at the least cost.
Costs of reviews
You cannot gain the benefits of reviews without investing in doing them, and this does have a cost. As a rough guide, something between 5% and 15% of project effort would typically be spent on reviews. If Inspections are being introduced into an organisation, then 15% is a recommended guideline. Once the Inspection process is mature, this may go down to around 5%. Note that 10% is half a day a week.
Remember that the cost of reviews always needs to be balanced against the cost of not doing them, and finding the faults (which are already there) much later when it will be much more expensive to fix them.
The costs of reviews are mainly in people's time, i.e. it is an effort cost, but the cost varies depending on the type of review. The leader or moderator of the review may need to spend time in planning the review (this would not be done for an informal review, but is required for Inspection). The studying of the documents to be reviewed by each participant on their own is normally the main cost (although in practice this may not be done as thoroughly as it should). If a meeting is held, the cost is the length of the meeting times the number of people present. The fixing of any faults found or the resolution of issues found may or may not be followed up by the leader. In the more formal review techniques, metrics or statistics are recorded and analysed to ensure the continued effectiveness and efficiency of the review process. Process improvement should also be a part of any review process, so that lessons learned in a review can be folded back into development and testing processes. (Inspection formally includes process improvement; most other forms of review do not.)
Types of review
We have now established that reviews are an important part of software testing. Testers should be involved in reviewing the development documents that tests are based on, and should also review their own test documentation.
In this section, we will look at different types of reviews, and the activities that are done to a greater or lesser extent in all of them. We will also look at the Inspection process in a bit more detail, as it is the most effective of all review types.
Characteristics of different review types
Informal review
As its name implies, this is very much an ad hoc process. Normally it simply consists of someone giving their document to someone else and asking them to look it over. A document may be distributed to a number of people, and the author of the document would hope to receive back some helpful comments. It is a very cheap form of review because there is no monitoring of metrics, no meeting and no follow--up. It is generally perceived to be useful,
and compared to not doing any reviews at all, it is. However, it is probably the least effective form of review (although no one can prove that since no measurements are ever done!)
Technical review or Peer review
A technical review may have varying degrees of formality. This type of review does focus on technical issues and technical documents. A peer review would exclude managers from the review. The success of this type of review typically depends on the individuals involved - they can be very effective and useful, but sometimes they are very wasteful (especially if the meetings are not well disciplined), and can be rather subjective. Often this level of review will have some documentation, even if just a list of issues raised. Sometimes metrics will be kept. This type of review can find important faults, but can also be used to resolve difficult technical problems, for example deciding on the best way to implement a design.
Decision-making review
This type of review is closely related to the previous one (in fact the syllabus does not distinguish them). In this type of review, which may be technical or managerial, the focus is on discussing the issues, coming to a consensus and making decisions, for example about whether a given feature should be included in the next release or not.
Walkthrough
A walkthrough is typically led by the author of a document, for the purpose of educating the participants about the content so that everyone understands the same thing. A walkthrough may include "dry runs" of business scenarios to show how the system would handle certain specific situations. For technical documents, it is often a peer group technique.
Inspection
An Inspection is the most formal of the formal review techniques. There are strict entry and exit criteria to the Inspection process, it is led by a trained Leader or moderator (not the author), there are defined roles for searching for faults based on defined rules and checklists. Metrics are a required part of the process.
Characteristics of reviews in general
Objectives and goals
The objectives and goals of reviews in general normally include the verification and validation of documents against specifications and standards.
Some types of review also have an objective of achieving a consensus among the attendees (but not Inspection).
Some types of review have process improvement as a goal (this is formally included in Inspection).
Activities
There are a number of activities that may take place for any review. The planning stage is part of all except informal reviews.
In Inspection (and possibly other reviews), an overview or kickoff meeting is held to put everyone "in the picture" about what is to be reviewed and how the review is to be conducted.