•
What is Selenium?Selenium is a suite of tools to automate web applications testing across many platforms
•
Have you read any good books on Selenium?There are several books covering Selenium automation tool
o
An Introduction to Testing Web Applications with twill and Selenium by C. Titus Brown, Gheorghe Gheorghiu, Jason Hugginso
Java Power Tools by John Ferguson Smart•
What test can Selenium do?Selenium could be used for the functional, regression, load testing of the web based applications. The automation tool could be implemented for post release validation with continuous integration tools like Hudson or CruiseControl.
•
What is the price of Selenium license per server?Selenium is open source software, released under the Apache 2.0 license and can be downloaded and used without charge.
•
How much does Selenium license cost per client machine?Selenium is open source software, released under the Apache 2.0 license and can be downloaded and used without charge.
•
Where to download Selenium?Selenium can be downloaded and installed for free from seleniumhq.org
•
What is the latest version of Selenium?The latest version of Selenium Core, Selenium IDE, Selenium RC and Selenium Grid can be found on Selenium download page
•
What is Selenium IDE?Selenium IDE is a Firefox add-on that records clicks, typing, and other actions to make a test, which you can play back in the browser
•
What does SIDE stand for?•
What is Selenium Remote Control (RC) tool?Selenium Remote Control (RC) runs your tests in multiple browsers and platforms. Tweak your tests in your preferred language.
•
What is Selenium Grid?Selenium Grid extends Selenium RC to distribute your tests across multiple servers, saving you time by running tests in parallel.
•
How many browsers are supported by Selenium IDE?Test Engineer can record and playback test with Selenium IDE in Firefox.
•
How many browsers are supported by Selenium Remote Control?QA Engineer can use Firefox, IE 7, Safari and Opera browsers to run actuall tests in Selenium RC.
•
How many programming languages can you use in Selenium RC?Several programming languages are supported by Selenium Remote Control. C# Java Perl PHP Python Ruby
•
What are the advantages using Selenium as testing tool?If QA Engineer would compare Selenium with HP QTP or Micro Focus SilkTest, QA Engineer would easily notice tremendous cost savings for Selenium. In contrast to expensive SilkTest or QTP license, Selenium automation tool is absolutely free. Selenium allows writing and executing test cases in various programming languages including C#, Java, PERL, Python, PHP and even HTML. Selenium allows simple and powerful DOM-level testing and in the same time could be used for testing in the traditional waterfall or modern Agile environments. Selenium would be definitely a great fit for the continuous integration.
•
What are the disadvantages of using Selenium as testing tool?Selenium weak points are tricky setup; dreary errors diagnosis; tests only web applications
•
How to developer Selenium Test Cases?Using the Selenium IDE, QA Tester can record a test to comprehend the syntax of Selenium IDE commands, or to check the basic syntax for a specific type of user interface. Keep in mind that Selenium IDE recorder is not clever as QA Testers want it to be. Quality assurance team should never consider Selenium IDE as a "record, save, and run it" tool, all the time anticipate reworking a recorded test cases to make them maintainable in the future.
•
Describe some problem that you had with Selenium tool?As with any other type of test automation tools like SilkTest, HP QTP, Watir, Canoo Webtest, Selenium allows to record, edit, and debug tests cases. However there are several problems that seriously affect maintainability of recorded test cases.
The most obvious problem is complex ID for an HTML element. If IDs is auto-generated, the recorder test cases may fail during playback. The work around is to use XPath to find required HTML element.
Selenium supports AJAX without problems, but QA Tester should be aware that Selenium does not know when AJAX action is completed, so ClickAndWait will not work. Instead QA tester could use pause, but the snowballing effect of several 'pause' commands would really slow down total testing time of test cases.
•
Is there training available for Selenium?•
Do you know any alternative test automation tools for Selenium?Selenium appears to be the mainstream open source tool for browser side testing, but there are many alternatives. Canoo Webtest is a great Selenium alternative and it is probably the fastest automation tool. Another Selenium alternative is Watir, but in order to use Watir QA Tester has to learn Ruby. One more alternative to Selenium is Sahi, but is has confusing interface and small developers community.
•
Compare HP QTP vs Selenium? Read Selenium vs QTP comparison•
Compare Borland Silktest vs Selenium? Check Selenium vs SilkTest comparison Q1. What is Selenium?Ans. Selenium is a set of tools that supports rapid development of test automation scripts for web
based applications. Selenium testing tools provides a rich set of testing functions specifically designed to fulfil needs of testing of a web based application.
Q2. What are the main components of Selenium testing tools? Ans. Selenium IDE, Selenium RC and Selenium Grid
Q3. What is Selenium IDE?
Ans. Selenium IDE is for building Selenium test cases. It operates as a Mozilla Firefox add on and
provides an easy to use interface for developing and running individual test cases or entire test suites. Selenium-IDE has a recording feature, which will keep account of user actions as they are performed and store them as a reusable script to play back.
Q4. What is the use of context menu in Selenium IDE?
Ans. It allows the user to pick from a list of assertions and verifications for the selected location.
Q5. Can tests recorded using Selenium IDE be run in other browsers?
Ans. Yes. Although Selenium IDE is a Firefox add on, however, tests created in it can also be run in
other browsers by using Selenium RC (Selenium Remote Control) and specifying the name of the test suite in command line.
Q6. What are the advantage and features of Selenium IDE?
Ans. 1. Intelligent field selection will use IDs, names, or XPath as needed
2. It is a record & playback tool and the script format can be written in various languages including C#, Java, PERL, Python, PHP, HTML
3. Auto complete for all common Selenium commands 4. Debug and set breakpoints
5. Option to automatically assert the title of every page 6. Support for Selenium user-extensions.js file
Q7. What are the disadvantage of Selenium IDE tool?
Ans. 1. Selenium IDE tool can only be used in Mozilla Firefox browser. 2. It is not playing multiple windows when we record it.
Q8. What is Selenium RC (Remote Control)?
Ans. Selenium RC allows the test automation expert to use a programming language for maximum
flexibility and extensibility in developing test logic. For example, if the application under test returns a result set and the automated test program needs to run tests on each element in the result set, the iteration / loop support of programming language’s can be used to iterate through the result set, calling Selenium commands to run tests on each item.
Selenium RC provides an API and library for each of its supported languages. This ability to use Selenium RC with a high level programming language to develop test cases also allows the automated testing to be integrated with the project’s automated build environment.
Q9. What is Selenium Grid?
that must be run in multiple environments. Selenium Grid can be used to run multiple instances of Selenium RC on various operating system and browser configurations.
Q10. How Selenium Grid works?
Ans. Selenium Grid sent the tests to the hub. Then tests are redirected to an available Selenium RC,
which launch the browser and run the test. Thus, it allows for running tests in parallel with the entire test suite.
Q 11. What you say about the flexibility of Selenium test suite?
Ans. Selenium testing suite is highly flexible. There are multiple ways to add functionality to Selenium
framework to customize test automation. As compared to other test automation tools, it is
Selenium’s strongest characteristic. Selenium Remote Control support for multiple programming and scripting languages allows the test automation engineer to build any logic they need into their automated testing and to use a preferred programming or scripting language of one’s choice. Also, the Selenium testing suite is an open source project where code can be modified and enhancements can be submitted for contribution.
Q12. What test can Selenium do?
Ans. Selenium is basically used for the functional testing of web based applications. It can be used for
testing in the continuous integration environment. It is also useful for agile testing
Q13. What is the cost of Selenium test suite?
Ans. Selenium test suite a set of open source software tool, it is free of cost.
Q14. What browsers are supported by Selenium Remote Control?
Ans. The test automation expert can use Firefox, IE 7/8, Safari and Opera browsers to run tests in
Selenium Remote Control.
Q15. What programming languages can you use in Selenium RC? Ans. C#, Java, Perl, PHP, Python, Ruby
Q16. What are the advantages and disadvantages of using Selenium as testing tool?
Ans. Advantages: Free, Simple and powerful DOM (document object model) level testing, can be used
for continuous integration; great fit with Agile projects.
Disadvantages: Tricky setup; dreary errors diagnosis; can not test client server applications.
Q17. What is difference between QTP and Selenium?
Ans. Only web applications can be testing using Selenium testing suite. However, QTP can be used for
testing client server applications. Selenium supports following web browsers: Internet Explorer, Firefox, Safari, Opera or Konqueror on Windows, Mac OS X and Linux. However, QTP is limited to Internet Explorer on Windows.
QTP uses scripting language implemented on top of VB Script. However, Selenium test suite has the flexibility to use many languages like Java, .Net, Perl, PHP, Python, and Ruby.
Q18. What is difference between Borland Silk test and Selenium?
Ans. Selenium is completely free test automation tool, while Silk Test is not. Only web applications
can be testing using Selenium testing suite. However, Silk Test can be used for testing client server applications. Selenium supports following web browsers: Internet Explorer, Firefox, Safari, Opera or Konqueror on Windows, Mac OS X and Linux. However, Silk Test is limited to Internet Explorer and Firefox.
Silk Test uses 4Test scripting language. However, Selenium test suite has the flexibility to use many languages like Java, .Net, Perl, PHP, Python, and Ruby.
Selenium tool
First of all it will better if you use selenium RC to automate multiple test cases by using your choice of language.
You have to follow below mentioned steps when you have to automate multiple test cases: 1. Choose your preferred language i.e.
RUBY JavaScript PERL etc.
2. Create a automation framework in your selected language. While creating automation framework you have to notice some things like Logical Data Independence Reporting Error log handling centralized library etc.
3. Record scripts through selenium IDE and convert it to your selected language via IDE. 4. Create functions for each test script and put centralized code in centralized library file. 5. Create a driver file and write command for execution sequence for test script.
1) Understand the basics right : - Basics of Web testing
- Selenium API - http://seleniumhq.org/documentation/core/reference.html#actions 2) How selenium Indentifies web element on the page?
- By Id or Name or Using an Xpath 3) Different Flavours of selenium - Selenium Core
- Selenium RC - Selenium Grid
4) Selenium - How to use IDE?
5) Is IDE good enough to automate your tests? 6) What are the practical issues, while using selenium? 7) Selenium Vs Other tools (Like QTP)
Jason Huggins on Selenium’s Challenges
Jason Huggins, one of the founders of Selenium, spoke at the Agile Developer user group meeting recently. Matt Raible of DZone gives a good summary of Jason’s comments.
Among them:
•
Selenium started at Thoughtworks. They were challenged to fix an Ajax bug in their expense reporting system. JWebUnit, HtmlUnit, JsUnit, Driftwood, and FIT did not meet their needs. They invented Selenese as a notation for tests first.•
Selenium Core, as a JavaScript embedded test playing robot, came next. Then Selenium RC and Grid.•
Selenium test playback is slow. Parallelization can solve some of the slowness problems.•
JavaScript sandbox, Flash, Applets, Silverlight, and HTML 5’s Canvas all present problems inSelenium. Read the article here. Here are my thoughts:
Thanks for the good write-up. Too bad about your battery. I would like to hear more.
PushToTest integrated Selenium into TestMaker a couple of years ago. Selenium works very well for testing Ajax applications. And, TestMaker runs Selenium tests as functional tests, load and performance tests, and business service monitors. TestMaker runs these tests in your QA lab, in the Cloud, or both. See http://www.pushtotest.com/products/cloudtesting
What does SIDE stand for?
What is the difference between an assert and a verify with Selenium commands?
What Selenese commands can be used to help debug a regexp?
What is one big difference between SilkTest and Selenium, excluding the price?
Which browsers can Selenium IDE be run in?
If a Selenium function requires a script argument, what would that argument look like in general terms?
If a Selenium function requires a pattern argument, what five prefixes might that argument have?
What is the regular expression sequence that loosely translates to "anything or nothing?"
What is the globbing sequence that loosely translates to "anything or nothing?
What does a character class for all alphabetic characters and digits look like in regular expressions?
What does a character class for all alphabetic characters and digits look like in globbing?
What must one set within SIDE in order to run a test from the beginning to a certain point within the test?
What does a right-pointing green triangle at the beginning of a command in SIDE indicate?
How does one get rid of the right-pointing green triangle?
How can one add vertical white space between sections of a single test?
What Selenium functionality uses wildcards?
Which wildcards does SIDE support?
What are the four types of regular expression quantifiers which we've studied?
What regular expression special character(s) means "any character?"
What distinguishes between an absolute and relative URL in SIDE?
What Selenese command can be used to display the value of a variable in the log file, which can be very valuable for debugging?
If one wanted to display the value of a variable named answer in the log file, what would the first argument to the previous command look like?
Where did the name "Selenium" come from?
Which Selenium command(s) simulates selecting a link?
Which two commands can be used to check that an alert with a particular message popped up?
What does a comment look like in Column view?
What does a comment look like in Source view?
What are Selenium tests normally named (as displayed at the top of each test when viewed from within a browser)?
What command simulates selecting the browser's Back button?
If the Test Case frame contains several test cases, how can one execute just the selected one of those test cases?
What globbing functionality is NOT supported by SIDE?
What is wrong with this character class range? [A-z]
What are four ways of specifying an uppercase or lowercase M in a Selenese pattern?
What does this regular expression match? regexp:[1-9][0-9],[0-9]{3},[0-9]{3}
What are two ways to match an asterisk within a Selenese regexp?
What is the generic name for an argument (to a Selenese command) which starts with //?
What Selenese command is used to choose an item from a list?
How many matches exist for this pattern? regexp:[13579][02468]
What is the oddity associated with testing an alert?
How can one get SIDE to always record an absolute URL for the open command's argument?
What Selenese command and argument can be used to transfer the value of a JavaScript variable into a SIDE variable?
How would one access the value of a SIDE variable named name from within a JavaScript snippet used as the argument to a Selenese command?
What is the name of the type of JavaScript entity represented by the last answer?
What string(s) does this regular expression match? regexp:August|April 5, 1908
What Selenium regular expression pattern can be used instead of the glob below to produce the same results?
verifyTextPresent | glob:9512?
What Selenium globbing pattern can be used instead of the regexp below to produce the same results?
Discover the automating power of Selenium
Functional testing and black box is a methodology used to test the behaviour that has an application from the viewpoint of their functions, validated for this purpose various aspects ranging from the aesthetics of the front end, the navigation within your pages between pages within the forms, the compliance of technical specifications associated with fields, buttons, bars, among other pages, entry permits and access to consultations and modifications, management of parameters, the management of the modules that constitute , and other conditions that make up the various "features" expected to provide the system to operate the end user as a normal and correct.
To meet this objective, the tester must choose a set of inputs under certain pre-defined within a certain context, and to check whether the outputs are correct or incorrect depending on the outcome defined in advance between the parties and / or techniques for the customer / supplier.
This form of testing an application is made "outside", that is why "black box testing" because the test covers the roads that follow the internal procedures of the program.
In connection with this test, although there are many tools now that this one out for various reasons will be published in future articles is: Selenium.
Selenium works directly on the web browser, its installation is simple and handling is so intuitive that allows quickly define and configure a test case, recording the journey in a page and then save the sequence of steps as a test script and and then play it when you want.
Selenium is an open-source tool that not only allows the testing of the system but also facilitates the acceptance testing of web applications.
Integrates with Firefox, and includes the ability to write the tests directly in Java, C #, Python and Ruby.
This solution has three basic tools to record a sequence of steps within a website, simulate the test with different browsers and automated test generation.
Selenium IDE is a plug-in for Firefox which allows you to record and execute scripts directly from your browser.
Selenium RC is a library and server written in Java that allows you to run scripts from local or remote through commands.
Grids Selenium: Selenium server allows multiple coordinate in order to run scripts on multiple platforms and devices at the same time.
Software testing Metrices - Test Case Review Effectiveness
Metrics are the means by which the software quality can be measured; they give you confidence in the product. You may consider these product management indicators, which can be either quantitative or qualitative. They are typically the providers of the visibility you need.
Metrics for Test Case Review Effectiveness: 1. Major Defects Per Test Case Review 2. Minor Defects Per Test Case Review 3. Total Defects Per Test Case Review
4. Ratio of Major to Minor Defects Per Test Case Review 5. Total Defects Per Test Case Review Hour
6. Major Defects Per Test Case Review Hour
7. Ratio of Major to Minor Defects Per Test Case Review Hour 8. Number of Open Defects Per Test Review
9. Number of Closed Defects Per Test Case Review 10. Ratio of Closed to Open Defects Per Test Case Review 11. Number of Major Open Defects Per Test Case Review 12. Number of Major Closed Defects Per Test Case Review 13. Ratio of Major Closed to Open Defects Per Test Case Review 14. Number of Minor Open Defects Per Test Case Review 15. Number of Minor Closed Defects Per Test Case Review 16. Ratio of Minor Closed to Open Defects Per Test Case Review 17. Percent of Total Defects Captured Per Test Case Review 18. Percent of Major Defects Captured Per Test Case Review 19. Percent of Minor Defects Captured Per Test Case Review
20. Ratio of Percent Major to Minor Defects Captured Per Test Case Review 21. Percent of Total Defects Captured Per Test Case Review Hour
22. Percent of Major Defects Captured Per Test Case Review Hour 23. Percent of Minor Defects Captured Per Test Case Review Hour
24. Ratio of Percent Major to Minor Defects Captured Per Test Case Review Hour 25. Percent of Total Defect Residual Per Test Case Review
26. Percent of Major Defect Residual Per Test Case Review 27. Percent of Minor Defect Residual Per Test Case Review
28. Ratio of Percent Major to Minor Defect Residual Per Test Case Review 29. Percent of Total Defect Residual Per Test Case Review Hour
30. Percent of Major Defect Residual Per Test Case Review Hour 31. Percent of Minor Defect Residual Per Test Case Review Hour
32. Ratio of Percent Major to Minor Defect Residual Per Test Case Review Hour 33. Number of Planned Test Case Reviews
34. Number of Held Test Case Reviews 35. Ratio of Planned to Held Test Case Reviews 36. Number of Reviewed Test Cases
37. Number of Unreviewed Test Cases
38. Ratio of Reviewed to Unreviewed Test Cases 39. Number of Compliant Test Case Reviews 40. Number of Non-Compliant Test Case Reviews
41. Ratio of Compliant to Non-Compliant Test Case Reviews 42. Compliance of Test Case Reviews
43. Non-Compliance of Test Case Reviews
44. Ratio of Compliance to Non-Compliance of Test Case Reviews Risk Analysis in Software Testing
Subscribe the QA and Software Testing Newsletter Post Your Queries | QA and Testing - Table of Contents
SPONSORED LINKS
A risk is a potential for loss or damage to an Organization from materialized threats. Risk Analysis attempts to identify all the risks and then quantify the severity of the risks.A threat as we have seen is a possible damaging event. If it occurs, it exploits vulnerability in the security of a computer based system.
Risk Identification:
1. Software Risks: Knowledge of the most common risks associated with Software development, and the platform you are working on.
3. Testing Risks: Knowledge of the most common risks associated with Software Testing for the platform you are working on, tools being used, and test methods being applied.
4. Premature Release Risk: Ability to determine the risk associated with releasing unsatisfactory or untested Software Prodicts.
5. Risk Methods: Strategies and approaches for identifying risks or problems associated with implementing and operating information technology, products and process; assessing their likelihood, and initiating strategies to test those risks.
Traceability means that you would like to be able to trace back and forth how and where any work product fulfills the directions of the preceding (source-) product. The matrix deals with the where, while the how you have to do yourself, once you know the where.
Take e.g. the Requirement of User Friendliness (UF). Since UF is a complex concept, it is not solved by just one design-solution and it is not solved by one line of code. Many partial design-solutions may contribute to this Requirement and many groups of lines of code may contribute to it.
A Requirements-Design Traceability Matrix puts on one side (e.g. left) the sub-requirements that together are supposed to solve the UF requirement, along with other (sub-)requirements. On the other side (e.g. top) you specify all design solutions. Now you can connect on the crosspoints of the matrix, which design solutions solve (more, or less) any requirement. If a design solution does not solve any requirement, it should be deleted, as it is of no value.
Having this matrix, you can check whether any requirement has at least one design solution and by checking the solution(s) you may see whether the requirement is sufficiently solved by this (or the set of) connected design(s).
If you have to change any requirement, you can see which designs are affected. And if you change any design, you can check which requirements may be affected and see what the impact is.
In a Design-Code Traceability Matrix you can do the same to keep trace of how and which code solves a particular design and how changes in design or code affect each other.
Demonstrates that the implemented system meets the user requirements. Serves as a single source for tracking purposes.
Identifies gaps in the design and testing.
Prevents delays in the project timeline, which can be brought about by having to backtrack to fill the gaps. Risk analysis is appropriate to most software development projects.
Use risk analysis to determine where testing should be focused: 1. Which functionality is most important to project
2. Which functionality is most visible to user 3. Which functionality has largest impact on users.
4. Which aspects of the application is most important to the customer 5. Which aspects of the application is tested early in the development cycle. These are some of the risk analysis factors to test an application.
The two most important characteristics of Risks are a. Its Adverse effect to the success of the project b. Its uncertainty.
Typical parameters used in analysis of risks are 1. Probability of occurrence
2. Impact of the risk - this is sometimes a product of Severity of the impact and period in which the risk might occur.
priority of the risks to be mitigated or acted upon could be decide.
Risks are always as against the business requirements for which the software has been created. So, it is always very important to understand the risks that can affect the client's business and its impact. Testers need inputs from client as well as developers with regard to this understanding. What you may consider as risk from testing point of view may not be as seen by the client. So, it is necessary to associate risk level with each requirement and also the priority. This will be very beneficial while suggesting go/no go for production.
Software Testing Requirements
Subscribe the QA and Software Testing Newsletter Post Your Queries | QA and Testing - Table of Contents
SPONSORED LINKS
Continuing the Basics of Software Testing series for Freshers,,
Software testing is not an activity to take up when the product is ready. An effective testing begins with a proper plan from the user requirements stage itself. Software testability is the ease with which a computer program is tested. Metrics can be used to measure the testability of a product. The requirements for effective testing are given in the following sub-sections.
Operability:
The better the software works, the more efficiently it can be tested.
•The system has few bugs (bugs add analysis and reporting overhead to the test process) •No bugs block the execution of tests
The product evolves in functional stages (allows simultaneous development & testing)
Observability:
What is seen is what is tested
•Distinct output is generated for each input
•System states and variables are visible or queriable during execution
•Past system states and variables are visible or queriable (eg., transaction logs) •All factors affecting the output are visible
•Incorrect output is easily identified •Incorrect input is easily identified
•Internal errors are automatically detected through self-testing mechanism •Internally errors are automatically reported
•Source code is accessible
Controllability:
•All possible outputs can be generated through some combination of input •All code is executable through some combination of input
•Software and hardware states can be controlled directly by testing •Input and output formats are consistent and structured
•Tests can be conveniently specified, automated, and reproduced.
Decomposability:
By controlling the scope of testing, problems can be isolated quickly, and smarter testing can be performed.
•The software system is built from independent modules •Software modules can be tested independently
Simplicity:
The less there is to test, the more quickly it can be tested •Functional simplicity
•Structural simplicity •Code simplicity
Stability:
The fewer the changes, the fewer the disruptions to testing •Changes to the software are infrequent
•Changes to the software are controlled
•Changes to the software do not invalidate existing tests •The software recovers well from failures
Understandability:
The more information we have, the smarter we will test •The design is well understood
•Dependencies between internal external and shared components are well understood. •Changes to the design are communicated.
•Technical documentation is instantly accessible •Technical documentation is well organized •Technical documentation is specific and detailed •Technical documentation is accurate
Software Testing Effort Estimating
Subscribe the QA and Software Testing Newsletter Post Your Queries | QA and Testing - Table of Contents
SPONSORED LINKS
Test Estimation Challenges
1. Successful test estimation is a challenge for most organizations because:
b. Test Effort Estimates also includes the Debugging Effort
c. Difficult to attempt testing estimates without first having detailed information about a project d. Software Testing Myth that Testing can be performed at the End
e. Difficult to attempt testing estimates without an understanding of what should be included in a 'testing' estimation for a project (functional testing? unit testing? reviews? inspections? load testing? security testing?)
Traditional Practices
1. Current Test Planning Methods include a. Using Percentage of Development Effort
i. Depends on the accuracy of the Development Effort ii. Does not account revisit of Development Effort b. Using Tester Developer Ratio
i. May not be same for all types of Projects ii. Does not consider the size of the Project c. Using KLOC
i. Does not consider Complexity, Criticality & Priority of the Project.
What’s the Best Approach to Test Estimation?
There is no simple answer for this. The 'best approach' is highly dependent on the particular organization and project and the experience of the personnel involved.
For example, given two software projects of similar complexity and size, the appropriate test effort for one project might be very large if it was for life-critical medical equipment software, but might be much smaller for the other project if it was for a low-cost computer game. A test estimation approach that only
considered size and complexity might be appropriate for one project but not for the other. Approaches to Test Estimation
1. Implicit Risk Context Approach: 2. Metrics-Based Approach: 3. Test Work Breakdown Approach: 4. Iterative Approach:
5. Percentage-of-Development Approach: Test Estimation Process – A Practical Approach 1. Combination of all the Approaches
2. Considers Risk & Complexity Factors
3. Based on Previous History i.e. Organization or Project Metrics 4. Based on Work Breakdown Structure
5. Based on Iterative Model
Top Five Pitfalls of Manual Software Testing Subscribe the QA and Software Testing Newsletter Post Your Queries | QA and Testing - Table of Contents
SPONSORED LINKS
Manual software testing is a necessity, and an unavoidable part of the software product development process. How much testing you do manually, as compared to using test automation, can
make the difference between a project’s success and failure. We will discuss test automation in more detail in a later chapter, but the top five pitfalls of manual software testing illuminate areas where improvements can be made. The pitfalls are listed and described below. 1. Manual testing is slow and costly. Because it is very labor-intensive, it takes a long time to complete tests. To try to accelerate testing, you may increase the headcount of the test organization. This
increases the labor as well as the communication costs.
2. Manual tests don’t scale well. As the complexity of the software increases, the complexity of the testing problem grows exponentially. If tests are detailed and must be performed manually, performing them can take quite a bit of time and effort. This leads to an increase in the total time devoted to testing as well as the total cost of testing. Even with these increases in the time and cost, the test coverage goes down as the complexity goes up because of the exponential growth rate. 3. Manual testing is not consistent or repeatable. Variations in how the tests are performed are inevitable, for various reasons. One tester may approach and perform a certain test differently from another, resulting in different results on the same test, because the tests are not being performed identically. As another example, if there are differences in the location a mouse is pointed when its button is clicked, or how fast operations are performed, these could potentially produce different results. 4. Lack of training is a common problem, although not unique to manual software testing. The staff should be well-trained in the different phases of software testing:
– Test design
– Test execution
– Test result evaluation
5. Testing is difficult to manage. There are more unknowns and greater uncertainty in testing than in code development. Modern software development practices are well-structured, but if you don’t have sufficient structure in testing, it will be difficult to manage. Consider a case in which the development phase of a project schedule slips. Since manual software testing takes more time, more resources, and is costly, that schedule slip can be difficult to manage. A delay in getting the software to the test team on schedule can result in significant wasted resources. Manual testing, as well as badly designed automated testing, are also not agile. Therefore, changes in test focus or product requirements make these efforts even more difficult to manage.
What is Traceability Matrix from Software Testing perspective? Subscribe the QA and Software Testing Newsletter
Post Your Queries | QA and Testing - Table of Contents
SPONSORED LINKS
The concept of Traceability Matrix is very important from the Testing perspective. It is document which maps requirements with test cases. By preparing Traceability matrix, we can ensure that we have covered all the required functionalities of the application in our test cases.
What is Traceability Matrix from Software Testing perspective?
The concept of Traceability Matrix is very important from the Testing perspective. It is document which maps requirements with test cases. By preparing Traceability matrix, we can ensure that we have covered all the required functionalities of the application in our test cases. Some of the features of the traceability matrix:
•
It is a method for tracing each requirement from its point of origin, through each development phase and work product, to the delivered product•
Can indicate through identifiers where the requirement is originated, specified, created, tested, and delivered•
Will indicate for each work product the requirement(s) this work product satisfies•
Facilitates communications, helping customer relationship management and commitment negotiationTraceability matrix is the answer of the following basic questions of any Software Project:
•
How is it possible to ensure, for each phase of the lifecycle, that I have correctly accounted for all the customer’s needs?•
How can I ensure that the final software product will meet the customer’s needs? For example I have a functionality which checks if I put invalid password in the password field the application throws an error message “Invalid password”. Now we can only make sure this requirement is captured in the test case by traceability matrix.Some more challenges we can overcome by Traceability matrix:
•
Demonstrate to the customer that the requested contents have been developed•
Ensure that all requirements are correct and included in the test plan and the test cases•
Ensure that developers are not creating features that no one has requested•
The system that is built may not have the necessary functionality to meet the customers and users needs and expectations. How to identify the missing parts?•
If there are modifications in the design specifications, there is no means of tracking the changes•
If there is no mapping of test cases to the requirements, it may result in missing a major defect inthe system
•
The completed system may have “Extra” functionality that may have not been specified in the design specification, resulting in wastage of manpower, time and effort.•
If the code component that constitutes the customer’s high priority requirements is not known, then the areas that need to be worked first may not be known thereby decreasing the chances of shipping a useful product on schedule•
A seemingly simple request might involve changes to several parts of the system and if proper Traceability process is not followed, the evaluation of the work that may be needed to satisfy the request may not be correctly evaluatedStep by step process of creating a Traceability Matrix from requirements:
step1: Identify all the testable requirements in granular level from various requirement specification documents. These documents vary from project to project. Typical requirements you need to capture are as follows:
Used cases (all the flows are captured) Error Messages Business rules Functional rules SRS FRS So on…
example requirements: login functionality, generate report, update something etc.
step2: In every project you must be creating test-cases to test the functionality as defined by the requirements. In this case you want to extend the traceability to those test-cases. In the example table below the test-cases are identified with a TC_ prefix.
Put all those requirements in the top row of a spreadsheet. And use the right hand column of the
spreadsheet to jot down all the test cases you have written for that particular requirement. In most of the cases you will have multiple test cases you have written to test one requirement. See the sample spreadsheet below:
step3: Put cross against each of the test case to each requirement if that particular test case is checking that particular requirement partially or completely. In the above table you can see REQ1 UC1.1 is checked by three test cases. (TC1.1.1, TC1.1.3, TC1.1.5).
Another example of traceability matrix where requirement documents (use case) are mapped back to the test cases.
Change management through traceability matrix:
It will be lot easier for you to track changes if you have a good traceability matrix in place. For example REQ1 UC1.1 we know upfront from the traceability matrix that what test cases I need to modify to incorporate those changes. In the above case we need to modify TC1.1.1, TC1.1.3 and TC1.1.5 only. Software Testing Strategy and Methodology
Subscribe the QA and Software Testing Newsletter Post Your Queries | QA and Testing - Table of Contents
SPONSORED LINKS
Testing strategy (methodology)
A test strategy describes the approach, objectives, and direction of the effort test. The purpose of a testing strategy or method is to minimize risk and ultimately provide the best software for the client. The testing strategy of choice for a particular application can vary depending on the software, amount of use, and its objectives. For example, the testing strategy for a transactional system like Oracle will be very different from the strategy developed to test an analytical tool as the Data Warehouse. In addition, the strategy chosen for testing a campus-wide purchasing system to a limited number of users of the tool housing requires a very different test strategies. Because some of these examples have higher exposure, they also have a higher risk.
STAGES OF THE TEST OF LIFE CYCLE
A Software has used several variations of one or more test methods. Lets say, if software is “IMSS” Typical stages include preparation, conference room pilot (CRP), unity, integration, system testing and user acceptance.
These steps are also called stages or levels. The project manager should review the steps below and consider the same terminology and sequence. If it makes sense, certain phases and tasks May be deleted. In other cases, tasks and phases of May should be added. May perform some tasks in parallel, and some steps can be combined. In most cases, each phase must be completed before another can begin.
Duration of the tasks vary depending on the timing and risk of the project manager is ready to absorb. Test Preparation Phase (before testing begins)
Team - Develop test strategy Task - Develop high-level test plan Team - Identify the test cases Task - Develop scenarios, test scripts Team - Identify and share test data
Team - Identify the processes, procedures, standards, documentation requirements Team - Identify and create test environment
Task - Identify test team (s) Team - Train testers
Unit test phase - The purpose of this testing is to verify and validate the function modules
correctly. This is completed by developers and must be completed before future phases can begin. The Testing Manager are not normally involved in this phase.
Phase CRP (Conference Room Pilot - optional). The purpose of this phase is to verify proof of concept. The CRP is generally necessary for new, large, not projects.
Assumption - Test instance is ready
Assumption - Metadata is inserted test example
Assumption - The unit tests and simulations has been completed Assumption - test scenarios have been identified (by script or ad hoc) Task - Identify CRP participants
Team - Determine and establish logistics CRP Task - Define expectations.
Team - Start of CRP
Task - Collect and document feedback Task - End CRP
Team - Obtain approval phase / sign-off
Team - Collect / share / integrate lessons learned, incorporate the necessary changes Task - Tune / revise and approve the new test plan
Integration of the testing phase - The purpose of this testing is to verify and validate all the modules are interface and work together.
Assumption - Requirements are frozen and the design is determined Assumption - Application is ready for integration tests
Assumption - Metadata was populated by such test tables Assumption - Unit testing is complete
Task - Test system and document using the test scripts Team - Test interfaces
Task - Identify and report bugs
Task - Retest fixed bugs / regression test Team - Test Security
Team - Test browsers / platforms / operating systems Team - Obtain approval phase / sign-off
Team - Collect / share / integrate lessons learned Task - Tune / revise and approve the new test plan
The test phase of the system - The purpose of this testing is to verify and validate the system works as if it were production.
Assumption - Metadata was populated pending test
Assumption - Application is ready and has successfully completed the integration tests Task - Test system and document using the test scripts
Task - Identify and report bugs
Task - Retest fixed bugs / regression test Team - test business processes and reports Team - Stress test
Task - test performance (eg, refreshes the screen) Team - test connection security, responsibilities, piracy Team - Obtain approval phase / sign-off
Team - Collect / share / integrate lessons learned Task - Tune / revise and approve the new test plan
Acceptance phase users - The objective of this testing is to verify and validate the system and for end users as if it were the production
Assumption - Show-stoppers and the highest level of bugs were fixed and work around have been identified and approved
Assumption - All other phases have been signed off
Assumption - Application is ready for acceptance testing by the user Assumption - Metadata was populated by such test tables
Team - Train users testers
Task - Populate and approve test scripts
Task - Test system and document using the test scripts Team - Obtain approval phase / sign-off
Team - Collect / share / integrate lessons learned
Guidelines for automated testing of web applications Subscribe the QA and Software Testing Newsletter
Post Your Queries | QA and Testing - Table of Contents
SPONSORED LINKS
One of the key reasons for doing automated testing is to ensure that time is not spent on doing repetitive tasks which can be completed by tools without human intervention. Automation could be one of the most effective tools in your toolbox but it is not a silver bullet that will solve all the problems and improve quality. Automation tools are obedient servants, and as a tester we need to become their master and use them properly to realize their full potential. It is very important to understand that automation tools are only as good as we use them. Converting test cases from manual to automated is not the best use of automation tools. They can be used in much more effective ways.
Creating robust and useful test automation framework is a very difficult task. In the web world, this task becomes even more difficult because things might change overnight. If we follow so called best practices of automation taken from stable, desktop applications, they will not be suitable in web environment and probably will have negative impact on the project's quality.
Many problems in the web world are identical to one another. For example irrespective of any web application we always need to validate things such as presence of title on all the pages.Depending on your context may be the presence of meta data on every page, presence of tracking code, presence of ad code,
size and number of advertising units and so on.
Solution presented in this article can be used to validate all, or any of the rules mentioned above , across all the pages in any domain / website. We were given a mandate to ensure that specific tracking code is present on all the pages of a big website. In a true agile fashion, once this problem was solved it was extended and re-factored to incorporate many rules on all the pages.
This solution was developed using Selenium Remote Control with Python as scripting language. One of the main reason for using tools such as Selenium RC is their ability to allow us to code in any language and this allow us to utilize full power of standard language. For this solution, a python library called Beautiful Soup was used to parse HTML pages. This solution was ported to another tool called Twill to make it faster. Since the initial code was also developed in Python, converting it to Twill was a piece of cake.
Essentially this solution / script is a small web crawler, which will visit all the pages of any website and validate certain rules. As mentioned earlier, problem statement for this is very simple i.e. “ Validate certain rules on every webpage for any given website ”. In order to achieve this, following steps were followed
1.Get Page 2. Get All the links
3. Get first link and if link is not external and crawler has not visited it, open link. 4. Get Page Source
5. Validate all the rules you want to validate on this page 6. Repeat 1 to 5 for all the pages.
It is worth mentioning here that rules that can be validated using this framework are the rules, which can be validated by looking at the source code for the page. Some of the rules that can be validated using this script are –
1. Make sure that title is present for all the pages and is not generic
2. Check the presence of meta tags like keywords and description on all the pages. 3. Ensure that instrumentation code is present on all the pages
4. Ensure that every image has an alternative text associated with it
5. Ensure that ad code is coming from the right server and has all the relevant information we need. 6. Ensure that size of the banners and skyscrapers used for advertisement is proper.
7. Ensure that every page contain at least two advertisements and no page should have more than four advertisements, except home page.
8. Ensure that master CSS is applied on all the pages for a given domain.
9. Make sure that all the styles are coming from the CSS files and styles are not present for any element on a web page.
Above mentioned list might give you some idea of what can be achieved using this approach. This list can be extended very easily. It is limited only by your imagination :)
In the next article, we will look at the code snippets and explain how easily these rules can be customized and validated across all the pages on any given domain.
Impact Analysis Checklist for Requirements Changes
Implications of the Proposed Change
❏ Identify any existing requirements in the baseline that conflict with the proposed change. ❏ Identify any other pending requirement changes that conflict with the proposed change. ❏ What are the consequences of not making the change?
❏ What are possible adverse side effects or other risks of making the proposed change?
❏ Will the proposed change adversely affect performance requirements or other quality attributes? ❏ Will the change affect any system component that affects critical properties such as safety and
security, or involve a product change that triggers recertification of any kind?
❏ Is the proposed change feasible within known technical constraints and current staff skills?
❏ Will the proposed change place unacceptable demands on any computer resources required for the development, test, or operating environments?
❏ How will the proposed change affect the sequence, dependencies, effort, or duration of any tasks currently in the project plan?
❏ Will prototyping or other user input be required to verify the proposed change?
❏ How much effort that has already been invested in the project will be lost if this change is accepted? ❏ Will the proposed change cause an increase in product unit cost, such as by increasing third-party
product licensing fees?
❏ Will the change affect any marketing, manufacturing, training, or customer support plans?
System Elements Affected by the Proposed Change
❏ Identify any user interface changes, additions, or deletions required.
❏ Identify any changes, additions, or deletions required in reports, databases, or data files. ❏ Identify the design components that must be created, modified, or deleted.
❏ Identify hardware components that must be added, altered, or deleted. ❏ Identify the source code files that must be created, modified, or deleted. ❏ Identify any changes required in build files.
❏ Identify existing unit, integration, system, and acceptance test cases that must be modified or deleted.
❏ Estimate the number of new unit, integration, system, and acceptance test cases that will be required. ❏ Identify any help screens, user manuals, training materials, or other documentation that must be
created or modified.
❏ Identify any other systems, applications, libraries, or hardware components affected by the change. ❏ Identify any third party software that must be purchased.
❏ Identify any impact the proposed change will have on the project’s software project management plan, software quality assurance plan, software configuration management plan, or other plans. ❏ Quantify any effects the proposed change will have on budgets of scarce resources, such as memory,
processing power, network bandwidth, real-time schedule.
❏ Identify any impact the proposed change will have on fielded systems if the affected component is not perfectly backward compatible.
Effort Estimation for a Requirements Change
Effort
(Labor Hours) Task
Update the SRS or requirements database with the new requirement Develop and evaluate prototype
Create new design components Modify existing design components Develop new user interface components Modify existing user interface components Develop new user publications and help screens Modify existing user publications and help screens Develop new source code
Modify existing source code
Purchase and integrate third party software
Identify, purchase, and integrate hardware components; qualify vendor Modify build files
Develop new unit and integration tests Modify existing unit and integration tests
Perform unit and integration testing after implementation Write new system and acceptance test cases
Modify existing system and acceptance test cases Modify automated test drivers
Perform regression testing at unit, integration, and system levels Develop new reports
Modify existing reports
Develop new database elements Modify existing database elements Develop new data files
Modify existing data files
Modify various project plans Update other documentation
Update requirements traceability matrix Review modified work products
Perform rework following reviews and testing
Recertify product as being safe, secure, and compliant with standards. Other additional tasks
TOTAL ESTIMATED EFFORT
Procedure:
1. Identify the subset of the above tasks that will have to be done. 2. Allocate resources to tasks.
3. Estimate effort required for pertinent tasks listed above, based on assigned resources. 4. Total the effort estimates.
5. Sequence tasks and identify predecessors.
Impact Analysis Report Template Change Request ID: ______________
Title: ______________________________________________________ Description: ______________________________________________________ ______________________________________________________ Analyst: __________________________ Date Prepared: __________________________ Prioritization Estimates: Relative Benefit: (1-9) Relative Penalty: (1-9) Relative Cost: (1-9) Relative Risk: (1-9)
Calculated Priority: (relative to other pending requirements) Estimated total effort: ___________ labor hours
Estimated lost effort: ___________ labor hours (from discarded work) Estimated schedule impact: ___________ days
Additional cost impact: ___________ dollars
Quality impact: _______________________________________________ _______________________________________________ Other requirements affected: ____________________________________________________
____________________________________________________ Other tasks affected: ____________________________________________________ ____________________________________________________ Integration issues: ____________________________________________________ Life cycle cost issues: ____________________________________________________ Other components to examine ____________________________________________________ for possible changes: ____________________________________________________
Peer Test Reviews
Subscribe the QA and Software Testing Newsletter Post Your Queries | QA and Testing - Table of Contents
SPONSORED LINKS
It's always been a source of bafflement (technical term) that within the testing domain Walkthroughs and Peer Reviews are not more widely practiced. I recall being 'invited' to a code review where a developer went through their code line by line in dolby prologic monotone. It was almost as painful as the many walkthroughs of Test Plans I've subjected folks to. What I took away from the meeting was how incredibly useful and interesting it 'could' have been. Here was the opportunity to have code explained line by line, to be provided a live narration of the thinking, logic and reasoning behind what had been created. What's more our own views and opinions could be
incorporated into what he been made.
If this was some contempory artist or author giving a narration of one of thier works we'd be fascinated and consider ourselves fortunate that we might influence the work. But it's just some bloke we work with
and it's code, so we fall asleep.
The problem is it can be hard to get excited about code, even harder to get excited about someone talking about it! The reality is most Walkthroughs and Code Reviews are brain numbingly boring. You've probably heard of and been subjected to Walkthroughs and Code Reviews of various types, the idea that an author of something (code, plans, schedules, etc.) will sit down with an interested group and walk them through, line by line, clause by clause, page by page, explaining what was written and why. Occasionally asking for input, "all ok?", "Seem to make sense?" so that after say 30 minutes or maybe an hour everyone has found they're not interested anymore and are half asleep. Actually, probably asleep. It
makes me feel tired just describing it!
Peer Code Reviews on the other hand are meant to be snappier, more active and energetic. Think more in terms of having produced a usable piece(s) of code, say a set of core functions or an interface onto those APIs your buddy wrote. Then with this small chunk in hand get it reviewed. The review is say 10 minutes
long, you're doing a Walkthrough but it's a lively narrative.
Suggested Best Practices
- Small, manageable review items: not weeks of coding
- 5 or 10 minutes review time: not 1 or 2 hour brain numbing marathons - Grab reviewers: don't book out meetings, keep it proactive (but don't break folks concentration if they're
'in the zone'!)
- Actively seek review from different colleagues: cross mentoring and knowledge sharing - Keep responsive to new perspectives: review isn't just bug finding, it's learning tricks and tips too.
Peer Test Reviews
The interesting twist for us in the test domain is that we can apply this approach to our Test Artefacts. When we're in an agile environment it serves us well to be working in the same spirit as our developer colleagues.
Why not get Peer Test Review of those 20 or so Test Cases you just put together for that module? Incrementally delivering Ready for Execution Test Cases is a great way to help the likes of Project Managers feel relaxed that we're making progress on our planning and prep. Doing the same with Test Plans, Designs, Breakdowns or whatever other artefacts you produce is also a win. This lightweight approach achieves our objectives but stops us getting bogged down in heavyweight process.
Follow the above Best Practices and keep the event lively, if you really must book out a meeting to coordinate review with several people at the same time, that's OK. Just go a little overboard with your presentation. Print copies in colour if you can or let folks access it on their lap-tops to save trees. Use the projector to make viewing easier, create slides that are already noted up or can be 'written on', what ever keeps the energy levels up.
Regression Testing: "What" to test and "When"
Regression testing is often seen as an area in which companies hesitate to allocate resources. We often hear statements such as: "The developer said the defect is fixed. Do we need to test it again?" And the answer should be: "Well, the developer probably said the product had no defects to begin with." The truth of the matter is, in today's world of extremely complex devices and software applications, the quantity and quality of regression testing performed on a product are directly proportional to the commitment vendors have to their customer base. This does not mean that the more regression testing, the better. It simply means that we must make sure that regression testing is done in the right amount and with the right approach.
The two main challenges presented by regression testing are: 1. What do we test?
2. When do we test it?
The purpose of this article is to outline a few techniques that will help us answer these questions. The first issue we should consider is the fact that it is not necessary to execute our regression at the end of our testing cycle. Much of the regression effort can be accomplished simultaneously to all other testing activities. The supporting assumption for this approach is:
"We do not wait until all testing is done to fix our defects."
Therefore, much of the regression effort can be accomplished long before the end of the project, if the project is of reasonable length. If our testing effort will only last one week, the following techniques may have to be modified. However, it is not usual for a product to be tested in such a short period of time. Furthermore, as you study the techniques outlined below, you will see that as the project's length increases, the benefits offered by these techniques also increase.
To answer the questions of what should we test and when, we will begin with a simple suite of ten tests. In the real world, this suite would obviously be much larger, and not necessarily static, meaning that the number of tests can increase or decrease as the need arises. After our first test run with the first beta (which we will call "Code Drop 1") of our hypothetical software product, our matrix looks like this. In the matrix above, we have cross-referenced the defects we found, with the tests that caused them. As you can see, defect number 1 was caused by test 2, but it also occurred on test 3. The remaining failures caused unique defects.
As we prepare to execute our second test run (Code Drop 2), we must decide what tests will be executed. The rules we will use only apply to our regression effort. There are rules we can apply to the subset of tests that have passed, in order to find out which ones we should re-execute. However, that will be the topic of another article.
The fundamental question we must now ask is: "Have any of the defects found been fixed?" Let us suppose that defects 1, 2, and 3 have, in fact, been reported as fixed by our developers. Let us also suppose that three more tests have been added to our test suite. After "Code Drop 2", our matrix looks as follows:
A few key points to notice are:
Of the tests that previously failed, only the tests that were associated with defects that were supposedly fixed were executed. Test number 9, which caused defect number 4, was not executed on Code Drop 2, because defect number 4 is not fixed.
Defect number 1 is fixed, because tests 2 and 3 have finally passed. Test number 7 still fails. Therefore, the defect remains.
Test number 13 is a new test, and it caused a new defect.
We chose not to execute tests that had passed on Code Drop 1. This may often not be the case, since turmoil in our code or the area's importance (such as a new feature, an improvement to an old feature, or a feature as a key selling point of the product) may prompt us to re-execute these tests.
This simple, but efficient approach ensures that our matrix will never look like the matrix below (in order to more clearly show the problem, we will omit the Defect # column after each code drop). We will also consider Code Drop 5 to be our final regression pass.
We will address tests 2, 7, and 9 later, but here are a few key points to notice about this matrix: Why were tests 1, 4, 5, 6, 10, 11, and 12 executed up to five times? They passed every single time. Why were tests 3 and 8 executed up to five times? They first failed and were fixed. Did they need to be executed on every code drop after the failure?
If test 13 failed, was the testing team erroneously told it had been fixed on each code drop? If not, why was it executed four times with the same result? We can also ask the question: "Why isn't it fixed?" But we will not concern ourselves with that issue, since we are only addressing the topic of regression.
In conclusion, we will list some general rules we can apply to our testing effort that will ensure our regression efforts are justified and accurate. These rules are:
1. A test that has passed twice should be considered as regressed, unless turmoil in the code (or other reasons previously stated, such as a feature's importance) indicates otherwise. By this we mean that the only time a test should be executed more than twice is if changes to the code in the area the test exercises (or the importance of the particular feature) justify sufficient concerns about the test's state or the feature's condition.
2. A test that has failed once should not be re-executed unless the developer informs the test team that the defect has been fixed. This is the case for tests 7 and 9. They should not have been re-executed until Code Drops 4 and 5 respectively.
3. We must implement accurate algorithms to find out what tests that have already passed once should be re-executed, in order to be aware of situations such as the one of test number 2. This test passed twice after its initial failure and it failed again on Code Drop 4. Just as an additional note of caution: "When in doubt, execute."
4. For tests that have already passed once, the second execution should be reserved for the final regression pass, unless turmoil in the code indicates otherwise, or unless we do not have enough tests to execute. However, we must be careful. Although it is true that this allows us to get some of the regression effort out of the way earlier in the project, it may limit our ability to find defects introduced later in the project.
5. The final regression pass should not consist of more than 30% to 40% of the total number of tests in our suite. This subset should be allocated using the following priorities:
a. All tests that have failed more than once. By this we mean the tests that failed, the developer reported them as fixed, and yet they failed again either immediately after they were fixed or some time during the remainder of the testing effort.
b. All tests that failed once and then passed, once they were reported as fixed. c. All, or a carefully chosen subset of the tests that have passed only once.
