8.1 Conclusion
The aim of this thesis is to investigate the best test and QA practices in industry and to design and evaluate a process for implementing best practices in the software lifecycle of a small to medium enterprise (SME) over successive projects. This thesis was the culmination of over five years of software testing and quality assurance research and practice improvements for software projects in two different SME organisations. To this end the aim of the thesis has been successfully completed. Each of the four objectives in succession led to the resolution of a quality problem in one organisation and for the creation of a framework of proven test and QA practices.
The research into software testing was insightful and of benefit for testing multiple products in different company‟s. Testing is difficult and requires detailed test plans. These plans must tie the testing approach to the software design and development schedule. This requires careful consideration of the product and demands that resources are prepared in advance of testing. The test plan ideally should be risk based so that it can yield better test benefits where test execution time is limited. Software testing is not sufficient in its own right to ensure that a quality product is realised. There are other quality factors that have to be considered and planned into the project lifecycle. The software test plan should tie in with a project lifecycle process. This project lifecycle process needs to incorporate quality assurance for each deliverable of the project stages to address the quality factors.
Quality assurance from all team members in addition to testers is needed to address all quality factors. The testing of software and QA of each software deliverable requires structure and needs to be an endemic part of a project team. Where each project raises its own difficulties, a process for having QA at each stage of the project is a benefit in surmounting such obstacles The QA process needs to be incorporated into the project lifecycle with the facility for improvements at project end for feed back into the next project, this continuity of process refinement aids with quality improvements.
If the QA process consists of a combined development and testing process, it is more beneficial in improving the quality of each project phase. With the emphasis of quality in this process, the experience of the QA team can strengthen the project team as a whole in the mindset of Quality Assurance. While the QA process is a combined effort, if the QA team can report independently of the development team, it can be more effective than a dependent team. In addition to an independent QA team, the inclusion of customers in the QA aspect of the project can also have a contribution to improved quality and reduced defects. It is also more effective to have the customers assess quality during different stages of the development cycle. The customers themselves may be included or a body of representatives which can assist with determining the quality assessment of the software.
Software quality metrics are required to track the defects and quality improvements at each stage of the project lifecycle. Graphs of the metrics can be used to plot trends over time of these software quality improvements to assist with the management of the test execution and quality initiative.
8.1.1 Limitations of research
There are limitations in this thesis in respect to the quantitative data used to extrapolate the benefits of the research and also due to the individualistic nature of the project work itself.
Where defect rates and lines of code are determined, they are accumulated over several months of project work and are accurate at the point of their recording from the respective artefacts in which they are stored. There is no allowance made for code that was rewritten a number of times. A simple line code counting application was used to determine as best as possible the number of lines of code for each application.
Every effort was made for the allowance of defects that were opened in error and defects that were assigned an incorrect severity as far as was possible. The man hour and milestone dates are representative of the project target dates and scheduled timeframe. Accurate data was accumulated over the duration of six projects over three years of project work, every attempt was made to keep accurate recordings of each projects respective data irrespective of other projects taking precedence and resources being temporarily reassigned.
The other major limitations to the research are that the projects were carried out by many different individuals; each individual had different work experience and education. The number of lines of code and the number of defects detected are attributed to the work of the individual developers and testers respectively on each project. The exact value of each statistic is determined on a project basis and individual allowances are not represented.
8.1.2
Improvements to practicesThe first metric that should be obtained that was not recorded in enough detail would be the number of items detected (design defects) during the review of any design documentation. This could be a peer review of an artefact or the analysis of a document during the test design stage. The cost benefits analysis of the time spent on reviews would be more transparent and support the early inclusion of QA in the projects. This is not the case in most projects.
During the test execution of projects, testing is frequently held up by late delivery of builds or that certain features are not implemented, these test blockages (blocked test cases) should also be recorded as evidence of delays that are not attributed to testing, it would be prudent to include test cases that are blocked and for the duration in which this is the case. Once the testing phase begins, any delays are automatically assumed to be the result of the testing itself. This is frequently not the case.
The additional metrics of design defects and blocked test cases would further support the case for QA reporting on software quality before there is a number of test cases executed and defects reported. It is frequently too late to make significant changes to the software at the test execution phase. The inclusion of metrics at the end of each project phase (the Go / No-Go meetings) would again add weight to any opinions expressed in terms of software quality before proceeding to the next phase. The enthusiasm of developers can often out weigh the pessimism of QA when a project manager‟s project is under the scrutiny of senior management at meetings.
8.2 Further Work
The areas that could be explored further in relation to this testing process is to be more accurately with the test effort and test outcome. The determination of test effort in terms of the number of resources (man hours) and the test outcome in terms of the number of defects anticipated that a project would produce from both testing and development perspectives based on the number of function points.
Two of the projects were developed off shore, this is an increasingly more frequent approach to software projects and it is an area worth examining further in relation to GSD (Global software development) and the testing of the software developed in this manner. It is increasingly more difficult to co-ordinate a distributed team (virtual team) of developers, testers or business analysts for the purposes of artefact reviews, team meetings and deployment of software builds and releases.
During the testing of some of the projects some of the test cases were automated in conjunction with the maintenance of the test cases. This is a worthwhile activity, but the test automation tools are frequently of the record and playback variety which can extend the project lifecycle. The inclusion of test automation during the development and unit testing of components would be an area that would be worth further pursuit. With Java a test tool „Junit‟ was utilized for the unit testing of the applications in project FIIS. This could be extended further and used in a broader sense for System testing the application in conjunction with the test data for further test coverage and extending the automation of tests. The QA effort while very beneficial for early inclusion in the project perhaps would be best utilized for Test Driven Development (TDD where the test cases are developed before the code is actually written.
The Framework was evaluated in a second company. Further research is necessary on the frameworks adoption across different industry sectors and company‟s. Only after this research is conducted would the academic community accept its validity and benefits.