2
Course Information
Specification Author:
Nikita Nalyutin, Associate Professor, PhD (Science and Technology Studies: Software Engineering)
Subject Title in English:
Software testing and quality assurance
1. Application Guidelines and Regulations
This specification presents a detailed description of the educational purpose, curriculum, and assessment methods for the discipline “Software testing and Quality Assurance”. This course is delivered to students of master’s program “Software Engineering”, computer science faculty, HSE.
The specification is intended to be used as a source of information by:
students and potential students;
lecturers delivering lectures and conducting practical classes on the course or related disciplines;
professional and statutory regulatory bodies when carrying out accreditation.
2. Course Objective
Primary course objective is to develop student professional competences in software testing and verification techniques and to give them practical skills for software testing tools usage.
Course topics set is based on enterprise testing approaches and contains topics on test environment managements, test planning, coverage analysis and improvement, integration and load testing, UI testing. Also, course covers project documentation verification and formal inspection process. At the end of the course the overview of Quality Assurance process is given.
3. Learning Outcomes
During the course, the students will:
Study the basic terms, definitions and principles of software testing and quality assurance;
Study state-of-the-art technologies and products for software testing;
Study regulatory documents in the area of software testing;
Master methods and tools for development of robust test cases for different types of software systems;
Master industrial standards for software quality assurance;
Master models of testing process in terms of lifecycle phases, role based project structure, test effort management, test coverage.;
Acquire practical skills in test analysis and test development;
Acquire practical skills in test management and test planning;
The course contributes to the development of the following professional competencies [3]:
3 1. Research activities
The ability to manage both self-paced and team work on research projects (ПК-3).
2. Project activities
The ability to approach projects in a systematic way, to build and apply descriptive and forecasting models, to conduct qualitative and quantitative analysis (ПК-6);
3. Technical activities
The ability to apply state-of-the-art development technologies for the development of complex software systems, to take the advantage of automated job scheduling and control tools, and to make use of quality assurance techniques (ПК-9).
4. Discipline in the Educational Program
Curriculum:
The course length is 80 academic hours, including 20 hours of lectures, 20 hours of practice, and 40 hours of self-study. Academic control forms are one home assignment checked at the final practical seminar. It is a part of specialized curricula unit, and it is delivered in module 1 of the first academic year. Number of credits is 3.
Prerequisites:
The course is based on the knowledge of foundations of general technical disciplines, mathematics, computer science and fundamentals of the decision theory.
4
5. Subject Structure and Contents
No. Topic name Course hours,
Total
Audience Hours
Self-study Lectures Practical
studies
1.
Software testing techniques and approaches. Testing in software development life cycle.
4 2 2 4
2.
Test analysis and test development. Test coverage. Repeatability of testing.
Regression testing 4 2 2 4
3.
Test process documentation: from test
strategy to test coverage reports. 4 2 2 4
4.
Formal inspections
4 2 2 4
5. Unit testing 4 2 2 4
6.
Integration testing
4 2 2 4
7.
User interface testing
4 2 2 4
8.
System testing
4 2 2 4
9.
Robust software development techniques
4 2 2 4
10.
Enterprise approaches for test process
organization. Quality Assurance. 4 2 2 4
Total: 80 20 20 40
6. Grading and Assessment
Type Form 1 year Notes
1 2 3 4 Progress
check
Homework assignment
* 1) Research paper
2) ISTQB mock tests 3) Defect reports case study
Final check Oral exam Talk on research paper topic and
several ISQTB-like tasks
6.1. Evaluation criteria Homework assignment
5 Homework consists of 3 parts:
1) Research paper development and analysis. The students select the subject based on their interests. Once approval of the topic is obtained from course instructor, the student writes the research paper based on the information from public sources, personal experience and imagination.
2) Passing of 2 mock tests for International Software Testers Qualification Board (ISTQB) certification. Link to the tests is given by course instructor.
3) Preparation of several defect reports for Web sites selected by students.
Evaluation criteria for student’s homework assignment are presented in the table below. A maximum score of 10 can be achieved.
Development research paper Analysis of research paper
• Relevancy of information
• Real-life story basis
• Structure of text
• Sufficiency of selected material
• Freshness of ideas
• Accuracy of problem statement
• Author’s contribution and analytical processing of information from public sources
• Demonstration of good command of theoretical knowledge
• Ability to identify both common and specific problems
• Exploring various solutions
• Openness to different perspectives
• Being persuasive in argumentation
• Application of risk management principles and decision making methods
Oral exam
Final exam is in the form of homework review. Student obtain 3 questions on research paper topic, 2 tasks similar to ISTQB test and 2 questions on defect report. Evaluation criteria for student’s homework assignment are presented in the table above. A maximum score of 10 can be achieved.
Penalties
Should plagiarism be identified in the student’s homework, disciplinary measures are applied as appropriate per the HSE Charter.
6.2. Overall Score
Overall score on the course Ofinal is determined using the following formula:
Ofinal = (Ohome<4||Oexam<4)? min (Ohome, Oexam):
0,5*Ohome+0,5Oexam where
- Ohome- score achieved by the student for the homework assignment;
- Oexam - score achieved by the student for the written exam.
Should the student fail to either pass the written exam or submit the case before the end of the course, overall score for the next attempts is determined using the following formula:
Ofinal = (Ohome<4||Oexam<4)? min (Ohome, Oexam):
0,8*(0.5Ohome + 0.5Oexam)
6 A score of 4 or higher means successful completion of the course (‘pass’). A score of 3 or lower means failure to complete the course (‘fail’).
7. Detailed Curriculum Plan
Topic 1. Software testing techniques and approaches. Testing in software development life cycle.
1. Software development life cycle 2. Life cycle models
Waterfall
Spiral
Prototyping
Rapid
3. Software development process frameworks:
Microsoft Solutions Framework
Rational Unified Process
eXtreme Programming
Agile
Scrum
4. Roles in software development team, role interaction for different processes
5. Goals of software testing. Difference between testing and quality assurance. Difference between testing, verification and validation.
6. Types of testing process and test integration into the lifecycle:
Unit testing
Integration testing
System testing
Load testing
Static testing techniques
7. Testing of mission-critical software and the difference with business software.
Topic 2. Test analysis and test development. Test coverage. Repeatability of testing. Regression testing
.
1. Test approaches:
Black box
White box
Model checking
Static checking 2. Test environment
Drivers and stubs
Testing classes
Signal generators 3. Test cases
Requirements based test case development
Explorative test case development
7
Equivalence partitioning
Test plan
Test traceability
Test execution statistics 4. Test coverage
Requirements and code coverage
Types of code coverage: lines, branches, condition, decision, condition/decision, modified condition/decision.
Equivalence partitioning
Boundary conditions
Robustness checks
Coverage completeness checks
5. Repeatability of tests in enterprise environment, regression testing
6. Test pre-condition, test dependencies, test environment management, types of test environments
Topic 3. Test process documentation: from test strategy to test coverage reports.
1. Test processes and test documentation for each process.
2. Test strategy 3. Test plan
4. Test requirements. Traceability, completeness, consistency, testability.
5. Test scenarios. Types of test scenarios (sequences, checklists, tables, automata). Test generators.
6. Test execution reports. Types of test execution reports. Execution reports analysis.
7. Coverage reports. Types of coverage reports. Coverage reports analysis.
8. Defect reports. Types of defect reports. Defect reports analysis, release planning based on defect reports.
Topic 4. Formal inspections.
1. Types of static techniques. Types of project reviews. Difference of formal inspection from other types of review.
2. Formal inspection roles, life cycle
3. Code formal inspection: role setting, preparation, results documentation
4. Requirements formal inspection: role setting, preparation, results documentation
Topic 5. Unit testing.
1. Goals of unit testing 2. Unit boundaries
3. Test environment architectural approaches 4. Data for unit testing
5. Unit testing repeatability, continuous integration
8 Topic 6. Integration testing.
1. Goals of integration testing 2. Integration testing approaches:
Early integration
Big bang integration
Layer-based integration
Hierarchical integration 3. Module interface testing
4. Requirements limits, testing missing integration requirements.
Topic 7. User interface testing.
1. Goals of user interface testing
2. User interface testing approaches: manual testing, scenarios, event simulation 3. Interface testing automation, test supportability
4. Completeness criteria for user interface testing
Topic 8. System testing.
1. Goals of system testing 2. System testing planning 3. Test environment set-up 4. Load testing
5. Mission critical software system tesing
Topic 9. Robust software development.
1. Critical points 2. Assertions
3. Exception handling
4. Automated code robustness analysis
Topic 10. Enterprise approaches for test process organization. Quality Assurance.
1. Quality assurance. Goals. Quality management (ISO 9000, CMMI). Quality audits.
Corrections and corrective actions. Process optimization
2. Configuration management. Goals. Identification. Version control. Life cycle control.
Traceability. Baselines.
3. Enterprise-level quality management
9
8. Assessment Methods
Research paper
Sample topics:
1. Pros and cons for explorative testing and scenario-based testing 2. Pros and cons of waterfall vs. agile, testing in these lifecycles 3. Project risks for software testing
4. Configuration management for software testing 5. Testing of high-load internet sites
6. Test tool selection strategy for software projects 7. Testing vs. quality assurance
8. Context testing school
9. Test effort estimation techniques
10. Move from manual to automated testing: benefits and problems 11. Test team management: goals, tasks, risks
12. UX and usability testing techniques
13. Regression testing as project health indicator
14. Test end criteria and their impact to software quality 15. Test effectiveness estimation
16. Types of test coverage 17. Test documentation flows
18. Testing without requirements - pros and cons 19. Test stragegy
20. Outsourced test management 21. Project risks for outsourced testing 22. Mobile applications testing
Assesment:
Evaluation of research paper text as a homework is performed using academic-wide set of criteria to evaluate research papers (completeness, innovation, original material etc), giving maximum of 6 points to Ohome.
On oral exam student should answer to 3 questions based on his research paper topic, giving maximum 6 points to Oexam (2 points each question).
Mock tests
Samples of mock tests questions:
1) An input field takes the year of birth between 1900 and 2004. The boundary values for testing this field are:
A. 0,1900,2004,2005 B. 1900, 2004
C. 1899,1900,2004,2005
D. 1899, 1900, 1901,2003,2004,2005
10 2) Pick the best definition of quality
A. Quality is job one B. Zero defects
C. Conformance to requirements D. Work as designed
Assesment:
During the homework student should pass minimum 2 mock tests giving him up to 4 point (2 points each test) to Ohome.
On oral exam student should answer to 2 questions similar to questions in mock test, giving him up to 2 points to Oexam.
Defect reports
During the homework student should prepare minimum 2 defect reports for his favorite Web site, giving him no extra points to Ohome.
On oral exam student should answer to 2 questions on his defect reports, giving him up to 2 points to Oexam.
9. Learning Resources (in Russian and English)
1. Блэк Р. Ключевые процессы тестирования. Планирование, подготовка, проведение, совершенствование. М.: Лори, 2006. // Rex Black. Critical Testing Processes: Plan, Prepare, Perform, Perfect
2. Савин Р. Тестирование Дот Ком, или пособие по жестокому обращению с багами в интернет-стартапах. М.: Дело, 2007.
3. Липаев В.В. Тестирование компонентов и комплексов программ. М.:Синтег, 2010.
4. Криспин Л., Грегори Д. Гибкое тестирование. Практическое руководство для тестировщиков ПО и гибких команд. М.:Вильямс, 2010..// Lisa Crispin and Janet Gregory. Agile Testing: A Practical Guide for Testers and Agile Teams
5. Бейзер Б. Тестирование черного ящика. Технологии функционального тестирования программного обеспечения и систем. СПб.:Питер, 2004. // Boris Beizer. Black-Box Testing: Techniques for Functional Testing of Software and Systems
6. Канер С., Фолк Д., Нгуен Е.К. Тестирование программного обеспечения.
Фундаментальные концепции менеджмента бизнес-приложений. М.:ДиаСофт, 2001. //
Cem Kaner et al. Testing Computer Software, 2nd Edition
7. Дастин Э., Рашка Д., Пол Д. Автоматизированное тестирование программного обеспечения. М.:Лори, 2003. // Elfriede Dustin et al. Automated Software Testing:
Introduction, Management, and Performance: Introduction, Management, and Performance 8. Тамре Л. Введение в тестирование программного обеспечения. М.:Вильямс, 2003. //
Louse Tamres. Introducing Software Testing
9. Котляров В.П., Коликова Т.В. Основы тестирования программного обеспечения.
М.:ИНТУИТ, 2009.
10. Синицын С.В., Налютин Н.Ю. Верификация программного обеспечения. М.:ИНТУИТ, 2008.
11. Синицын С.В., Налютин Н.Ю. Верификация программного обеспечения.
http://www.intuit.ru/department/se/verify/
11 12. ГОСТ Р ИСО 9001-2001. Системы менеджмента качества. Требования
М.: ИПК Изд-во стандартов, 2001. // ISO 9000 Quality Systems Handbook
13. RTCA/DO-178B. Software considerations in airborne system and equipment certification RTCA Inc, 1992.
14. IEEE 1012-1998. IEEE Standard for Software Verification and Validation Institute of Electrical and Electronics Engineers. 01-May-1998.
15. IEEE 829-1998. IEEE Standard for Software Test Documentation Institute of Electrical and Electronics Engineers. 01-May-1998.
16. IEEE 1008-1987. IEEE Standard for Software Unit Testing. Institute of Electrical and Electronics Engineers. 01-May-1987.
17. ГОСТ 27.002-89. Надежность в технике. Основные понятия. Термины и определения.
М.: Издательство стандартов, 1990.
18. ГОСТ 13377-75. Надежность в технике. Термины и определения. М.: Изд-во стандартов, 1975.
19. IEEE 1042-1987. IEEE Guide to Software Configuration Management. Institute of Electrical and Electronics Engineers. 10-Sep-1987.
20. Майерс Г. Искусство тестирования программ. М.:Финансы и статистика, 1982, 176 с //
Glenford Myers. The Art of Software testing
21. ISO/IEC 9126-1:2001. Software engineering -- Product quality -- Part 1: Quality model International Organization for Standardization/International Electrotechnical Commission.
01-Jun-2001.
22. Microsoft Solutions Framework. Методология создания программных решений http://www.microsoft.com/Rus/Msdn/msf/Default.mspx
Special Equipment
Practical studies are conducted in a computerized laboratory equipped with an overhead projector and audio portables. Every student has a PC with high speed internet connection and access to the HSE electronic library.
References
1. Master curriculum of MSc educational programme 09.04.04“Software Engineering”, specialization “Software development management”. Moscow, HSE, 2014.
2. Curriculum of 1st year MSc educational programme 09.04.04 “Software Engineering”, specialization “Software development management”. - Moscow, HSE, 2014.
3. Educational standard of the Federal State Autonomous Educational Institution for higher professional education "National Research University "The Higher School of Economics"
(HSE)/educational course area code 09.04.04 "Software Engineering", MS degree level.
The author of the program: ______________Nalyutin N.
