THE NECESSARY SOFTWARE MEASUREMENT KNOWLEDGE IN SOFTWARE
ENGINEERING EDUCATION FROM THE PRACTITIONERS’ POINT OF VIEW
Monica Villavicencio
1,2, Alain Abran
1 1École de technologie supérieure, Montréal, Canada
2CIDIS-FIEC, Escuela Superior Politécnica del Litoral (ESPOL), Guayaquil, Ecuador
ABSTRACT
The enhancement of the teaching of software measurement in software engineering education must be based on a sound assessment of the impact of previous teaching and the needs in industry. This paper investigates through a survey of software practitioners their opinions about the software measurement knowledge that new software engineers currently bring to their employers’ organizations. In addition, the survey asked for their opinions on software measurement topics they see as needed in software engineering university curriculum. Survey findings indicate that basic concepts of software measurement are of great relevance for organizations. Additionally, the software measurement process, techniques and tools, measures for the requirements phase and standards are some of the preferred topics to be taught in software engineering programs.
Index Terms— Software measurement, software engineering, engineering education.
1. INTRODUCTION
Software measurement is one of the factors that influence Software Process Improvement (SPI) [1, 2]. According to the literature, some of the key reasons identified for measuring software processes and products are [3-6]: 1) to understand the current software development process; 2) to monitor and verify the progress of projects in course; 3) to improve the quality of the products generated; 4) to improve productivity; 5) to experimentally validate best practices; 6) to observe trends; 7) to know if the process and product goals have been met; 8) to conduct benchmarking; and 9) to make better decisions.
Although the number of studies related to software measurement is growing, only a few are addressing the teaching challenges of this subject [2, 7, 8]. To improve the teaching and learning of software measurement topics in software engineering programs at the undergraduate level a series of studies have been planned at École de technologie supérieure (ETS), including the two summarized in [9, 10].
This paper reports on a third survey of software practitioners to gain insights on:
A) The impact in industry of the teaching of software measurement in software engineering.
B) The software measurement topics that should be taught in undergraduate programs in software engineering.
The remainder of this paper is organized as follows. Section 2 describes the research methodology. Section 3 presents the survey findings and Section 4 presents the conclusions.
2. RESEARCH METHODOLOGY
To gather data for this study a survey was administered to software practitioners. The sample includes, by design, only professionals working on software process improvement programs as well as software measurement specialists from private or public organizations. Software measurement is a specialized subject and, software measurement-related issues in organizations should be handled by qualified practitioners, such as those in charge of software process improvement programs. The list of worldwide potential survey respondents was obtained from several sources, including: software measurement associations (i.e. GUFPI-ISMA); software measurement conferences organizers (i.e. IWSM-MENSURA, UKSMA) and digital libraries (i.e. engineering village and IEEE Xplore). From these sources, 659 people were contacted by e-mail and invited to participate. The leaders of the contacted organizations (i.e. GUFPI-ISMA, IWSM-MENSURA, etc) were asked to forward the e-mail to their colleagues and to provide to the researchers the total number of recipients. Based on the information provided by them, 292 extra people received the invitation. In addition, an announcement of the study was published in Linkedin specialized groups (i.e. Measurement & analysis forum). Sixty one practitioners answered the questionnaires from March 30 to May 30, 2011. Out of this sample, only 52 were considered valid responses, meaning that all the data was provided for a proper analysis (see Table 1).
2012 25th IEEE Canadian Conference on Electrical and Computer Engineering (CCECE) 978-1-4673-1433-6/12/$31.00 ©2012 IEEE
Table 1: Demographic information (N=52)* Organizations characteristics % Location of headquarter North America 27 South America 35 Europe 37 Asia 2
Size of the organization**
Micro (< 10 employees) 17
Small (< 50 employees) 8
Medium (<250 employees) 15
Large (250 or more employees) 60
Business sector
Aerospace-Automotive 6
Banking 8
Computers and software 33
Defense 2
Education 4
Electricity - water – gas 4
Electronics 4
Finance and business services 6
Government 4 Insurance 2 Manufacturing 2 Professional services 6 Telecommunications 14 Other 8 Market Local 29
Both - mainly local 40
Both - mainly foreign 31
SPI programs with SPI 96 without SPI 4 Certification with certification 39 without certification 62 Measurement programs
with measurement program 52
without measurement program 48
Role of the respondent
Requirements or Business Analyst 8
Software architect or Designer 8
Developer-Programmer 6
QA-Tester 6
Project manager leader 27
IT-MIS manager 4
Metrics manager – consultant 35
Other 8
(*) The total may be greater than 100% due to rounding
(**) European commission of micro, small and medium-sized enterprises[11]
This survey sample is partially biased since people attending international software measurement conferences are therefore typically much more specialized in the main topic of such conference than the general population of software developers or software engineers. On the other
hand, this is a positive bias in the sense that they are more competent to assess and answer this specialized survey within their domain of expertise; much more that the overall none specialized population of software and non software engineers. The survey instrument was pilot-tested with two distinct questionnaires for two types of respondents: practitioners and consultants. This pilot-testing was performed twice: 1) using at least 3 individuals to test the initial version of each questionnaire; and 2) using the IWSM-MENSURA 2010 attendees who volunteered to take part in the survey (see details in [10]). The two questionnaires of the pilot-test were next consolidated into one instrument since the initial versions of the questionnaires had similar questions. Based on the lessons learned from the pilot tests, the wording and structure of the questions were improved.
3. RESULTS AND ANALYSIS
This section presents the findings of the questionnaire for practitioners. The questionnaire included 22 questions divided in two sections: 1) general information; and 2) software process improvement and measurement. Table 1 presents the demographics of the 52 respondents representing organizations from 18 countries. From them, 96% of the organizations have software process improvement programs and 39% are certified.
Table 2 shows a detailed view of the SPI programs followed by the respondents’ organizations. It is important to mention that this classification refers not only to organizations following SPI programs but also to those that are in the process of getting certified. Table 2 shows that 46% of the respondents have “in house” SPI programs, 40% are with ISO9001 and 37% with CMMI. Table 3 shows that ISO9001 and CMMI certifications are prevalent within the organizations already certified.
Table 2: SPI programs followed by organizations
Type of SPI program %
None 4 In house 46 CMMI 37 ISO9001 40 ISO9126 15 ISO15504 4 Six Sigma 17 Other 23
Table 3: Certifications obtained by organizations Certification % CMMI 45 ISO9001 85 ISO27000 15 Other 15
Table 4 presents the percentages of employees in the organizations (column 1) that respondents categorized by educational level (columns 2 to 5): high school, bachelor, masters and PhDs. To better understand this table, what follows is a brief explanation of how the employees are categorized in high school level: 50% of the organizations do not have employees in this category; 27% of them have equal or less than 25% of their employees with high school level; 15% with more than 25% and less than 50% employees in this category; and so on.
Table 4: Education level of employees
% Employees in the organization % High School % Bachelor Master % PhD % 0% 50 15 10 42
Equal or less than
25% 27 12 48 56
More than 25%
and less than 50% 15 15 12 0
Equal or more than 50% and less than
75% 6 35 15 0
Equal or more than
75% 2 23 15 2
For a better insight on the categorization of employees in certified organizations, Table 5 shows the data divided in certified and non certified organizations. According to these percentages, certified organizations represented in this survey hire more people with master and PhD degrees than those without certification.
Table 6 presents the percentages of certified organizations according to their size; while Table 7 shows the type of certifications of these organizations. It is interesting to highlight that the certified-micro organizations have an employee population with master and PhD degrees. The representativeness of this sample with respect to the overall population is not known.
Table 6: Certified organizations according to their size
Size of the organization % Micro 15 Small 5 Medium 10 Large 70
Table 7: Certifications vs. the organizations’ size
% Type of Certification (N2=20)
Size of the
organization CMM 9001 ISO ISO27000 Other
Micro 11 12 0 33
Small 0 6 0 0
Medium 11 6 0 33
Large 78 76 100 33
To explore the characteristics of employees, respondents were asked to rank the level of software measurement knowledge possessed by new employees coming with bachelor and master degree level (see Table 8). An ordinal scale from “none” to “more than expected” was used to capture the perception of respondents regarding the initial knowledge that their employees or team members had at the beginning of their measurement activities. It is worth mentioning that each level of the scale had an explanatory text to avoid misinterpretations and subjectivity. For example, “Little” and “None” were presented as follows:
Table 5: Educational level of employees in certified and non certified organizations
% Employees
% High School % Bachelors % Masters % PhD
Cert. Non Cert. Cert. Non Cert. Cert. Non Cert. Cert. Non Cert. 0% 35 58 20 13 10 9 20 56
Equal or less than 25% 40 18 20 6 45 50 75 44
More than 25% and less
than 50% 20 12 15 16 10 13 0 0
Equal or more than 50%
and less than 75% 5 9 15 47 15 16 0 0
“Little”: New employees have heard about some measurement concepts and remember them, but they are not able to work right away in measurement activities into the organization. Some training is required to let them understand how to apply the theory in practice.
“None”: Non knowledge of software measurement. A whole training program is needed to teach them the basis of measurement and their application in the real world.
Table 8 shows that nearly 60% of respondents consider that new employees with a bachelor degree have little or none knowledge in software measurement. For employees with a master’s degree, nearly 27% of respondents believe that these employees have good or more than expected knowledge in this subject compared to the 13% reached by the bachelors in these scale levels. Nevertheless, 50% of respondents think that masters have little or none knowledge in software measurement.
To complement the collected data, respondents were asked to choose, in order of importance, three software measurement topics that should be emphasized in university courses, according to the relevance for their organizations. The questionnaire included the 12 topics of the software measurement body of knowledge [12], known as the SWEBOK guide – chapter 12 [13]. They were listed in the same order of appearance of the abovementioned document. Only one of the topics was not selected by respondents as the three most relevant, that is, “Measures for the maintenance process”.
Table 9 summarizes the percentage obtained for each topic. The first column (% Total) shows the percentage without considering the certification status, while the others (% Cert. and % Non Cert) do consider certification for comparison purposes. The first topic of the list “Basic concepts in software measurement” was considered as the most important by respondents regardless of the certification status of their organizations. For positions of importance 2 and 3 there are differences between certified and non certified organizations. For the former, the topics B (The measurement process) and C (Measurement standards) are both important. On the other hand, non certified organizations are of the opinion that the topics J (Software engineering management measures) and D (Techniques and Tools) are the ones that deserve positions 2 and 3. One possible explanation for these results might be that non certified organizations have difficulties managing their processes: that may be why they need measures, techniques and tools.
Another important finding is that practitioners’ opinions vary significantly for two of the software measurement topics, including: F (Measures for the requirement phase) and I (Measures for testing). In the case of certified organizations, it seems to be crucial to do effort estimation for project planning, assuming that the functional size of the product (measures for requirement phase) is known. It is also important for these organizations to know, for instance, the number of the specified, executed and failed test cases
(measures for the testing phase) to determine the cost of software quality.
Table 8: Perceived software measurement knowledge of new employees
Perceived knowledge in software
measurement of new employees
Educational level of employees
% Bachelor % Master
More than expected 9 8
Good 4 19
Normal 30 23
Little 30 35
None 26 15
Table 9: Software measurement topics to be taught at university courses
Software Measurement Topics
% Total % Cert. % Non Cert. A. Basic concepts in software
measurement 60 55 63
B. The measurement process 40 45 38
C. Measurement standards 27 40 19
D. Techniques and tools 40 30 47
E. Repositories for quantitative data 4 5 3
F. Measures for requirement phase 23 35 16
G. Measures for the design phase 2 5 0
H. Measures for construction phase 2 0 3
I. Measures for testing 10 25 0
J. Software engineering
management measures 38 15 53
K. Measures related to quality 38 30 44
4. CONCLUSIONS AND FUTURE WORK
The present study was designed to gain insights about the opinion of software practitioners regarding a number of issues related to software measurement teaching at the university level. The focus of this paper was on 3 key issues: the educational level of the employees’ population working at software related organizations or software engineering departments; the level of software measurement knowledge of new employees; and the software measurement topics that practitioners consider as important to be taught at university.
The findings suggest that there is a need, especially at the undergraduate level, for improving the teaching of software measurement in higher education: new software engineering hires from universities present a lack of knowledge on software measurement subject. This confirms Jones 2008 concern about the absence or inadequate training on software measurement that undergraduate and graduate students receive at universities [14]. Another important observation is that in this survey sample working
in software process improvement and measurement programs, is mainly composed by people holding a bachelor diploma. Furthermore, the results indicate that no clear direction is observed regarding a consensus on software measurement topics in university programs. This means that the differences in opinions of people coming from certified and non certified organizations do not allow scholars to have clear guidance for improving the teaching of software measurement. The findings, however, provide some initial thoughts for further research to determine common areas that have to be emphasized in university curriculum for software engineers.
To address some of the educational issues identified in this survey, a Delphi study has been planned for 2012. It will include a survey for asking software measurement experts the same questions regarding the topics that have to be emphasized in higher education. They will also be asked about the complementary skills to be instilled in university students. There will be two experts’ panels: university teachers and practitioners. Both types of participants will be selected according to certain criteria to assure their level of expertise.
REFERENCES
[1] A. Rainer and T. Hall, "A quantitative and qualitative analysis of factors affecting software processes," Journal of Systems and Software, vol. 66, pp. 7-21, 2003.
[2] C. Gresse von Wangenheim, et al., "Empirical evaluation of an educational game on software measurement," Empirical Software Engineering, vol. 14, pp. 418-452, 2009.
[3] N. E. Fenton and S. L. Pfleeger, Software Metrics: A Rigorous and Practical Approach. London: PWS Publishing Co., 1998.
[4] W. Florac and A. Carleton, Eds., Measuring the software process. Statistical process control for software process improvement (The SEI Series in Software Engineering. Westford, Massachusetts: Addison Wesley, 1999.
[5] A. Gopal, et al., "Measurement programs in software development: determinants of success," IEEE Transactions on Software Engineering, , vol. 28, pp. 863-875, 2002.
[6] C. Gresse von Wangenheim, et al., "Software Measurement for Small and Medium Enterprises - A Brazilian-German view on extending the GQM method," in 7th International conference on Empirical Assessment in Software Engineering (EASE), Keele University, Staffordshire, UK, April 8 -10th, 2003.
[7] M. Bush and N. Ashley, "Metkit: toolkit for metrics education," Software, IEEE, vol. 10, pp. 52-54, 1993.
[8] J. Cuadrado-Gallego, et al., "Visual learning techniques for software measurement," presented at the 4th International Conference on Computer Science and Software Engineering, Montreal, Quebec, Canada, 2011.
[9] M. Villavicencio and A. Abran, "Software Measurement in Software Engineering Education: A Comparative Analysis,"
in International Conferences on Software Measurement IWSM/MetriKon/Mensura, Stuttgart, Germany, 2010, pp. 633-644.
[10] M. Villavicencio and A. Abran, "Facts and Perceptions Regarding Software Measurement in Education and in Practice: Preliminary Results," Journal of Software Engineering and Applications vol. 4, pp. 227-234, 2011-04-29 2011.
[11] EuropeanCommission. (2009, 28 Abril 2010). Definition of micro, small and medium-sized enterprises. European Commission, Enterprise and Industry. Available:
http://ec.europa.eu/enterprise/policies/sme/facts-figures-analysis/sme-definition/index_en.htm
[12] A. Abran, et al., "Software Measurement Body of
Knowledge," in Encyclopedia of Software Engineering vol. 1:1. London: Taylor & Francis, 2010, p. 1157 — 1168. [13] P. Bourque, et al. (2008, October 15, 2010). SWEBOK
Version 3. 18. Available: http://www2.computer.org/
cms/Computer.org/SWEBOK/MeasurementKA-Draft-Feb2008.pdf
[14] C. Jones, Applied Software Measurement: Global Analysis of Productivity and Quality, Third ed.: McGraw-Hill Osborne Media, 2008.
