An Improved Comprehensive Evaluation Method for Software Development Ability

(1)

An Improved Comprehensive Evaluation Method for Software Development Ability

Xiao Yu

Beijing Institute of Tracking and Telecommunication Technology

Wenhong Liu Beijing Institute of Tracking and Telecommunication Technology

Jun Yang

China Aerospace Academy of Systems Science and Engineer

Abstract—This article in view of the existing software development capability assessment method based on CMMI in the limitations of technical ability and attention on the quality of the product is not enough, an improved software development ability comprehensive evaluation method, the method on the basis of software development capability maturity evaluation, joined the organization developed the project value of the process of developing software and software problem evaluation after delivery, in a more objective, accurately reflect the software development organization's software development capabilities.

Keywords—CMMI development process evaluation Software problem

I. INTRODUCTION

Aerospace software development is a complex system engineering, which involves many organizations and different levels of capabilities. How to evaluate objectively and accurately the software development capability of these organizations? There are many theories to evaluate the software development process. For example, ISO/IEC 330xx series standards provide an assessment of the information technology process, and CMMI (Capability Maturity Model Integration) provides guidance on the improvement of the software development process and the assessment of the software development capability. CMMI, which is more common in the industry, promotes the continuous improvement of the development process and capability of software development organizations through the requirements of different capability levels and process areas. However, there are still some problems in the actual evaluation of aerospace software development capability.

In this paper, chapter II explains the existing software development capability evaluation method in practice the problems found. Chapter III puts forward combining with the history in the development of the organization bearing research project, join the project in the process of the development of the actual problem back after evaluation feedback and product delivery, as the existing software development capability assessment method of correction and improvement. In chapter IV, the improved software development ability comprehensive evaluation method is used to score two different research organizations according to their original research ability score and the quality information of their actual research projects. Chapter V summarizes and forecasts the other expected work of the improved method.

II. PROBLEMS OF EXISTING SOFTWARE DEVELOPMENT CAPABILITY EVALUATION METHODS

The existing method to evaluate the software development capability of various aerospace software development organizations relies on a set of software development capability maturity model based on CMMI 1.2, which applies

for certification step by step and is evaluated and scored by experts on the spot^[1]. This certification system has covered all relevant software development organizations for more than ten years since its operation, accumulated a large number of software development capability assessment data, and its software development capability maturity level and score have been recognized by all parties, which can reflect the software development capability of software development organizations relatively accurately. However, in the actual operation of the evaluation process, we still find some limitations. Specifically, it includes the following aspects:

The existing evaluation methods of research and development capability do not focus on the management requirements and engineering requirements, and the software management requirements have the same weight as the technical requirements, and the management requirements are more than the technical requirements. In the actual implementation, the development organization has invested a lot of energy in how to realize the management requirements, and the technical capacity has not been effectively improved

[2] [5] [6].

The existing development capability evaluation methods pay more attention to the development process capability of typical projects. At present, the research capability evaluation method is used to evaluate the self-recommended projects of the research organization, which only reflects the research capability of the four projects with the best process, and fails to analyze and evaluate all the historical data of the actual research software of the research organization. There is a wide range of situations in which the level of "model projects" and actual work projects are quite different^[3].

The existing methodology for development capability assessment focuses on the capability assessment of the development process and does not include effective assessment of problems that arise after coordination, testing and delivery. When the development organization receives the evaluation, it will not reflect the problems after the coordination, testing and delivery, so the evaluation team will not be able to accurately understand the actual situation of the software^{[4] [7]}.

Given to the use of software development capability assessment data before, we need to be based on the existing software development capability assessment method to improve, in combination with the history of the development organization bearing research project, combined with the technology in the process of evaluation feedback, back after product delivery problem, with the introduction of these factors to fix software development capabilities, a comprehensive evaluation model to solve the problems in existing research capability evaluation method.

2020 IEEE 20th International Conference on Software Quality, Reliability and Security Companion (QRS-C)

(2)

III. IMPROVED COMPREHENSIVE EVALUATION METHOD FOR SOFTWARE DEVELOPMENT CAPABILITY

A. Develop capability evaluation models

In the process of evaluating software development capabilities, taking into account that if the model contains too many evaluation indicators, it will complicate the processing process and the modeling process, and the determination of the importance of the indicators will become difficult. The possibility will increase; on the contrary, if the number of evaluation indicators is too small, it will be difficult to objectively and comprehensively describe the development status of the system.

Software development capabilities rely on the technical capabilities of software development and testers on the one hand, and reasonable software development process management can reduce unnecessary errors in the development process, which is conducive to later maintenance and improve software quality. High-quality software is inseparable from the high-quality software development process. The software development capability is also directly linked to the quality of the products it develops and is constantly updated in the actual project cumulative data.

Taking into account the above factors, this article comprehensively evaluates the software development capability from two aspects: the software development process side and the project / product cumulative data side.

Among them, the software development capability maturity score is used as the development process side parameter, and the accumulated data of each project / product that the development unit has developed currently is used as the development unit product quality side parameter to obtain.

The specific structure is shown in figure 1.

Improved software development capability scoring model

Process side Project / Product

Cumulative Data Side

Historical Contracting Software Project Evaluation of each project

development process

Software problem evaluation after each product delivery

Software Security Critical Level Software scale lines Number of questions raised by the requirements review Number of questions raised by the

design review Number of developer unit test cases Number of developer unit testProblems Number of developer confirm testproblems

Number of developer confirm testcases Problem1 Problem2 ProblemN

Difficulty of Finding the problem Occurrence stage of the problem

Severity of the problem Difficulty of Finding the problem Occurrence stage of the problem

Severity of the problem

Software development capability maturity

score

Problem Number

Fig. 1. Improved software development capability evaluation model Considering the implementation of the software development capability evaluation, the overall organization of aerospace software development evaluates the development organizations of each subsystem. The overall organization can get a large amount of quality information of historical software projects when collecting data of various models.

Therefore, the evaluation model set up in this way is feasible in practice.

B. Acquisition of evaluation indicators

1) Software Development Capability Maturity Score and Normalization.

Considering that the units involved in the development of aerospace software have all passed the software development capability maturity evaluation, they can directly read in the scores obtained by the unit in the software development capability maturity evaluation.

According to the requirements of the CMMI, the evaluation agency uses the SCAMPI method to evaluate the maturity of software development capabilities, and the evaluated unit provides its own items for measurement, and evaluates each practice area based on the relevant quality data of these items Give each practice a score of S / L / P / NY. The final statistical scoring is based on the principle of wooden barrels, with shortboards as the recording points of abilities. If one goal is not achieved, the overall process area is evaluated as failed; if a process area is not passed, the evaluation at this level is not passed. Then, through the units of the corresponding level of software development capability maturity, all their practices are counted, and the final score is recorded as A. A must be greater than 6 points, the maximum can get 8 points^[10].

There are five levels of software development capability maturity. After reading the corresponding software development capability maturity scores, the scores need to be adjusted through weights based on the maturity of the specific unit. For example, when the scores read are the same, the unit score of the third level of maturity is higher than the score of the second level of maturity.

We set the maturity level 5 weight to 1.0. The lower the maturity level, the lower the weight. If you directly read the software development capability maturity score of an unit as A and the unit's maturity level is N0, assuming that the weight that decreases with each level of maturity level reduction is p0, the unit's software development capability maturity The scoring score calculation formula is:

P= A 1 p (5 N) Among them, the initial value of p can be determined according to the experience of experts. After the historical data is accumulated in the later period, the original data can be further adjusted by obtaining the weights through mathematical processing. A is a continuous value between 0 and 8. The value range of P is 0 to 8.

After normalization, the software development capability maturity score score calculation formula is:

P=A

8 1 p (5 N) 2) Evaluation and normalization of each project development process.

The indicators examined at this stage include the number of requirements review questions, the number of unit test questions, and the number of test cases. The index of each stage has a reference value. The larger the deviation from the reference value, the lower the evaluation score of the development process.

(3)

Set the benchmark values of indicators at each stage as

a

10,

a

20,

a

30..., the measured values are

a

1,

a

2,

a

3..., and select the number of indicators as n. The calculation formula for the project software development process evaluation is:

P= 1 ÷ 1

n(a a)

= n (a a)

Dividing by n takes into account that the number of indicators selected when evaluating different projects may be different.

Suppose that the number of inspection indicators (or the number of inspection indicators of a given standard) in all participating units is at most n, and the normalized calculation formula for P is:

P= 1 ÷ 1

n(a a)

= n

n1 (a a)

3) Evaluation and normalization of software problems after each product delivery.

The indicators for post-delivery problem evaluation and inspection include the difficulty of finding the problem, the severity of the problem, the stage of finding the problem, and the number of problems found.

For each problem, there are three attributes: difficulty of finding the problem, severity of the problem, and the occurrence of the problem. .

Suppose that the difficulty of finding problems includes five levels of 1, 2, 3, 4, and 5. Among them, level 1 is a problem that can be found by performing basic software functions; level 2 is a problem that requires a certain use case design method and can only be discovered by executing a more logical branch of logic; so recursive until level 5 is extremely hidden and difficult to find problem. For each level of difficulty found, the weight increases by , and the difficulty level of the discovered problem is N. The formula for the weight of the evaluation result of the problem after delivery is found to be:

P= 1 p (5 N+ 1) Suppose that the severity level of the problem is found to be 1 (fatal), 2 (severe), 3 (fair), 4 (mild), 5 (recommended). If the severity of the problem is decreased by one level (the corresponding value is increased), then The weight is increased by , and the difficulty level of the problem found is N. The formula for the weight of the evaluation result of the software problem after delivery is found to be:

P= 1 p (5 N+ 1) Suppose that the problem discovery stage includes scheme, requirement, design, coding, use, etc. Assuming that the number of division stages is S, and the corresponding label

values according to the order of the stages are 1, 2, 3, ..., S, the problem discovery stage is next. Phase, the weight is increased by , and the corresponding value of the problem occurrence stage is N. The formula for the weight of the evaluation result of the software problem after delivery is found to be:

P= 1 p (5 N+ 1) Suppose that this stage is divided into B. If a total of k problems are found after the software is delivered, the index value of the difficulty of the problem, the severity of the problem, and the stage of the occurrence of the problem at the ith problem are N, N and N (when there is no problem, k=0, N= 6ǃN= 6ǃN= + 1), the weight of the score value corresponding to the stage is:

P!= " #P P P

= " #[1 p (5 N+ 1) 1 p

$%

$

(5 N+ 1) 1 p

(S N+ 1) ]

As before, the initial values of p, pand p can be determined based on expert experience. After the historical data is accumulated in the later period, the original data can be adjusted by mathematically obtaining weights.

The calculation formula of P_! after normalization is:

P= #[1 p (5 N+ 1) 1 p

$%

$

(5 N+ 1) 1 p

(S N+ 1) ]

C. Comprehensive evaluation method

The comprehensive evaluation method uses a weighted average method. Assume that the full score of the comprehensive evaluation is 100, and the weights of the evaluation results in the three aspects are WǃWǃW, and W+ W+ W= 1 is satisfied.

The calculation formula of the comprehensive evaluation score Q is:

& = 100 ' _$ *_$

$

IV. CURRENT WORK AND VERIFICATION

We use this method of software quality evaluation based on process evaluation and test results to evaluate the price of two different software development units of a certain system A. Improved Evaluation of Software Development

Capability of Company A

For company A, first read its software development capability maturity level II scores as shown in the following table:

(4)

TABLE I. COMPANY A’S SOFTWARE DEVELOPMENT CAPABILITY MATURITY SCORES

Score SG1 SG2 SG3 GG2

Configuration

management 7.00 7.17 7.50 7.00 7.70 Measurement

and analysis 7.13 7.13 7.25 -- 7.80 Project

monitoring 7.14 7.14 7.50 -- 7.65 Project

planning 7.07 7.25 7.07 7.83 7.70 Process and

product quality assurance

6.25 6.50 6.25 -- 7.60

Requirements

management 7.20 7.20 -- -- 7.90 The company is responsible for the development of 3 software configuration items in the supporting software of this system, and the development of each software configuration item can be regarded as an independent development project for statistics. Corresponding statistics of the three software configuration items during the development process are as follows:

TABLE II. COMPANY A’S CSCI

CSCI 1 CSCI 2 CSCI 3 Software Security Critical Level B A D Software scale lines 10574 6437 202768 Number of questions raised by the

requirements review 3 3 2

Number of questions raised by the

design Review 10 7 15

Number of developer unit test cases 763 412 1579 Number of developer unit test

problems 52 35 108

Number of developer confirm test

cases 248 134 583

Number of developer confirm test

problems 10 4 16

The following table shows the software problems of the three configuration items after delivery:

TABLE III. PROBLEMS OF COMPANY A’S CSCI1 Difficulty of

finding problem

Severity of the problem

Occurrence stage of the problem

Problem1 4 general requirement

Problem2 3 general coding

Problem3 2 slight coding

TABLE IV. PROBLEMS OF COMPANY A’S CSCI2 Difficulty of

finding problem

TABLE V. PROBLEMS OF COMPANY A’S CSCI3 Difficulty of

finding problem

Occurrence stage of the problem Problem1 2 general requirement Problem2 3 general design Problem3 3 serious coding Problem4 2 general coding Problem5 2 slight coding Problem6 2 slight coding After the data of this unit was substituted into the value matrix of the relevant parameters of expert experience p and a and calculated, P= 0.871, P= 0.881, P= 0.936, and the final score Q = 89.6.

B. Improved Evaluation of Software Development Capability of Company B

For company B, first read its software development capability maturity level II scores as shown in the following table:

TABLE VI. COMPANY B’S SOFTWARE DEVELOPMENT CAPABILITY MATURITY SCORES

Score SG1 SG2 SG3 GG2

Configuration

management 7.00 7.83 7.00 7.75 7.80 Measurement

and analysis 7.00 7.00 7.88 -- 7.80 Project

monitoring 7.67 7.71 7.67 -- 7.90 Project

planning 7.00 7.50 7.00 8.00 7.60 Process and

product quality assurance

6.75 6.75 7.25 -- 7.80

Requirements

management 7.20 7.20 -- -- 7.60 The company is responsible for the development of 2 software configuration items in the supporting software of this system, and the development of each software configuration item can be regarded as an independent development project for statistics. Corresponding statistics of the two software configuration items during the development process are as follows:

TABLE VII. ^COMPANYB’S CSCI

CSCI 1 CSCI 2 Software Security Critical Level B C

Software scale lines 4186 9633

Number of questions raised by the

requirements review 2 3

Number of questions raised by the design

review 5 4

Number of developer unit test cases 217 341 Number of developer unit test problems 15 23 Number of developer confirm test cases 104 157 Number of developer confirm test problems 3 6

The following table shows the software problems of the two configuration items after delivery:

(5)

TABLE VIII. PROBLEMS OF COMPANY B’S CSCI1 Difficulty of

finding problem

Problem1 2 serious design

Problem3 2 general coding Problem4 1 general coding Problem5 1 general coding

TABLE IX. PROBLEMS OF COMPANY B’S CSCI2 Difficulty of

finding problem

Problem1 2 serious requirement

Problem2 2 serious coding

Problem3 3 general coding Problem4 2 general coding Problem5 2 general coding Problem6 1 general coding Problem7 1 general coding

After the data of this unit was substituted into the value matrix of the relevant parameters of expert experience p and a and calculated, P= 0.888, P= 0.884, P= 0.818, and the final score Q = 86.3.

C. Comparison of R & D capabilities between units

Judging from the scoring of the original software development capabilities of the two units, both units are secondary units of software maturity, and the scoring unit B is slightly better than unit A.

From the perspective of the software development tasks undertaken by this model, among the software undertaken by Unit A, there are relatively few low-level errors that are easier to detect after delivery, but there are some requirements and problems at the design stage that are not in the development process. Check out in time. The number of post-delivery problems in the software undertaken by unit B is relatively large and the level is relatively high, but it is mainly the problems that occur during the coding stage. There are still some problems that are easier to detect after delivery, which were not detected during the development process, indicating that their management capabilities are better (the same as the original software development capability score) but their technical capabilities are slightly weaker.

The above situation is reflected in the final scores of the two units. The score of unit B has changed from the original pure software development capability score to higher than A unit, and has become lower than A unit under the improved comprehensive evaluation method of software development capability. The evaluation method takes into account the actual technical capabilities of each unit and the product quality of the research and development project.

V. SUMMARY AND OTHER RELEVANT WORK

In this paper, based on the existing CMMI evaluation criteria, the comprehensive evaluation method of software development capability of the research sponsoring organization is improved, and the evaluation value of each project software development process undertaken by the research sponsoring organization and the evaluation value of software problems after delivery are also included, which is closer to the actual situation of software development.

At present, the method mainly adopts the subjective weight method, and the expert scoring method is used to determine the selection of each weight. After accumulating more historical data of the project, the algorithm can be modified constantly to adjust the subjective weight to the objective weight, and determine by using the mutual relationship between various indicators or the information provided. In fact, the weight can be obtained through mathematical processing of the original data, so as to obtain more accurate evaluation results.

With the introduction of CMMI 2.0, the existing development capability evaluation criteria used in aerospace software are currently under consideration for corresponding changes. Before the new development capability evaluation standard is released, this method can help the overall organization of aerospace software development to quickly evaluate the actual software development capability of each research organization by combining the existing research capability evaluation data of each research organization and the quality data of the research project actually undertaken by each research organization. Evaluation standards in new development ability, also can consider using the method of this article, will have developed the operation of the software product maintenance and problem feedback into the final software development capability maturity model (CMM), at the same time improve the evaluation process, the daily research data is also included in evaluation of the project, make the new standard of the evaluation results can be more close to the actual work.

REFERENCES

[1] General Armament Department of the People's Liberation Army.

Maturity model of military software development capability: GJB 5000A-2008[S]. 2008:1.

[2] Yu Zhiyong. Thoughts and suggestions for effectively carrying out objective evaluation of software work products[J]. Quality and Reliability, 2019(03): 55-57+62.

[3] Hu Weiqiong. Analysis and improvement of the "two skins" problem in the implementation of GJB 5000A "Military Software Capability Maturity Model" [J]. Quality and Reliability, 2017(03): 40-43.

[4] Pang Haiou, Zhong Xiaojun. Practice and thinking of implementing GJB 5000A "Military Software Development Capability Maturity Model" [J]. Ship Standardization and Quality, 2014(04): 5-9.

[5] Gao Jiaqi. Research on the application of quality assurance method in GJB5000A in aerospace embedded software [D]. Lanzhou University, 2019.

[6] Liu Chang, Hao Jianing. Software quality assurance based on GJB 5000A [J]. Quality and reliability, 2020(01): 58-62.

[7] Tu Buhua, Liang Hong, Zhang Shuming, Tong Weiqi. Multi-system integration practice based on GJB5000A model[J]. Electronic Technology and Software Engineering, 2020(06): 60-64.

[8] Li Qingfei, Zhang Shenhao, Gong Shanshan. GJB5000A Level 2 in scientific research [J]. Digital Communication World, 2019(10): 28-57.

(6)

[9] Wei Jurong. GJB5000A second-level software process improvement activities and practical experience [J]. Electronic Technology and Software Engineering, 2020(05): 46-47.

[10] Maturity model of military software development capability and its application [M]. China Standard Press, Shi Zhu, 2009