SN Applied Sciences (2021) 3:18 | https://doi.org/10.1007/s42452-020-04015-z
A time‑variant fault detection software reliability model
Kamlesh Kumar Raghuvanshi1 · Arun Agarwal1 · Khushboo Jain2 · V. B. Singh3 Received: 14 September 2020 / Accepted: 21 December 2020 / Published online: 7 January 2021 © The Author(s) 2020 OPEN
Abstract
In this work, we propose a time-variant software reliability model (SRM)which considers the fault detection and the highest number of faults in software. The time-variant genetic algorithm process is implemented for the assessment of the SRM parameters. The proposed model works upon a non-homogeneous Poisson process (NHPP) and incorporates fault dependent detection and software failure intensity and the un-removed error in the software. We had considered programmers proficiency, software complexity, organization hierarchy, and perfect debugging as the determining factors for SRM. The dataset collected from 74 software projects was experimented with to establish and validate the proposed software reliability model’s better fit. Data is collected over a period, which is initiated with the start of the project and is continuously monitored until its completion. Several parameters are analyzed, and a collection of 115 attributes are given with 11 different time frames in terms of product and process characteristics. A total of 383 persons were involved in software design, where the issue count total is 255. The proposed time-variant fault detection SRM is implemented in Jira and is also compared with the existing reliability model presented in the literature. It is observed that the proposed fault detection SRM works better in terms of different parameters like mean square error (MSE), root mean square error (RMSE), and r-squared (R2).
• The work is carried out, ensuring time-varying fault detection, which is measured by considering response count, coding and non-coding deliverables, and the number of bugs in the software.
• We considered the programmer’s proficiency, software complexity, organization hierarchy, and perfect
debug-ging as the determining factors for presenting the soft-ware reliability model.
• The proposed Software reliability model shows improvement over existing algorithms as the residual errors are reduced, and prediction accuracy is high in terms of cumulative fault detection.
Keywords Fault detection · Mean square error (MSE) · Non-homogeneous poisson process NHPP · Root mean square error (RMSE) · Software reliability model (SRM) · R-square · Time variant
1 Introduction
Software design is one of the prime areas of research for developers in this emerging technology era in both hardware and software. Software design comprises a
few well-defined phases, including fault detection, bug fixing, and on-time reliable delivery. The inherent limi-tation of software development is a maintenance cost, which may be substantially high compared to the total development cost. In case of failure, the software returns
nothing but all the cost is directly converted to penalty. Another significant issue is that software development has a few quantifiable measures for accounting. Most of the well-defined software metrics result in categorical data that may be utilized to assess software quality.
Programming complexity and completeness are criti-cal estimations to describe the quality and decide a few questions concerning testing. It includes questions like when to quit testing, what is more, to test, which mod-ule has high coupling, the bug occurrence pattern. The objective is to release software as early as possible with-out identifying bugs in the system, and all stated targets are achieved [1]. Several programming driven models have been given in the literature to decide the release time, which may depend upon a non-homogeneous Poisson process (NHPP) [2], or consider proper investiga-tion for targets defined some uses defect amplificainvestiga-tion model. The fault reduction factor(FRF) given by Musa [3] gives the fraction of the number of faults reduced to the number of faults that occurred, and the execu-tion time is related to the amount of FRF. To assess the quality of software and measure its efficiency before release is one of the prime objectives of software reliabil-ity modeling. Reliabilreliabil-ity metrics are majorly dependent upon how to use allocated system resources for testing. Several other deterministic factors for estimating soft-ware reliability include programmer efficiency, interde-pendency between software modules, learning process, fault occurrence probability, and debugging time lag. Another major constraint for software release estimation is cost. Most of the software estimates move around cost. Delay in any of the predefined set of software develop-ment processes leads to an increase in cost, which is not required and may lead to software failure.
Concerning open source development, software development has its issues because it adopts a simple mechanism "release early and often release" [4]. This mechanism requires fast development and release with minimum desirable functionalities that may be extended as and when required. This software comes in small bun-dles whose functionalities may be expanded or replaced easily with each update. The development of open-source software follows either feature-driven release or duration specific release. Release management in open source is a complex process dependent upon users’ and developers’ procedures. The empirical study of open-source software may extract several qualitative features. This study’s few attributes are the organization’s back-ground developing the software, the process adopted for software development, inter-organizational arrange-ments, and software tools. Still, release management is considered as an under-researched area in the case of open-source software.
Software reliability estimation is essential because it helps in resource utilization, cost management, effec-tive time management, and fault prediction. The primary research objective is to develop a reliability estimation model that utilizes bug detection and prediction of bug occurrence, which follows a few sub-processes, including initial screening of bugs, pre-processing, sampling, and the regression model, and the correlation between actual and predicted bugs. The "Least square estimation" and "Maximum likelihood estimation" techniques are widely used for non-linear regression. Software failures are con-sidered stochastic, resulting in errors being customarily distributed, and the collected sample set is large.
The motivations of this paper are described as follows. Firstly, some works [9, 10, 13, 15] have integrated the com-plexity of environments or a single factor in the creation of the model, but with random application environments they do not represent a generalised SRM. Secondly, envi-ronmental variables were not clearly specified, such as the amount of programming effort, the number of team mem-bers, the organisation of programmers, human nature, the testing environment, meeting hours, software complex-ity, design methodology. We also proposed a generalised multiple-environmental-factors SRM with multiple envi-ronmental factors in view of certain critical impacts.
The remainder of this paper is structured into parts. In Sect. 2, we discuss work carried out in the past on the estimation of software reliability. A brief analysis is carried out in this section, highlighting the primary techniques proposed in different articles presented before. Section 3 discusses assumptions and proposed software reliability. The description of the proposed algorithm and the data set description are carried out in Sect. 4. In Sect. 5, a com-parison with similar algorithms is carried out to analyze the proposed model’s impact based upon several perfor-mance metrics. Finally, Sect. 6 discussion on the conclu-sions is given.
2 Related work
SN Applied Sciences (2021) 3:18 https://doi.org/10.1007/s42452-020-04015-z uncertainty is assessed using varying resource allocation.
It was suggested that most of the existing models consider time, cost, or quality as primary factors for determining reliability. They devised a solution to implement a trade-off among these factors. Defect removal and bug identifica-tion are included to provide robustness in the proposed algorithm.
The authors of [2] researched "blast furnace smelting of Magnitogorsk Iron and Steel Works." Testing is identified as a critical approach to software success. With the integra-tion of existing and new modules, unit testing is key to achieving success. They worked upon the tools and tech-niques used during regression testing in case of changes done after installation. The proposed solution results in a low cost of maintenance, thereby increasing the deliv-ered product’s reliability and scalability. Software reliabil-ity study is carried out in [8] in a different way, focusing on the "power law" software testing function. They high-lighted faults as the main concern for the testing process. Open-source software is selected to validate the proposed approach with the introduction of multigenerational faults. The software release time problem is identified, and a solution about the same is presented using the cost of software testing. Experiments are carried out using soft-ware failure data, and results are compared with Goel-Oku-moto and S-shaped model [9, 10]. The work given in [2] focuses on modularity. They used different mathematical equations such as Fokker-Plank and Petri net for reliability study in open-source software. The presented approach is a comparative analysis between testing and design strategies adopted to make a reliable software product. A network technology-based routing software technique is implemented in this paper.
The prediction model is built and analyzed for imposing security in software in [11]. The paper presents a detailed analysis of vulnerabilities and potential risks in software development. Machine learning is applied for estimat-ing time to fix an error. Another prediction technique is applied in [12] to provide a reliable software model that reduces faults in designing software. The paper identifies fault reduction factor (FRF) as a primary source of reliability growth. The authors suggest key factors behind FRF and develop a function that estimates the time-dependent value of FRF. By using the approach mentioned above, the authors prove that it enhances accuracy and provides reliability. The authors of [13] identify various factors that affect software reliability, such as software complexity, organizational policies, capabilities, and software design proficiency. The paper suggests a genetic algorithm-based approach for determining faults in software design. Vari-ous performance metrics are compared using the pro-posed model, and it outperforms several existing algo-rithms in terms of risk and prediction accuracy.
The authors of [14] suggest that removing risk at any stage of software development may result in new risks. The solution may be software release, including some low probable risks that might not negatively affect soft-ware performance. These leftover risks may be further taken into consideration during the maintenance phase. The authors carried out a comparative analysis for these leftover risks that may be dealt with before or after release. The book published by the authors of [15] provides a case study of various release modes of software in open source systems. They suggested that most of the developers pre-fer time-based software release in comparison to feature driven software release.
3 Proposed approach
3.1 AssumptionsA general assumption for most of the existing software reliability models is that fault is independent and, on detection is can be flawlessly removed. Nevertheless, it is not always correct because of several factors, including programmer skills and expertise, software complexity, organizational hierarchy. A software time-variant reliabil-ity model (SRM) based non-homogeneous Poisson process (NHPP) is presented in this paper. The Time variant fault detection SRM deliberates the detection of fault/failure reliant and inadequate fault/failure elimination process and considers the number of faults/failures based upon their extreme count. The working of a SRM in flowchart is illustrated in Fig. 1.
The following are the assumptions for this proposed Software reliability time-variant model based on NHPP:
• The software failure practice is an NHPP.
• Fault/failure detection is a progressive curve based upon learning-spectacle.
• Fault cannot be flawlessly upon detection.
• The proposed approach is based on fault/failure detec-tion.
• The debugging process can also announce new faults in the software. There can be a process that leads to imperfect debugging, but the number of faults limiting the software reliability is restricted to L.
• Each fault is independent and correspondingly likely to cause a failure during the testing of the program.
• The software failure intensity 𝜆(t) is enlightened about the proportion of errors that were successfully removed in the software product.
• The software contains N faults at the initial level, which are assumed to be unknown but fixed.
• The time-intervals among incidences of faults are also independent of each other.
3.2 Software reliability model representation Let ti be the time between (i − 1)thandi th failures in the
program. The total number of software failures at any time instance ti is be notated as N(ti) and depends on the non-homogeneous Poisson process (NHPP).
The estimated number of software failures until time ti is called the mean value function (MVF) and denoted by m(ti) . φ is notated as a proportional constant, which defines anyone’s fault makes to the whole program. At the i failure interval, the program failure rate is given by 𝜆(ti
)
𝜆(ti) , and L defines the maximum number of faults a soft-ware can contain.
Let (ti) , 𝛽(ti) , and 𝛾(ti) defines the fault content function at time instance ti , the rate at which software fault can be detected per time unit, and the rate of un-removed error per unit time.
The Software reliability function by time ti given a mis-sion time ti−1 is given by R(ti−1|ti).
Numerous reliability models prove that 𝜆(t) is directly proportional to the number of faults left out in the soft-ware in most NHPP models. Furthermore, the proposed models ensure that software results in independent faults that may be rectified entirely upon detection. Conse-quently, we may model generalized NHPP software MVF, which considers time-variant fault/failure detection rate is obtained by solving the following proposed differential equation (with the initial condition m(t0
) ! = 0)
In this work, we have considered time-variant fault detection rate 𝛽(ti) as follow:
The resolution of the expected number of software faults detected by time ti , i.e., mean value function (MVF) of the differential equation in Eq. (1) considering the time-variant fault detection rate 𝛽(ti) in Eq. (2), can be attained as follows:
To make it simple and more understandable, we named the time-variant fault-detection reliability model. From Eq. (3),m(ti)= L
1 + 𝜑 when the value of ti = 0 , and m(ti)=L when ti = ∞.
4 Performance evaluation
4.1 System platform:The proposed time-variant fault detection software reli-ability model is implemented in Jira [2] as it supports web services likes Representational State Transfer (REST), Simple Object Access Protocol (SOAP), and Extensible Markup Language- Remote Procedure Call (XML-RPC) for the integration of various sources control programs such as Clear case, Mercurial, Concurrent Versions System (CVS), Perforce, Subversion, and Team Foundation Server. Jira (1) 𝜕m(ti ) 𝜕ti =m ( ti)× 𝛽(ti)∗ ( 1 − m ( ti ) L ) −m(ti)× 𝛾(ti) (2) 𝛽(ti)= ( 𝛽 1 + 𝛼(ti)×e−𝛽t) (3) m(ti)= L 1 + 𝜑 × 𝛽(ti)+1 𝛽(ti)+e𝛽t
SN Applied Sciences (2021) 3:18 https://doi.org/10.1007/s42452-020-04015-z designed and developed in Java and procedures the Pico
inversion of control container, Apache OFBiz entity engine, and WebWork.
Four packages are commonly used with JIRA, i.e., Core, Software, Service Desk, and Ops. The product set JIRA is a software project tool specially designed for project man-agement and bug tracking. It allows Atlassian to devel-ops the management of products and services using agile methodology and the complete package. The given tool provides flexibility in the execution where the software or a product code can be executed from anywhere. A vari-ety of languages can be supported by this tool, including Japanese, English, Spanish, German, and French. In this context, where multiple languages can be supported by this tool, it becomes compelling and can be considered a multi-lingual tool. This tool’s scope is not limited to bug tracking; it also includes various testing methodologies, incident management, and a complete set of software design and workflow management tasks. The software provides an additional set of customized tools that add to its flexibility and usability. JIRA supports MySQL, Post-greSQL Oracle, and SQL Server in the backend. The han-dling of issues is made easy by JIRA’s dashboard, as it has many features and functionalities. JIRA allows tracking the progress of the project from time to time.
The dataset given in Table 1 contains software attrib-utes data, which is obtained for 74 software projects. Data is collected over a period, which is initiated with the start of the project and is continuously monitored until its completion. Several parameters are analyzed, and a collection of 115 attributes are given with 11 different
time frames in terms of product and process character-istics. A total of 383 persons were involved in software design, where the issue count total is 255. This dataset can be used to predict software release. The attribute collection is focused on qualitative and quantitative fea-tures during software development. This dataset gives deliverables about individual and team activity. The overall quality of the software design, requirements, and specifications are monitored continuously. It con-tains the analysis of bugs encountered, countermeasures taken, meeting statistics, time efficiency, team perfor-mance, leadership impact, gender-sensitive information, programmer proficiency. Figure 2 shows the dataset Description over Range of different attributes for the given time frame and Fig. 3 illustrates the Pair-plot of the dataset concerning team member response count, non-coding deliverables hours, coding deliverables hours, and total issue count.
4.2 Performance metrics
Various performance metrics are measured to assess the proposed software reliability time-variant fault detection model’s effectiveness and accuracy. The proposed tech-nique has been evaluated by comparing with Infection-S-shaped, Delayed Infection-S-shaped, and Goel–Okumoto (G–O) model. The performance metrics which needs to be evaluated to check the validity and performance of the proposed software reliability time-variant fault detection technique are:
Table 1 List of Attributes selected from the dataset
S.No Attribute Name Description
1 Team member count Total number of members of the team
2 Team member response count Number of active/proficient members
3 Meeting hours total Total meeting time
4 Inperson meeting hours total Individual meeting time
5 NonCoding deliverable hours total Planning hours
6 Coding deliverable hours total Useful hours
7 Help hours total Outsourcing hours
8 Lead admin hoursResponseCount Number of hours responses received from team lead in case of an issue
9 Leadadmin hours total Above multiplied by team lead proficiency
10 Average responses A student is a team member / it gives the total response
11 Commit count Includes commit for successful completion of part of the project
12 Commitmessage length total Equivalent LOC
13 Average commit The ratio of total commit by individual/team
14 Issue count Total number of bugs
15 Ontime issue count Before release issue count
16 Lateissue count After release issue count
4.2.1 Mean Square Error (MSE)
It gives an estimate of deviation, measured by finding out the square of the error term. The outcome of MSE is utilized as an indicator of accuracy evaluated for predicting any unseen faults/failures. The above metric is evaluated by the formula given in Eq. 4, which uses the errors’ squares. An error term is measured by finding out the symmetric dif-ference between the actual and the estimated value. MSE is used as a default metric for performance evaluation in most regression algorithms implemented in MATLAB, R, or python.
The total number of observations and the number of parameters for evaluation is given by N and n, respectively. The term m(ti) is the MVF by the software reliability model, (4) MSE = 1 N − n N ∑ i=1 ( m(ti)−yi)2
which estimates the number of faults expected, detected by time ti , and yi is the observation value.
4.2.2 Root Mean Square Error (RMSE)
It gives an estimate of deviation, measured by finding out the square root of the MSE term. The normalized values of RMSE are obtained using prediction model values and actual values. RMSE is utilized for finding out residuals and its standard deviation. The graph depicting residuals gives the deviation of the regression line from actual data points. It also gives the variation and shows the scattered nature of residuals concerning each data point’s best fit.
(5) MSE = √ √ √ √∑N i=1 m(ti)−yi)2 N − n
Fig. 2 Dataset Description–Range of different attributes for the given time frame a Distribution of Total issue count with each time frame b Distribution of nonCodingDeliverableHours with each time
SN Applied Sciences (2021) 3:18 https://doi.org/10.1007/s42452-020-04015-z
The total number of observations and the number of parameters for evaluation is given by N and n, respec-tively. The term m(ti) is the MVF by the software reli-ability model, which estimates the expected numbers of fault detected by time ti, and yi is the observation value.
4.2.3 R‑squared (R2)
The goodness-of-fit measure is frequently used for linear regression models. This parameter gives the statistic that indicates the variance ratio calculated with the dependent
and independent variables’ help. R-squared measures’ strength and effectiveness lie with the value that models the relationship between the proposed indicator and the dependent variable on a convenient scale of 0 to 100.R2is
the normalized version of MSE for reporting as it is a sim-ple metric and the loss function we are minimizing when we solve the standard equation.
m(ti)The total number of observations and the number of parameters for evaluation is given by N and n, respec-tively. The term m(ti) is the MVF by the software reliability model, which estimates the expected numbers of fault detected by time ti and yi is the observation value and
−
yi is the mean of the observed value.
5 Results and discussion
We apply the time-variant software reliability model to obtain the proposed time-variant fault detection model’s parameter estimates with Infection-S-shaped, Delayed S-shaped, and Goel–Okumoto (G–O) models. These models are implemented in JIRA as already discussed in Sect. 4.1. The validity, effectiveness, and goodness of fit can be expressed by evaluating MSE, RMSE, and R-squared criteria as described above in Sect. 4.2.
The proposed time-variant fault detection model ful-fills the condition 0 < m(t0
)
≤y1,at t = 0, by the number
of faults present initially, and y1 is the total instances of
observed failures t = 1 . The model initializes m(t0
) with an integer value. The integer value is selected based on a desirable constraint that a software tester uses for pre-analysis. The above procedure is performed to eliminate errors present in the software, whose impact is negligi-ble in the complete software’s performance. All those errors are trivial and must be ignored before the software moves to the formal testing phase. These insignificant errors are considered to appear due to unintentional humane mistakes or some other issues related to software (6) R2 = 1 − ∑N i=1 � m�ti�−yi�2 ∑N i=1 � m�ti�−yi�2
configuration and management. The objective is to con-sider the total number of faults having an extreme impact on the software. These errors are present in the modeling phase itself and may be removed as early as possible. All the trivial errors may be assumed to be a part of these total numbers of faults dealt during a complete set of modeling and may be ignored at initial phase fault detection using the proposed scheme.
The results of the parameters estimated by model execution and performance parameters (MSE, RMSE, and R-squared) corresponding to each of the compared algo-rithms are listed in Table 2. The values given below are the outcome of execution for the proposed model and other existing models. The system test data represents Infection-S-shaped. Therefore, it cannot seamlessly capture the char-acteristics of Delayed S-shaped and Goel–Okumoto (G–O) models.
For the system test data, the estimated parameters are
𝛼 = 13.45 , 𝛽 = 0.18,and 𝛾 = 0.34. As observed in Table 2,
the MSE, RMSE, and R-squared values for the proposed software reliability model are 0.51, 0.714, and 0.86, respec-tively, which are the minimum among all four models presented here. The estimated values for the Inflection S-shaped model are 𝛼 = 28.34 , 𝛽 = 0.24 and 𝛾 = 0.63. How-ever, the value of MSE, RMSE, and R-squared are 1.33.1.532 and 0.53, respectively, relatively low than the Delayed S-shaped and Goel–Okumoto (G–O) models. Though both these models cannot seamlessly capture the characteris-tics of the proposed software reliability model. The esti-mated parameter for Delayed S-shaped is 𝛼 = 47.64 and
𝛽 = 0.26 , which is slightly higher than that of the Inflection
S-shaped model. The MSE, RMSE, and R-squared are 2.07, 1.438, and 0.32, respectively. The RMSE and R-square value of the Delayed S-shaped model are relatively less than the Inflection S-shaped model. The estimated parameter value for Goel–Okumoto (G–O) is 𝛼 = 70.67 and 𝛽 = 0.35 . The performance parameter values of MSE, RMSE, and R-squared are 6.65, 2.578, and 0.09, respectively. Based on the above analysis, the conclusion may be drawn that the proposed software reliability time-variant fault detection model performs better (compared to other existing mod-els) and can be stated as the best fit for the given system.
Table 2 Comparison and parameter estimation of proposed software reliability and other existing technique
S. no Software Reliability Models MSE RMSE R2 Estimated parameter
𝛼 𝛽 𝛾
1 Proposed Model 0.51 0.714 0.86 13.45 0.18 0.34
2 Inflection S-shaped 1.33 1.532 0.53 28.34 0.24 0.63
3 Delayed S-shaped 2.07 1.438 0.32 47.64 0.26 –
SN Applied Sciences (2021) 3:18 https://doi.org/10.1007/s42452-020-04015-z
Figure 4 illustrates the evaluation of cumulative fail-ures (actual)with cumulative failfail-ures (predicted). Figure 5 depicts the comparison of MSE, RMSE, and R-squared parameters by four software reliability models.
6 Conclusion and future direction
In this paper, we introduced the Time-variant fault detec-tion Software Reliability Model that includes different software attributes for model building and compares the
same with several well-known algorithms upon different performance metrics. The proposed model is compared with Infection-S-Shaped, Delayed-S-Shaped, and Goel-Okumoto model in terms of cumulative failures, mean square error, and root mean square error parameters. The proposed model works upon a non-homogeneous Pois-son process and incorporates fault dependent detection and software failure intensity and un-removed error in the software. We considered the programmer’s proficiency, software complexity, organization hierarchy, and perfect debugging as the determining factors for presenting the software reliability model. This novel work is carried out, ensuring time-varying fault detection, measured by con-sidering response count, coding and non-coding delivera-bles, and bugs in the software. The findings show that the new model proposed has substantially better goodness-of-fit and predicts better value than the other compared models. The proposed model shows improvement over existing algorithms as the residual errors are reduced, and prediction accuracy is high in cumulative fault detection.
The proposed approach’s future work may include defining exact time boundaries for software release and assessing factors that need to be addressed for the soft-ware’s multi-release. Future work will involve broader vali-dation of this conclusion based on recent data sets.
Fig. 4 Comparison of predicted failures (cumulative)
Funding This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Complaince with ethical standards
Conflict of interest The authors declare that they have no conflict of interest.
Open Access This article is licensed under a Creative Commons Attri-bution 4.0 International License, which permits use, sharing, adap-tation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creat iveco mmons .org/licen ses/by/4.0/.
References
1. Majumdar R, Shrivastava AK, Kapur PK et al (2017) Release and testing stop time of software using multi-attribute utility theory. Life Cycle ReliabSafEng 6:47–55. https ://doi.org/10.1007/s4187 2-017-0005-9
2. V. Nagaraju (2018) Software Reliability Assessment: Modeling and Algorithms. 2018 IEEE International Symposium on Soft-ware Reliability Engineering Workshops (ISSREW), Memphis, TN, pp. 166-169, DOI: https ://doi.org/10.1109/ISSRE W.2018.000-4. 3. Pavlov N, Iliev A, Rahnev A, Kyurkchiev N (2019) Some
Trans-muted Software Reliability Models. Journal of Mathematical Sciences and Modelling 2(1):64–70. https ://doi.org/10.33187 / jmsm.43427 7
4. Malik A, Ahmed J, Qadir J, Ilyas MU (2017) A measurement study of open source SDN layers in OpenStack under network pertur-bation. Computer and Communication. 102:139–149
5. Erto P, Giorgio M, Lepore A (2020) The Generalized Inflection S-Shaped Software Reliability Growth Model. IEEE Trans Reliab 69(1):228–244
6. GholamrezaNorouzi MH (2015) Siamak Noori, and Morteza Bagherpour, “Developing a Mathematical Model for Schedul-ing and DeterminSchedul-ing Success Probability of Research Projects Considering Complex-Fuzzy Networks,.” Journal of Applied Mathematics, Hindawi 2015:1–16
7. Pietrantuono R, Potena P, Pecchia A, Rodriguez D, Russo S, Fernández-Sanz L (2018) Multiobjective Testing Resource Allo-cation Under Uncertainty. IEEE Trans Evol Comput 22(3):347–362 8. Li F, Yi ZL (2016) A new software reliability growth model:
Multigeneration faults and a power-law testing-effort func-tion. Mathematical Problems in Engineering. https ://doi. org/10.1155/2016/92760 93
9. Yamada S, Ohtera H, Narihisa H (1987) A testing-effort depend-ent software reliability model and its application. Microelectron Reliab 27(3):507–522
10. Yamada S, Ohba M, Osaki S (1984) S-shaped software reli-ability growth models and their applications. IEEE Trans Reliab 33(4):289–292
11. Ben Othmane L, Chehrazi G, Bodden E et al (2017) Time for addressing software security issues: prediction models and impacting factors. Data Sci. Eng. 2:107–124. https ://doi. org/10.1007/s4101 9-016-0019-8
12. Hsu C-J, Huang C-Y (2011) Jun-Ru Chang" Enhancing software reliability modeling and prediction through the introduction of time-variable fault reduction factor,". Applied Mathematical Modelling, Elsevier 35(1):506–521
13. Zhu M, Pham H (2016) A software reliability model with time-dependent fault detection and fault removal. Vietnam J Comput Sci 3:71–79. https ://doi.org/10.1007/s4059 5-016-0058-0
14. A. Anand, S. Das, O. 2016 SinghModeling software failures and reliability growth based on pre & post-release testing. 2016 5th International Conference on Reliability, Infocom Technolo-gies and Optimization (Trends and Future Directions) (ICRITO), Noida, pp. 139–144.
15. Jose Teixeira, Release Early, Release Often, and Release on Time. An Empirical Case Study of Release Management Open Source Systems: Towards Robust Practices, 2017, Volume 496, ISBN: 978–3–319–57734–0.
