The work in this thesis initially investigated the optimization of Staffing and Scheduling a Software Project (SSSP) problem. From the literature, we found that benchmarking and evaluating the approaches proposed to solve this problem has only been done in the context of a comprehensive survey and a systematic literature review. Therefore, a complexity classification with datasets corresponds to this classification were created to contribute to the SSSP literature. In addition, a process combined with a set of quality and performance measures were proposed. As these approaches are proposed to solve an industrial problems, nine well-known approaches were under investigation of their quality and suitability to software industry using the benchmarking process and the datasets. The insight gained from the findings of investigating these approaches has contributed in formalizing four team allocation methods into optimization problems. In addition, an empirical evaluation of Project Managers (PMs) performance from software industry was performed. Part of this evaluation was to assess the suitability of the SSSP approaches by validating the datasets used to benchmark and evaluate them.
The answers for the first and second questions outlined in Section 1.4 of the first aim of this research can be concluded as follow. With no prior knowledge about the SSSP approaches, is there an automated approach that reliably solves the SSSP problem. Many optimized approaches have been presented throughout the previous three decades as in [15, 22], and it is important to capture their potential capability and capacity for different management complexity problems. Work has been carried out for this thesis in exploring the capacity and capability for nine SSSP approaches.
The findings according to the measurements used for the approaches’ outcomes using MAAPE for accuracy, and CT score for performance show that for project time problem, some of the SSSP approaches vary in their outcomes of Estimated Project Time (EPT) and Computation Time (CT), and the SSSP approaches in [22] and [94] can outperform the others as they are capable of providing solutions close to the optimal one with reasonable amount of computation time. While the SSSP approaches are differing from each other, it was important to observe which of the
allocation methods adopted by the approaches is capable of providing better solutions. In this matter, formalization of four team allocation methods into optimization problems was proposed, and advanced experiments were performed, using uniform stochastic operations and optimization settings of GA, to capture which of the methods are best at handling complexity level of effort, dependency, skills, and productivity. The finding from these experiments is that Static Teams with Queueing Simulator for allocation (STQS) for scenario one and two, and Dynamic Team with Binary Participation (DTBP) methods, for scenario three, four, and five were good at enabling the approaches to heuristically search for near optimal solutions.
The second aim of this research was to answer whether these approaches outperform the expert intuition in solving SSSP problem. Accordingly, an industrial setting study was performed. PMs were the subjects in this study for experiments and interviews. Four of the PMs work for a large financial organization, one with a large international software development organization, and the last one with Start-up Company. Our study subjects have between 6 to 30 years of experience. This study encompassed mixed-methods to capture different quantitative and qualitative data important in providing comprehensive knowledge about the study subjects and industry practice.
The experimental part was performed to capture how PMs from the industry would perform for each scenario defined in the benchmark. As these scenarios are based on the dataset created for this thesis, the optimal project time was defined for each. There was two subjects that their solutions were similar to the best SSSP approaches. The key differences identified between those subjects and the others might have contributed to their high quality solutions. These keys are the allocation method they have used while solving the scenarios, as by dynamic teams, and distinguished years of experience they have, for 26 and 30 years. On the other hand, the accuracy of the solution provided by the SSSP approaches has a negative relationship with the level of scenario’s complexity. For instance, as the level of scenario’s increases from one to four, the results were less and less accurate as from 99.9% to 79.3%. In addition, with variability of resources’
productivity implemented in scenario five, none of the approaches were able to handle this level.
The third aim of this research, identified in Section 1.4, was to find whether the SSSP approaches reflect the real PMs’ needs. For this aim, interviews with PMs were conducted in the second phase carried out for the industrial settings evaluation study. This study was also designed to explore based on the subjects’ experience, what aspects and attributes in software projects are important to be considered by a PM for SSSP problem. Seven categories were the focal points to discuss with the PMs in the interview. These categories are the organization level they represent, their experience, project attributes, allocation methods, teams, scheduling, and management objectives they believed are important to resource allocation optimization. Based on the results from the
interviews and the experiments carried out for this thesis, almost all the project attributes are found important.
Three main attributes discussed by the PMs in the interviews and found important for a PM to consider for software project optimization are the precedence relationship between project tasks, resource’s skillset, and resource’s productivity. These attributes are represented in the study partially by the challenging scenario four and fully represented by scenario five. It is noteworthy that scenario five was not included within the industrial settings study for two reasons. The main reason is that although the experiments were intended to challenge the PMs’ capabilities, these experiments should also respect their time constraints too. The second reason is that the intended experiments and interviews are carefully planned to capture some targeted issues that can provide glimpse on PMs’ practices, so it is hard to bring all the scenarios, especially the fifth one, to be solved by the subjects while a similar can be found in the fourth.
The experiments carried out on the four team allocation methods complies with the findings from the second phase of the industrial settings evaluation, which implies that the PM should look at the different resource assignment and team allocation methods that (s)he can use. In this regard, there was a consensus between the subjects on the dynamic shifting of resources between teams and tasks especially when a skill is required. Dynamic assignment method has been evaluated with different simulation of methods, and it was found from the demonstration of the simulations presented in Chapter 5 that this method can outperform the others and solve the time problem more efficiently to more advance scenarios.
Project cost, on the other hand, is clearly an important part that should be included within the optimization problem of resource allocation. However, as searching for the most minimized cost of resource allocation alternative requires the identification of resources’ salary, the resources who are possessing the same skillset and doing the same job should have the same salary. Therefore, for an optimization problem that consider skills as one of the inputs, all the alternatives could have the same influence on project cost. The datasets used in this thesis moreover, have no information that can support this part due to the sensitivity of this information to the data contributors, which can show their key success in resourcing and payment structure. In addition, many SSSP approaches optimize either for time span or combine multi-objectives problem that includes project time within. Consequently, uniformalising the comparison problem for a shared objective –as this was the first intention of this thesis- of time span optimization problem is the only solution. Therefore, the main focus of the experiments carried out for this thesis was on project time span minimization using the described project in the datasets. Conforming to this conclusion the study subjects have stated in reaction to when the solutions of the scenarios was
revealed to them in the second phase that the important objective is time to how it can be managed after the agreement on quality and cost outcomes is established. To this end, it important to mention that four out of six subjects in the exit interviews stated that time is the ultimate objective in software projects.
7.1 Overall Findings and Lessons Learned
The overall findings throughout the thesis work are listed in the following bullet points.
Different methods can be used to allocate human resource in software projects adopted by the PMs, and yet no specific method can overcome project time optimization problem.
Teams are the solid base for software development, however, the assignment of team members can be represented by a rigid or dynamic formation and mainly less experience PMs tend to use the rigid one to avoid any conflicts.
The more experienced the PM is, the more (s)he tends to provide better and faster solution to the problem.
Complexity levels corresponding to PMs’ projects problem are found by challenging scenario 2, and 4.
Corresponding to the discussion made in [130], the work carried out for this thesis found that large organizations prefer a hybrid approach supported with a traditional development methodology, such as waterfall.
Project time span is important to be managed as software cost and quality are issues that are defined in early project stages and the PM is left to manage the time according to the stakeholder’s agreement.
None of the of SSSP approaches has presented their experiments computation time for a matter of fact and many have failed to report the crucial system capability they used such as CPU, and memory.
In addition, there are some lesson learned as a result from the experiments and industry settings evaluation study are:
It is hard to establish a communication with software industry for data to share.
There is no available dataset that can be used for this filed of research and many are using hypothetical data.
It is hard to bring more than one PM to an interview or experiment due to their work and time availability constraints, and that’s can make it hard for a researcher to standardise the experiment interpretations and meanings exactly the same to all subjects. That was
the reason for having exit interviews after the experiments, so to make sure that they have understood and solve the scenarios to the best they can do and as we expect.
Solutions to labour cost are mainly developed by the optimization approaches as in [93, 136] over time window and activity timing concepts. And this what has stopped us from including more approaches for SSSP.
7.2 Limitations and Future Work
The overall work carried out for this thesis has shed the light on some areas where improvements can be made. One of the improvements that needs to be done on the benchmarking side of SSSP approaches concerns the limitation of the datasets used in this thesis. These datasets present single problem of a small project with limited variables and information to a single organization, which can make it harder to conclude their applicability for different organizations and project scenarios. Moreover, there is a lack of representative datasets that includes the important parts and project attributes for SSSP optimization and none is freely available. Accordingly, gathering and collecting data from larger software projects is an important part in this research field to be made.
Different open source optimization toolboxes have been used by SSSP optimization papers, and yet no quality and performance information can be found on their outcomes compared against each other. Therefore, work has to be done on investigating and benchmarking these toolboxes especially for accuracy and computation time performance compared to the one used in the work for this thesis.
Moreover, the work carried out for this thesis has included nine approaches for specific problem comparison, however, work should be made to include up-to-date SSSP approaches in the benchmarking study. This work should investigate the novelty of the proposed approach and whether better outcomes are anticipated by the optimized allocation method proposed in that approach. In particular, benchmarking and comparing between the approaches adopting event-based, time-line, and time window scheduler are also important to be established to demonstrate their effectiveness and performance against each other. That includes developing a benchmark dataset for this particular approaches’ type too. These approaches can help to tackle resource availability problem by providing, on the project time-line base, when the possible shortage of resources can happen, and their availability that can be supported by limiting their participation rate percentage.