The following example illustrates a quality assessment in an open-source project hosted by the ASF. This project is developed in Java, and uses Jenkins- CI, JIRA, GitHub, and SonarQube as CI tools. A quality assessment in this project has such stakeholders: Shane the software engineer, Tim the test engi- neer, and Luci the lead engineer. At the beginning of every assessment, the stakeholders independently inspect the evolution of the project from different perspectives. For example, Shane starts with an inspection of newly intro- duced coding violations. He is primarily interested in violations that have a severity of medium, high, or blocker. In a further step, Shane digs deeper into
6.2 Motivating Example 171
selected violations. He first has a look onto the underlying commit in the VCS. In most of the cases, the violations are caused by a sub-optimal implementation chosen by the source code contributor. However, in some cases it is necessary that Shane looks up an issue description in the issue tracker to understand the purpose of a source code change. It may happen that a contributor is aware of the introduced violations, but the importance of, for example, fixing a bug was higher than the quality of the system. Tim on the other hand, uses a slightly different approach to assess the quality of a software system. As a test engineer, he is primarily interested in the evolution of the project’s test cover- age and the test cases. He starts with an inspection of issues that have been recently marked as resolved in the issue tracker. For each issue, he looks up the associated commits in the VCS to check if the contributing stakeholder has provided or updated the test code as well. Stakeholders provide, for example, fixes for an issue which is not covered by the existing test code. Many times, contributors forget to update the according test code, which prevents from an early detection of a reoccurring of the same issue. Luci follows yet another assessment approach. She typically starts with a look on the build status and the number of opened, reopened and resolved issues in the issue tracker. One important measure for her is the ratio of resolved to (re-)opened issues. She has to act in case of a relatively high number of opened, and especially, reopened issues compared to the number of closed issues. For example, a high number of reopened issues can be an indicator for a sub-optimal quality of the contributed source code. In such a case, Luci has to investigate the reasons. Potential reasons can be the complexity of the affected code or shortcomings of programming skills.
In the following, we illustrated the information gathering process without and with an integrated approach, such as SQA-Cockpit.
6.2.1
Scenario with state-of-the-art CI environment
In this scenario, we show how the information gathering of the described stakeholders takes place in a state-of-the-art CI environment.
Figure 6.1: Information gathering in a modern CI environment
Luci, Shane, and Tim need to access multiple CI tools (e.g., Jenkins-CI, JIRA, GitHub, and SonarQube) in order to get the data they need for their quality assessment. Figure 6.1 depicts the dashboards provided by the different CI tools. The highlighted (orange) elements represent data that is queried by a stakeholder in certain order (orange arrows). The selected elements vary, depending on the needs of a stakeholder. For example, Shane is less interested in data from the issue tracker than Tim or Luci. In short, different stakehold- ers need different data. However, modern CI tools provided rather static dashboard, which cannot satisfy the information needs of all stakeholders. In such an environment, it is extremely time-consuming to seek data for a quality assessment. First of all, a user has to switch between the different tools. Second, the artifacts (e.g., issues, commits) stored in the different CI tools are not linked. For example, Shane wants to lookup the quality of a commit. In
6.2 Motivating Example 173
modern CI tools, Shane has to manually seek the according commit in the quality management platform.
6.2.2
Scenario with SQA-Cockpit
In this scenario, we show how the information gathering of the described scenario takes place with an integrated approach, such as SQA-Cockpit.
Figure 6.2: Information gathering with SQA-Cockpit
Figure 6.2 depicts the proposed dashboard provided by the SQA-Cockpit approach as a mashup. Again, the orange elements indicate data that is relevant for a certain stakeholder. The elements are connected to each other to additionally foster the information seeking. Every view is automatically composed based on the activity profile of a stakeholder. Such a profile can be derived, for example, from the event logs of CI tools. The major differences of the SQA-Cockpit approach compared to state-of-the-art CI environments are (i) the integration of the data into one screen, and (ii) a personalized view based on a stakeholders activity profile.
of people in a software project. In modern CI tools, the presentation of data is not optimized for any of these groups. The views in these tools usually contain parts of data from various areas, which cannot satisfy all the needs of the mentioned groups. This increases the information seeking effort for each individual stakeholder every time an SQA is conducted. To overcome these limitations, we propose the SQA-Cockpit approach, which provides following mechanisms to enable an optimal workflow during an SQA:
• integration of data that is scattered across CI tools.
• tailoring of integrated CI data based on the needs of stakeholders. • presentation of tailored data for a faster information seeking process.