This section returns to the research question:
RQ 4 What security outcomes did the ‘Developer Security Essentials’ package have, and what aspects contributed most to those outcomes?
It explores how the answers to RQ 4.1, RQ 4.2 and RQ 4.3 in the previous sections provide a basis for the next stage of the research.
7.7.1 Estimate of Impact
While it is early to know the long-term impact, it is unlikely that that Company A’s teams will remove their component security evaluation and upgrade process, nor lose the benefit of the permissions improvements. Whether the remaining security enhancements discussed in the first follow-up session get incorporated is, correctly, a business rather than a technical decision; it is promising that the architects are asking the right questions (about risk evaluation) to support business in making that decision.
Similarly, we know from feedback (Section 7.5.2) that Company B’s project leaders now include security in their project planning, instructions for students, and general company management; and their students are more aware of security issues.
Some key learning points, especially that security is more than avoiding the OWASP Top Ten in your code, will remain with all the participants from Companies A and B.
In the case of Company C, the most we can say is that the interventions contributed to an existing trend of increased security awareness.
7.7.2 How the Interventions Worked
The impact of the interventions differed between teams: not only in the nature of the security issues addressed; but also, in the teams’ responses to the interventions and in how they benefitted. Team A introduced better development processes; Team B gained an awareness of several specific security improvements and the need for Threat Assessment; and for Team C the interventions prompted better communication and
understanding. Sections 7.4.1 to 7.4.3 explored the differences in the ways the teams responded to these interventions.
The successes identified came through the developers’ choices. As the Expert Survey concluded (see Section 4.14), to be effective a program needs to motivate rather than simply direct the teams involved. And, indeed, the interventions were successful to the extent that they could change the developers’ thinking, understanding and motivation.
The interventions involved, predominantly, conversations between developers, allowing them to learn mainly from each other, and to motivate themselves rather than respond to outside pressures. Table 15 and Table 14 suggest that this was an effective motivation and learning approach.
Indeed, by contrast to the results of earlier studies based on interventions using Penetration Testing as a motivator [134,172], in this study Table 15 shows that for both Teams A and B the long term impact after one year was still important.
7.7.2.1 Interaction with Other Stakeholders
The analysis of blockers and motivators identified during the interventions (section 7.6) found that a large majority of both involved interactions with either the business function or other aspects of the organisation. It follows to improve development security it will help to work explicitly on these interactions.
Of the key assurance techniques identified by the expert survey (Section 4.4), two are related to such interactions: Product Negotiation, and to a lesser extent, Threat Assessment. We conclude that exploring ways to enhance these two techniques by addressing blockers and encouraging motivators has the potential to deliver further improvements.
7.7.2.2 Learning Points for Developers
Table 17 highlights three learning points for software developers from the above discussions: the effectiveness of team activities, key assurance techniques, and the importance of organisational issues.
7.7.3 Impact of the Active Developer Model
Several aspects of the interventions suggested the effectiveness of the Active Developer Model, as follows.
Company A discovered security and privacy threats and acted as ‘customers’ for security enhancements independently of their company’s security specialists (Section 7.4.1), only becoming demotivated when they were disempowered from carrying out the mitigations they suggested (Section 7.5.1).
Company B lacked security specialists to provide an alternative to the Active Developer Model; yet participants B1 and B2 adopted security principles enthusiastically (Section 7.4.2), and the discovery of B4’s aptitude for software security thinking even led him to a career in the area (Section 7.5.2).
Though the impact of Developer Security Essentials in Company C was limited, the fact that their software security practices were effective was itself a vindication of the Active Developer Model; since their relationship with the security specialists in the company were dysfunctional (Section 7.6.1), the success of the software security can only be attributed to being driven by the developers.
Chapter 7: Package Trials (Magid)
7.7.4 Future Work on Assurance Techniques
The importance suggested by Section 4.4 of a small number of assurance techniques provides an incentive for further research on those techniques. While Automated Static Analysis and Penetration Testing have received a good deal of research attention, participant comments, especially Blockers and Motivators, suggest areas for inquiry for others:
Threat Assessment Participants requested example expert assessments for different domains and types of software, to act as a basis for their own assessments.
Configuration Review
Several Blockers suggest a need for improvements to tools and to vulnerability databases to support more fine-grained component analysis.
Code Review Traditional line-by-line code review may not be optimal for security issues: one participant, for example, described instead asking developers to show in their code how they addressed specific security issues. There is a need for experimentation investigating the merits of different approaches.
Table 17: Advice for Software Development Teams Apply Team
Activities to Security
All the workshops derive their effectiveness more from discussions between participants than from any information provided by the intervener, and as Section shows the nature of these discussions was different for each team. So, the success of these interventions can be attributed to the team nature of the activities, and on the participants bringing their own unique range of expertise and knowledge to them. Whether or not a given team uses the specific workshops described here, we conclude that there is benefit in regarding software security as a team, as much as an individual developer, process.
Focus on Key Assurance Techniques
In an example of the Pareto Principle, that 80% of the benefit often derives from 20% of the input [42], the results of this chapter show that introducing three assurance techniques that are within the scope of most development teams, out of out of twenty in use by industry, are together capable of delivering a large impact. We conclude that teams will benefit from concentrating first on these techniques, namely Threat Assessment,
Automated Static Analysis, and Configuration Review.
Address
Organizational Issues
As Figure 41 shows, some 40% of mentions of issues by
participants, both of Motivators supporting security improvement, and Blockers discouraging it, are ascribable to organizational issues. Whilst this finding will not surprise any security
professional, it emphasizes the need to regard the promotion of software development security as a systemic, rather than purely a development team, matter.
Product Negotiation Participants requested methods to express specific security improvements as organisation benefits; and ways to identify the probability of different security breaches. From Section 7.7.2.1, we also conclude a need also to find ways to identify and address blockers and motivators during the process.
Incentivisation Session
Alternatives to the Agile Security Game include Capture-the-flag games, Penetration Test-based sessions, and case study-based training. While this work proves the success of the first, research would be valuable to compare other approaches in differing situations.
On-the-job Training The interventions provide only a one-off security improvement.
Games such as the ‘covert saboteur’ in Team A offer opportunities for developers to develop their skills further.
However, as we saw, the effectiveness of such approaches depends on personal aspects and team dynamics (Section 6.2.1).
Research is needed to provide low-time-cost ways to continue the team security improvement process.
7.7.5 Viral Distribution?
For the interventions to have a longer-term impact, we wanted participants with the intervention work not only to improve their own projects, but also to understand how to take the interventions and use them themselves in other contexts.
As discussed in Section 7.4.4, the results were generally disappointing, though the leaders of Teams A and B did show understanding of Threat Assessment and Product Negotiation.
7.7.6 Next Steps
We identified three key areas for future work on the interventions. First, the participant-driven nature of the workshops meant that not every technique was covered for every team: Team B did not discuss Automated Static Analysis, Penetration Testing, nor Code Review, for example. One participant suggested a checklist or take-away sheet after the first day’s presentation:
I think maybe some sort of tick sheet in terms of “have you got these things in place?” to take away, that might be a good addition (A1).
Second, for the program to scale to a wider number of participant teams, we needed intervention leaders who appreciated the aims of the different sessions, such as the importance of an Incentivisation Session to achieve team motivation. Yet Table 14 suggests that this knowledge was not successfully conveyed to many of the participants.
Nor did any participants learn how to use the Developer Security Essentials intervention themselves. We identified a need to motivate and teach some participants to lead the intervention.
Third, an important area for improvement was in Product Negotiation: both in methods to express specific security improvements as organisation benefits (Section 7.7.4) and to gain the time and ‘mind-space’ to use effectively the security learning the team had gained (Section 7.6.3). We considered that an extended workshop might be a suitable way to support developers in doing this.
Chapter 7: Package Trials (Magid)
In terms of the methodology used for the trials, we identified two improvements to make:
1. There were too few companies involved for us to deduce much about the effects of different contexts on the effectiveness of the intervention. Involving more would give more scope for such deductions.
2. The changes resulting from the interventions were discovered at haphazard; the analysis approach derived example improvements but did not necessarily provide consistency between companies. For further work, we wanted a more consistent analysis of the understanding gained and techniques implemented.
These improvements were incorporated into the next project.
8 Further Trials (Magid 2)
This chapter describes a further project, Magid 2, involving a second round of trials with an improved Developer Security Essentials package.
This project reflected an emphasis on making the intervention potentially scalable for use by large numbers of organisations. Its research question was therefore:
RQ 5 Which aspects of the ‘Developer Security Essentials’ intervention are effective at improving security when used independently by teams from a variety of cultures and different types of organisation, and why?
The three most salient changes from the previous project are:
Independent Facilitators: We trained one or two facilitators from each organisation, and they then managed the intervention. The purpose of this was to transfer the ability and confidence to run the workshops to the facilitators, potentially allowing them to run further workshops without the support of the research team.
Variety of Cultures and Types of Organisation: In this project we carried out the interventions with many more groups: eight as compared with three in the previous project. In addition to the aspects of organisation, such as size and security capability, we also identified organisation culture types for each group.
Support for Product Negotiation: A further ‘Threat Sales’ workshop was added after the ‘Threat Assessment’ workshop, to help developers in representing security and privacy issues as business factors.
Section 8.4 gives more details.