In response to:
RQ 2.1 What approach to interaction with software development teams leads to the best results in encouraging secure development?
the Expert Survey established the ‘Active Developer Model’ theory, that developers must drive the introduction of security improvements.
Chapter 4: Expert Survey
Encouragingly, we note that the ‘Motivating Jenny’ project (Section 2.2.6) came to a similar conclusion of the need to raise developers’ security awareness, from their later ethnographic work.
In response to:
RQ 2.2 What specific intervention techniques do specialists consider most cost-effective in helping developers improve security and privacy in their code?
the Expert Survey established six key Assurance Techniques as the most recommended by the experts for adoption by developers: Threat Assessment, Stakeholder Negotiation, Configuration Review, Source Code Review, Automated Static Analysis, and Penetration Testing. Plus, it established two further techniques used by the experts to encourage developers to drive security: Incentivisation Session and On-the-Job Training.
4.13.1 Choosing Assurance Techniques
Table 8 summarises the Assurance Techniques discussed in Sections 4.5 to 4.10. It highlights the ‘dialectic’ nature of each of the techniques (see Section 4.1), by showing
‘friendly counterparties’ involved with each one; such a counterparty may be a tool.
Interestingly, end users and operations staff were not mentioned in the context of any of the Assurance Techniques.
We observe that this list of key Assurance Techniques is nevertheless a small subset of the 20 techniques identified by Such et al. [164]: the explicit focus that the surveyed experts placed on them is therefore important.
Based on Table 8 and the expert cost estimates from Such et al., we can further characterise the assurance techniques in terms of two important practical considerations:
their cost and their need for security expertise. Figure 16 shows this characterisation, using Such et al.’s modal estimate of cost for each technique, and assigning ‘cheap’ to Product Negotiation, since it is incorporated into activities already carried out by a development team. In this and the following figures, the Assurance Techniques are coloured according to their type: blue for process techniques; orange for vulnerability finding techniques.
Table 8: Security Assurance Techniques and Participants
Members of Development Team Stakeholders, e.g. Product Managers Automated Code Analysis Tools Security Experts Penetration Test Experts Other Development Teams End Users and Operations Threat Assessment
Stakeholder Negotiation*
Configuration Review Source Code Review Automated Static Analysis
Penetration Test
Based on this, we deduce that the most promising Assurance Techniques are Threat Assessment, Product Negotiation, Configuration Review and Automated Static Analysis;
and that teams with greater resources will benefit from adding Source Code Review and Penetration Testing.
Interestingly, the choice of which tools to use for Configuration Review and Automatic Static Analysis, and indeed for the other techniques, was rarely discussed by the experts.
We conclude that developers are already highly skilled at choosing between competing tools and methods, and do not need explicit support for choosing security assurance tools.
4.13.2 Incorporating the Assurance Techniques into the Development Cycle
The ordering of the assurance techniques in Section 4.4 is roughly chronological within a development cycle. Figure 17 illustrates how they might be incorporated into an iterative cycle. Threat Assessment and Product Negotiation affect what functionality is
Figure 17: Assurance Techniques in the Software Development Cycle
Threat Assessment
Configuration Review Automated Static Analysis
Source Code Review Penetration
Testing
Product Negotiation
Plan
Test Code
Release
Figure 16: Interventions in Terms of Cost and Specialist Requirements
Threat Assessment Configuration
Review
Automated Static Analysis
Source Code Review
Penetration Testing
Product Negotiation
Cheap
Requires Security Experts Team Only
Expensive
Chapter 4: Expert Survey
produced, so apply to the planning element; Configuration Review and Automated Static Analysis can be automated into a product build; Source Code Review needs a candidate complete implementation, so is typically done at the release stage; and Penetration Testing applies to an installed system, so comes in the test phase.
4.13.3 Sensitisation, Support and Affordability
The Active Developer model implies that for an intervention to be effective the development team must actively accept it, and therefore it must qualify in terms of Sensitisation, Support and Affordability (Section 4.3.2). Figure 18 shows how each of the identified Assurance Techniques qualifies, showing their effectiveness in providing Sensitisation or Support on the horizontal axis, and their Affordability on the vertical one.
4.13.4 Research Validity
What measure of certainty can we offer for the theory, the Active Developer Model, and for the assertion that the six Assurance Techniques and two further techniques highlighted in the research do represent best practice in Developer Centred Security interventions?
Considering first Conclusion Validity, do the research data justify the conclusions?
Grounded Theory’s rigorous process of line-by-line coding, categorisation, and sorting generates a theory that does reflect the interview data. The use of extensive quotations ensures that this can be at least partially checked.
In terms of Construct Validity, does the Active Developer theory represent real-world practice? Grounded Theory handles this primarily in terms of ‘theoretical saturation’, reached when new interviews do not add substantially to the theory (Section 3.3.3). A dozen interviews are often sufficient [77]. We believe we have reached saturation as regards the Active Developer concept and the list of Assurance Techniques, in that further interviews would be unlikely to modify the concept, or the list; obviously they would produce more detail for the descriptions. There is also a risk of bias in the choice of interviewees, and of questions. Examples might include selecting participants using only one form or intervention or one interaction approach; or asking only about certain aspects of their interventions. We addressed this bias with interviewees from a wide range of industry roles, and completely open questions.
Figure 18: Sensitisation, Support and Affordability
Threat Assessment
Configuration Review
Automated Static Analysis Source Code
Review Penetration
Testing
Product Negotiation
Cheap
Support Sensitization
Expensive
In terms of External Validity, can the results be generalised to a wider scope? Grounded Theory’s conclusions are always limited to the scope studied [33]. Specifically, the scope in this case covered commercial companies, predominantly in the UK (Section 4.2), and was limited to the views of security experts rather than software developers.