Recall from Section 3.6.2 that Design-Based Research has two sets of outcomes: first the practical outcomes in terms of measurements of the effects of the artefact in the trials and deductions in terms of future improvements for the artefact; and secondly new theory derived from the trial data.
Table 20: Inter-Rater Reliability Results
Description Metric
(All cases) Metric
(Active ratings)
Initial Krippendorff’s Alpha 0.67 0.18
Krippendorff’s Alpha after Discussion 0.73 0.46
Chapter 8: Further Trials (Magid 2)
This section explores the first set of outcomes: practical ones, addressing the first research sub-question:
RQ 5.1 To what extent did the groups learn about and adopt different security-enhancing activities as a result of the Developer Security Essentials intervention?
It describes first the resulting list of techniques based on the full analysis; then each of the groups undertaking the Developer Security Essentials intervention, and the results from each34. Each section introduces the organisation and teams involved and discusses the group’s reasons for wanting the intervention. It then describes the intervention as it happened for that group, then provides the results of the analysis of the transcriptions of all of the sessions.
8.6.1 Resulting List of Techniques
Table 21 shows the full list of Techniques derived from the coding. It divides them into categories: Vulnerability Finding, techniques to find specific vulnerabilities in created software; Process Improvements, to create an environment to better support the creation of secure code or reduce the impact of security issues; and Education, to teach participants and stakeholders about the previous techniques. Most of the Techniques were introduced in Chapter 4, but the coders identified five further techniques for security improvement from the discussions and interviews, each used by several of the teams; these are highlighted. They also found that that one previously used Assurance Technique, Incentivisation Session, did not reflect the way participants discussed their secure development improvement; instead participants discussed Further Workshops.
Incentivisation Session was therefore omitted.
The resulting full set of assurance techniques is described below:
Automated Penetration
Testing Using an automated tool to look for common, easily exploited, vulnerabilities in a website or web service.
Automated Static
Analysis Using automated tools to look for common vulnerabilities in source or binary code.
Configuration Review Choosing secure components and frameworks, and keeping them up to date
Code Review Scheduled meetings or pair programming to analyse code for security defects
Penetration Testing Having a Security Specialist look for vulnerabilities accessible via the web.
34 In other chapters the participant descriptions and results were separated. Given the large number of groups here, it is easier for the reader if they are kept together.
Table 21: Assurance Techniques
Threat Assessment Design-level analysis of possible attackers, motives, and vulnerability locations (a.k.a. Threat Modelling).
Product Negotiation Empowering product management to make security decisions.
Contingency Plan The advance creation of a plan to handle security incidents.
Security Champion Having a development team member, not usually a security expert, with a particular interest in security. They act as the go-to person for security issues within the development team.
Standardisation The creation of standard security configurations, ways of working, or ‘Secure Development Lifecycles’, plus auditing processes to validate these.
On-the-job Training Mentoring or informal workshops, used regularly with the development team
Further Workshops Using the entire Developer Security Essentials package with other teams, or for the same team in a new project or with new members.
8.6.2 Group D Results
Group D are a project team within a university, funded by a government grant to promote business innovation. Group D’s role is to developing proof of concept products and support applications, according to the requirements of specific businesses. The group had been in existence for only a year at the time of the intervention. Although members worked on several different projects at a time, all worked together as a team.
Unsurprisingly, the culture bears some resemblance to a start-up, but its situation within a university, the grant-based funding and perhaps the fact that the staff were on fixed-length contracts gives a more formal and disciplined feel to their ways of working.
Though academic organisations are often Dionysian, within the team the culture appeared Athenian, with a respect for technical expertise and management experience.
The group were aware of the importance of software security but had little practical knowledge; hence their request for the workshops. There were two projects considered in the D Threat Assessment workshop. First was a data collection app to package grassland information for remote analysis by an agricultural specialist, their client. Second was an app to support managing personal and medical information for patients of a medical services company. Unusually, the Threat Assessment ideation session found almost no concerns with the first project, so in that session the team moved on to discuss the second project.
Figure 44 shows the levels of adoption of the various kinds of assurance technique (see Section 3.6.4) before and after the intervention.
Since the group’s role was creating proof of concept applications, they realised that the actual implementation of security features and of security hardening was not important for them; what was important was that they supported their clients in implementing secure solutions when the clients came to implement the deployable applications. Thus the two techniques most important to team D were Threat Assessment (determining the security requirements) and Product Negotiation (conveying the analysis and importance of security to their clients).
Group D’s projects tended to be relatively small, typically a few developer months, and so the three months of the intervention provided time for several new projects starting;
Chapter 8: Further Trials (Magid 2)
the ‘Established’ rating for those two assurance techniques reflects that both were incorporated into the new projects and into the group’s process for further projects.
8.6.3 Group E Results
Group E work for a government department delivering software for sensitive government applications. The group was set up in the past couple of years, and many of its developers are less experienced than the average for the industry. The session leader E1, by contrast, was a highly experienced security specialist, now working on software development security. His role was to provide security support to a number of groups like group E, so his contribution to each project was relatively limited. E1 was keen to find ways to help the development teams he supported with security, and he requested the Developer Security Essentials Workshops accordingly.
Confidentiality restrictions limited what could be discussed in the presence of researchers.
E1 chose this particular group because their product was a form of data store, and though it was designed to handle sensitive data, its functionality and design could be discussed without compromising confidentiality. Indeed, to avoid the need for security clearance for the researcher, the workshop took place in an insecure ‘public’ space: a meeting room in a serviced office building.
The group was divided into two teams, working on distinct aspects of the product; 15 people attended the workshop. The group culture was professional and fairly formal.
Since the group’s raison d’être was their ability to deliver a security product, E1’s role gave him considerable authority. The group at the time had nobody in the product management role. The culture appeared Apollonian, with a clear sense of hierarchy.
The workshops showed a particular style of facilitation for the Threat Assessment session best described as ‘directive’. The conversations were dominated by the facilitator and other participants tended to listen respectfully. This was a big contrast to the previous team, D, where everyone participated relatively equally.
Figure 45 shows the impact of the intervention. The largest difference between Before and After was in Product Negotiation: prior to the interventions, both development teams had regarded security as an absolute; every security feature and requirement had to be
Figure 44: Group D Intervention Results
Automated Pen Testing
implemented without question. That attitude remained after the intervention; what changed was that the teams realised that some security aspects were needed earlier than others. The teams were using an agile development process, and now negotiated with their clients about the order of delivery of security features relative to other features.
E1 was pleased with the results of the intervention, and planned further uses of the workshop with other teams.
8.6.4 Group F Results
Group F were from a small surveying company that had created a specialist mapping product used by a large number of clients. The company’s role is to carry out the mapping, store the maps in a Geographical Information System (GIS) database, and provide the database entry and reporting facilities for clients to store their data associated with the maps. They provide this through a web-based front end, and use servers hosted on their office site.
The company and product are long-established, and the company remains dominated by the Managing Director, who did not participate in the workshops. Group F have a manager, F1, with development skills, and a single full-time developer, F2. The product manager, F3, also took part in the workshops. The culture was fairly relaxed and informal, though it was clear that the managing director was involved in all significant decisions (a Zeusian culture). The team have been working together for several years, and seemed easy with each other.
Several years before, one of the developers involved with creating the web-based version of their product had had a doctorate in software security, and the underlying design and implementation reflected an understanding of security issues. However, none of the current team had any knowledge of software security. At the time of the workshops they anticipated a future sale to a government organisation, with possible contractual requirements for security.
Figure 46 shows the impact of the workshops. F1 took trouble to extract benefit from the discussions, creating a table of the threats identified from the Threat Assessment session
Figure 45: Group E Intervention Results
Automated Pen Testing Automated Static Analysis
Configuration Review Code Review
Penetration Testing Threat Assessment
Product Negotiation Contingency Plan
Security Champion Standardisation
On-the-job Training Further Workshops
Before After Established
Using Planned Aware No mention
Chapter 8: Further Trials (Magid 2)
and returning to it in follow-up sessions. The Threat Assessment session identified several issues with the manual creation process of new customer accounts.
The government sale did occur, and the team used that Threat Assessment as a basis for Product Negotiation concerning which mitigations were now required for the new government customer and therefore got implemented; this Using of Product Negotiation is reflected in the figure.
8.6.5 Group G Results
Group G are from a web applications developer employing some 50-100 people, mainly development and product management staff. They produce a variety of applications for clients, ranging from stand-alone websites to front-ends supporting complex back end systems owned by the client. The two leads were G1 the group CTO and G2 the Development Manager.
The group came from several teams and included two product managers and a couple of Quality Assurance (QA) staff. Most of the participants were relatively experienced and correspondingly senior in the organisation. The CEO (not a participant) was still responsible for strategy and clearly involved in most aspects of the company: a Zeusian culture.
G1 and G2 had a particular reason for wanting the Developer Security Essentials Workshops. They were each personally expert in software security, and had a good knowledge of what was required in the websites their teams developed. However, increasingly they were finding that the effort their teams needed to ensure security was not reflected in the financial rewards the company received; neither was normally involved in pre-sale negotiations, and they had had recurring problems with being expected to provide costly security enhancements ‘for free’. They wanted to address this problem.
Figure 46: Group F Intervention Results
Automated Pen Testing Automated Static Analysis
Configuration Review Code Review
Penetration Testing Threat Assessment
Product Negotiation Contingency Plan
Security Champion Standardisation
On-the-job Training Further Workshops
Before After Established
Using Planned Aware No mention
Figure 47 shows the impact of G’s workshops. Given the problem being addressed, the Threat Assessment and Threat Sales workshops did not go into much technical detail.35 Instead, they looked at ways to reflect security requirements through the pre-sales and contracts processes. They came out with an impressively simple way to discuss security cost-benefit with a client:
The problem here is that it is a difficult conversation to have once a client’s signed off on a project… It should it should be done at the business
development stage. People in Biz Dev should be coming to you .. and saying
“Right, I've got this client on board. Here we go. There’s three packages:
our gold level hosting package, our bronze and our silver. Do you think they fit into any of these categories?” So, then we can go back to them as an additional add-on as we do with other little bits like maintenance … and just say “we've got these three packages and we reckon you fall into the bronze package” or “you fall into the silver package. It will be an extra blah blah blah, but we also do blah as standard and then these are the additional extras that you can buy…” [G Product Manager, Threat Sales Workshop]
This idea of Gold, Silver and Bronze packages was taken up enthusiastically. G1 later expanded it to five options to include other aspects of security; at the time of the exit interviews he was creating a guidance sheet for the sales and marketing team.
8.6.6 Group H Results
Group H work for a small company selling a range of Internet of Things (IoT) devices and their associated infrastructure. Though the company has been established providing research services for some 30 years, it is only recently that they have moved to be a product-based company, so the company has some of the attributes of a successful start-up.
35 Nevertheless, Automated Pen Testing and Static Analysis were introduced as a direct result of the workshops.
Figure 47: Group G Intervention Results
Automated Pen Testing Automated Static Analysis
Configuration Review Code Review
Penetration Testing Threat Assessment
Product Negotiation Contingency Plan
Security Champion Standardisation
On-the-job Training Further Workshops
Before After Established
Using Planned Aware No mention
Chapter 8: Further Trials (Magid 2)
The group all work as a team; product management is effectively shared between H1 the CEO, who decides strategy (so another Zeusian culture), and H3 the CTO, who makes the day-to-day decisions on what functionality to implement. The workshop lead H2 was a developer with a particular interest in security.
The group justifiably considered themselves good at software security; the service they provide has a strong security component and security is part of their Unique Selling Proposition (USP) against Asian competitors capable of providing cheaper hardware.
They viewed the workshops as a team-building style exercise to provide a measure of reward and entertainment to the participants.
Figure 48 shows the impact of the intervention for group H. They planned further training and possible workshops.
8.6.7 Group I Results
Group I are a well-established company providing the infrastructure required to allow a commodity trading market to function. The market uses internationally-agreed standards for the encryption and signing of trading messages, and this company provides systems with information on prices, bids and offers, and supports trades between users and with other participants in the market. It is a mature and respected company in its field, and the participants showed a relaxed confident attitude to development. Though they have considerable internal expertise in security, much of their security requirements have been satisfied in the past through perimeter security; only relatively trusted people have had access to the software. Now they have the possibility of delivering cloud-based services as well, where perimeter security will be less relevant.
Of the two workshop leaders, I1 has a largely managerial role; I2 has the most technical (and security) experience of the development teams in the company. Their purpose in running the workshops was to help developers and product management engage more effectively with security. Current versions of the product exist in a firewalled ‘virtual community’ where all systems owned by the trading companies are connected by VPN and are inaccessible from the wider Internet; future versions may be cloud-based.
The culture of the teams appeared relaxed and professional; there was no reference at any time to senior management, suggesting an Athenian culture.
Figure 48: Group H Intervention Results
Automated Pen Testing
The group engaged enthusiastically with the workshops. Indeed, the day after the workshops at which the researchers were present they decided that the outcomes from the Threat Assessment and Security Sales were insufficient; they re-run both workshops to gain a more complete idea of the threats and possible impact on customers. They also ran the full suite of workshops separately with further development teams.
Figure 49 shows the impact of the workshops. As discussed above, they found value in the Threat Assessment and Product Negotiation, and both were incorporated for the future, along with further training and workshops.
8.6.8 Group J Results
Group J were from a well-established large company providing web interfaces for retailers. The particular group involved had the responsibility of creating tools and services to support deployment, and comprised about a dozen staff. The company has development sites in Eastern Europe, and a policy of moving capable staff around, so the group had more staff originating from outside the UK than any of the others involved in this project. Unlike any of the other organisations except Group E, this company has a separate security function employing professional security experts; it also has a policy of assigning the security experts directly to the teams, and of training developers as security experts. The culture of the group was serious, with little banter among the teams except in the breaks, and an awareness of hierarchy in terms of both management and technical experience. That suggests an Athenian or Apollonian culture; there seemed emphasis on management control, so this text assigns the culture as Apollonian.
The two assigned leaders for the workshops were both security specialists: J1 was the security specialist assigned to the group; J2 was more recently a developer and is in
The two assigned leaders for the workshops were both security specialists: J1 was the security specialist assigned to the group; J2 was more recently a developer and is in