The VDDS has been characterised as a complex yet, tractable problem, inhabited by entities that interact over time (Malone et al., 2013; Rahimi et al., 2013). Therefore, the first phase of constructing theory, and subsequent model of the VDDS is an exploration of the human and social aspects of the system. To do this Thematic Analysis was employed as the technique of choice within the qualitative stage of the mixed methods approach. The mixed method approach provides differing lenses to observe the system and triangulate between, thus providing further confidence in the analysis (Creswell, 2014, p.207). Once collected data was arranged and categorised into data sets, items and extracts and finally coded with any observed patterns being noted as a potential theme. Once noted, codes were analysed and drawn out from the data items, leading to the formation of a potential themes, relationships and interactions.
4.1.1 Structuring the VDDS
Data collected about the VDDS is both unstructured and heterogeneous in nature as it is predominantly derived from entities who do not conform to a specific standard (Katal et al., 2013). Hence, once data was collected organisation and structure was required to make sense of the data and provide
a framework to undertake analysis. The Thematic Analysis approach provides a method that provides structure to data by allocating, or coding, data items with descriptive tags, highlighting relationships between data items. This approach allows for the identification of sub-themes and themes to emerge from within the data (Braun and Wilkinson, 2003; Braun et al., 2006).
4.1.1.1 Data Item Descriptions
Social media and blog data items contain rich seams of information whereby vulnerabilities are discussed, debated and interactions between entities are described and defined. Specifically social media platforms are seen to be more technology focused such as Reddit, Hackernews and 4chan are used for this purpose (4chan, 2017; Reddit, 2017; Ycombinator, 2017). Open social media platforms such as these are a valuable source of data and to the discourse around vulnerability discovery and disclosure as they provide a historical record, and environment to discuss vulnerabilities. This is alongside the need to discuss current and new tools for discovery.
A further primary source of evidence is email, which has been used for several years. In the context of vulnerability discovery large email archives exists, specifically a well-respected email distribution list known as ‘Full Disclosure’ (Seclists.org, 2002). The full disclosure email distribution list is the place to disclosure security vulnerabilities to the community and wider public. The full disclosure list is a further valuable source of evidence to support or refute hypotheses stated about the discovery process and - importantly - the disclosure process.
Alongside social media, a further medium that is used to communicate between vulnerability researchers and other participants within the vulnerability discovery system is speciality web forums. These web forums can be publicly available for comments, or closed requiring authentication to view posts. Given the nature of the subject matter, it is probable that a lot of discussion around the trading, discovery and production of software vulnerabilities is within these closed fora. Moreover, given the proliferation of anonymization tools such as ToR or L2P a lot of the discussions that occur are unobtainable for analysis. However, open
forums such as Reddit, and other bespoke open forums provide insight into the market participants and their interactions.
A further source of evidence that was used in constructing theory was the usage of interview transcripts. Interviews with vulnerability researchers or those who are involved within the brokering, disclosing or purchasing vulnerabilities is an invaluable source of information about how the system behaves and is structured. The interviews that have been analysed in this section are semi- structured, and were undertaken by third parties, typically conducted by social media outlets, and subsequently published on the web.
A significant number (>1000) of reports and publications have been produced by commercial organisations, which provide opinion and insight on the vulnerability discovery system (Ablon et al., 2014; Frei, 2013a, 2013b; Lewis and Baker, 2013; Miller, 2007; O’Gorman, Gavin McDonald, 2012). Typically reports of this nature are marketing and sales orientated, produced to enhance the image of the company, and are not peer reviewed or academic in nature. Nevertheless, they do provide insight into the discovery and disclosure system as most of the content is provided via researchers who have links into the community or are vulnerability researchers themselves. Furthermore, the introduction of commercial third parties such as vulnerability brokers provide first hand insight into the VDDS. Therefore, the usage of commercial reports as a secondary source of information is valid, with special attention given to the credibility of the source and any biases present.
4.1.2 Overview of Collected Qualitative Datasets
Within this research the total data corpus consists of seven distinct types of qualitative and quantitative data. For example, within the social media dataset there are in total 364 data items, 1413 data extracts, from 1998 unique actors, complemented with 2,746 emails published by the full disclosure email archive. For the purposes of this research data items are defined as discrete social media posts (including comments), commercial reports, interviews, blogs, emails or news articles. All data items were exhaustively read, and re-read as to gain the most accurate meaning and context from the data.
The initial step within the analysis phase was to first choose which of the identified data sources to use, whilst being pragmatic on how much data can be collected and processed. Yin (1994, p.17) advocates collecting data from the widest range of sources, yet being practical, thus providing confidence in the validity of the collected data corpus and analysis undertaken. By embracing this concept of pragmatic wideness, the collected data was acquired from as wide and extensive range of sources, whilst being realistic on the availability of data and the time available to collect it. An overview of the data sources is provided in Table 9 below.
Data Source Description and Bias
1.Blog posts from security community Vulnerability discovery is covered heavily by both security commentators and vulnerability researchers themselves. As such a large volume of data is available around the actors and relationship that are involved within discovery system.
Typical Data Item Description
Blog posts are generally considered a new phenomenon and are a product of the web 2.0 paradigm. They are published via personal or corporate mechanisms, and therefore can be in most cases considered a primary source of evidence, unless they are re-posting opinion from other sources. The blogs that have been selected are from known vulnerability research organisations or individuals. Where credibility cannot be independently confirmed, this is nonetheless considered as researchers may not be affiliated to a commercial organisation, as the blog poster may have considerable technical knowledge and experiences. Potential bias may include both marketing and sales pressures, feuding with rivals, posting for notoriety and untruthful posts. Validation of blog posts is difficult, unless peer review comments are post within the blog, this can provide a level of validation as to the authority of the author.
2.Open source web forums and news groups
Experiences around vulnerability discovery, and disclosure are regularly discussed on popular online webforums such as Reddit.com or 4chan.com. In addition specific hacker forums have been included within the evidence.
Typical Data Item Description
Webforums and their predecessor newsgroups are major sources of information about software vulnerability. They are generally used by technically literate users, and have a large user base. The popularity of newsgroups has declined over the past decade. Due large user base, the newer webforums can considered to be a primary source of information. The credibility of webforum posters is again variable, as posters may express an opinion rather than actually undertaking or being involved within a vulnerability discovery. Similarly the bias of posters can be heavily civil libertarian in nature. However, this type of post can be a good source of evidence.
3.Interviews performed by third parties Interviews performed, and posted in blogs and webpages. Typical Data Item Description
Third party interviews are Secondary sources of evidence and data. They are conducted as the interviewee has been validated as a person who is directly involved in vulnerability discovery, and has demonstrated their credibility.
These interview are unstructured as they
4.Email archives Archives of vulnerability discovery and disclosure are regularly posted to a open source distribution list known as ‘full-disclosure mailing list’ found at seclists.org. This is considered the defacto forum to disclose vulnerabilities.
Typical Data Item Description
Full disclosure email list is again a primary source of data both from a qualitative and quantitative basis. The email distribution list has thousands of subscribers, which allows security researchers to disclose to a large public distributions any vulnerabilities that they have discovered. Typically it is used as a quasi-official channel to disclose a vulnerability if all other avenues of resolution have been exhausted. The other way in which the distribution list is utilised is to ‘go-pubic’ without any prior discussion with software vendors or coordination organisations.
5.Commercially produced vulnerability reports
Vulnerability discovery and research is primarily undertaken via commercial organisations.
Typical Data Item Description
Due to the nature of cyber security and the monetarisation of vulnerability discovery a significant proportion of research that is undertaken is by commercial organisations. This research is generally undertaken as a marketing exercise and published to create hype or awareness of a current issue or trend. Ultimately this type of research is based upon real world observations, however must be considered in the context of the organisation that it is published by.
6Third party structured interviews Several structured interviews have been used that were sourced directly from (Radianti, 2010b). Furthermore these interviews are primary sources as the interviewee is a vulnerability discovery and disclosure actor.
Typical Data Item Description
Interviews that were collected are structured in natures, and consist of email questions and responses between Radanti and anonymous respondents. Each data item is in the form of email, and was provided by Radianti during March 2015, but cited in studies from 2008 – 2010.
7.Third party Semi-structured interviews
Several interview transcripts were collected. These took the form of magazine articles, blog posts or videos. All interview are scripted, and are considered to be popular in nature.
Typical Data Item Description
Interviews with vulnerabilities researchers and brokers are undertaken by popular bloggers or specialist new outlets. These articles provide insight in to how the vulnerability discovery and disclosure system operates from different actor perspectives. Generally, these articles are between 200-500 words and consist of experiences within the system.