The second artifact created as part of this research is an in- stantiation of the conceptual model for Data Driven Security Operations. We chose to create another artifact because of sev- eral reasons, the first of which is the fact that we needed a way to evaluate the requirements we had set for the concep- tual model itself. The implementation would complement the semi-structured interviews that we had setup for evaluation of the model. The second reason is that the conceptual model in itself mostly contains theoretical foundations, which could certainly help a Security Operations Centers (SOCs) in establish- ing a more data driven strategy to security operations, but did not offer practical value directly. We thought that making the
SOCthink about the model on the basis of predefined questions
would offer greater insights into the conceptual model, thereby making it more usable in practice.
As listed in Table 5.2we wanted our instantiation to be us- able in practice, which meant we needed to construct a way for
SOCsto interact with the model. TheSOCshould also be able to set goals, compare itself to otherSOCsand itself and the instan- tiation should provide SOCs practical value. We chose to im- plement an assessment consisting of questions that we deemed illustrative for the model. The questions were all created using the well known construct of Likert scales, thereby partly ad- dressing requirement R6. Results from the assessment would then be used to create a radar chart, showing the SOC where
6.2 instantiation 61 they fit within the conceptual model, addressing requirements R8, R9 and R10. The final report created by the instantiation would not only embed the radar chart, but also offer the SOC
some practical considerations for improvement and reaching its goals, which addresses requirements R7and R10.
The next sections describe our design rationale and how we implemented the instantiation, including the motivation for the questions and technical details.
6.2.1 Design Rationale
First we had to determine what the instantiation should encom- pass for aSOCto be usable in practice. We settled fairly quickly on an assessment because this seemed most practical to use in this setting. We also determined that all questions were to become Likert items, because these would allow for basic arith- metic to be used on them and to combine several questions in a single indicator in several ways. All of the Likert items were scaled on a range from 1 to5, inclusive, using different scales, ranging from the worst to the best possible rating. The ques- tions have not been evenly divided over all facets, because of the difference in breadth the facets encompass. The breadth of the facets is also the reason that the questions we devised are broad in nature, but we have take care to embed the most important aspects from each facet in them.
The main goal of the instantiation was to offer the SOC the ability to assess its maturity within the model for Data Driven Security Operations (R8). A radar chart seemed a good fit to realize this, because it is possible to visualize several quantita- tive variables in a single plot. As described before, the assess- ment was constructed from several Likert scales, six in total, each belonging to a single facet and consisting of several Likert items. In our instantiation we calculated the mean for each of the facets as usual, i.e. by summing the scores for each of the questions and dividing by the number of questions for a spe- cific facet. Plotting the scores for each of the facets in a single radar chart would then show the SOChow well it scores within ourData Driven Security Operations model.
We also wanted to include a way to show a SOC where it
falls short and should consider improving (R10). Because SOCs differ in the capabilities they offer, we introduced the setting of a goal level for each of the facets, such that SOCs can set goals for individual facets. Because in our opinion aSOCshould have
some leeway in assessing its score for specific facets, we also introduced a weighing factor. Using this factor a SOC can pri- oritize certain facets over others, resulting in easier or harder to reach goals. The final part of the design of the assessment would gauge theperceived level. This perceived level would help us to determine the validity of the questions posed in the assess- ment: when the perceived level of a facet is close to the score based on the answers, that’s an indication the questions posed in the assessment fit their purpose of assessing aSOC’s maturity
for that facet within the model for Data Driven Security Opera- tions.
6.2.2 Implementation
Our implementation consists of a front end and a back end that together result in a working instantiation for assessing a SOC’s maturity within our conceptual model. We will describe these parts in the next sections.
Assessment
Determining what questions to ask was the most important part of creating the instantiation. Because of the broad nature of both the conceptual model and each of the facets, we had to either include many detailed questions, resulting in a long test, or relatively high level and broad questions, resulting in a shorter test. Because this is the first version of the implementa- tion, we decided to create a shorter test, with questions broad in nature, but still touching upon each of the important aspects of the facets. This way the questions would engage participants to think about and comment on them, increasing the amount of qualitative data to assess. We had already decided to include only Likert items in the assessment because this would allow us to combine several questions into a single measure. Table D.1 lists the five Likert ratings that we applied. All of the questions posed in the assessment are listed in the tables in AppendixD. The last column shows the scale that was used for a specific question corresponding to TableD.1.
All of the questions were implemented in a Google Forms form, the front end of the instantiation. Google Forms was chosen as the front end because it supports all types of ques- tions necessary, especially the Likert scale. In addition to that, Google Forms can be accessed via an Application Program-
6.2 instantiation 63 ming Interface (API), through which the data entries can be ac-
cessed directly, and allows exporting the data to comma-separated value (CSV) format, which can be read in many software pack- ages and is practical for data processing. The elements of the form were constructed in the following order:
• Identifier for participant
• Weights for facets: six times a value from{1,2,3,4,5}
• Questions for Situational Awareness (Table D.2)
• Questions for Threat Intelligence (TableD.3)
• Questions for Detection Methods (Table D.4)
• Questions for Response & Investigation (TableD.5)
• Questions for SOCStaff (TableD.6)
• Questions for SOCInfrastructure (Table D.7)
• Perceived level for each of the facets: six times a value from{1,2,3,4,5}
All of the questions in Appendix Dwere added to the form in the same order as they appear in TablesD.2 toD.7. For each facet two additional questions were added at the end of each part: a mandatory goal level and an optional field for adding notes and remarks.
Report Generator
We implemented a back end in Python for processing the sub- mitted answers. It connects to Google Forms via its API to re- trieve the answers to the questions, but can also read them from aCSVfile. By default it always picks the most recent submission for a certain organizationidentifiersubmitted by the participant. It performs several checks to determine the validity of an entry, after which it starts the actual processing. The capped weighted average score for all of thenquestions belonging to each of the six (i ∈ {1..6}) facets fi having weight wfi is calculated as fol- lows: max(min((−(wfi−3.0)·0.1+1)· 1 nfi nfi ∑ j=1 fij,5.0),1.0) (6.1)
These capped average values are then used together with the goal levels for constructing the radar chart data to be em- bedded in the report.
In addition to showing the radar chart, the report also con- tains a management summary of the conclusions that can be drawn from the radar chart. These are short descriptions of what is visualized in the chart. The report also contains a full description of the six facets, which we included in AppendixB. The final part of the report contains concrete recommen- dations for a SOC based on the answers submitted during the assessment. These recommendations are linked to groupings of the questions, which are indicated by the Roman numerals in Tables D.2toD.7in AppendixD. These groupings are listed per facet: the Roman numeral I in Table D.2 for Situational Awareness does not correspond to the Roman numeral I in Ta- ble D.3 for Threat Intelligence, for example. The recommenda- tions include specific products, literature and advice, resulting from the high-level market analysis and literature research per- formed. By default theSOC will only receive recommendations for the average score of the groupings of questions within the facets that score lower than the goal set by theSOC. The order of appearance of recommendations depends on the absolute dif- ference between the average score of the grouping and the goal level, in order of highest to lowest difference. In the report we stress the fact that we are not affiliated with any product, both proprietary as well as open source, and point out that these serve illustrative purposes only. Many of the concepts behind the solutions that we recommend are important drivers for the Data Driven Security Operations Center. The complete set of recommendations available within the report generator can be found in Appendix E.
The final report is generated as a single, responsive Hy- perText Markup Language (HTML) page and contains the radar chart, management summary, descriptions of the facets forData Driven Security Operationsand recommendations for theSOC.
So far we haven’t discussed how the perceived level that a
SOC submits at the end of the test is used. These values are used to create a different radar chart showing the SOC’s per- ceived level and the capped weighted average score that was calculated. This chart is part of the evaluation of the instantia- tion, which is discussed in Chapter 7.
6.2 instantiation 65
SOC
Infrastructure
SOC
Staff
Response &
Investigation
Detection
Methods
Threat
Intelligence
Situational
Awareness
1
2
3
4
5
Figure 6.3: An example radar chart. The chart shows theSOC’s goal and calculated score in red and black, respectively. The illustration differs from the one in the final report because of a different type of renderer being used.
Data Driven Security Operations Quick Scan
ACME
Based on data supplied on February 1, 2016.
Menu
Summary Data Driven Security Recommendations
Report
Summary
This report describes the results of the Data Driven Security Operations Quick Scan performed by ACME on February 1, 2016. First a list of high level findings, including a graphical output of the assessment, is shown. It is followed by a brief overview of the facets of Data Driven Security. The report concludes with some concrete pointers for improvement.
Detection Methods
Based on the answers submitted during the assessment the target level for Detection Methods and processes around improving these is not reached. Critically consider investing in ways to improve detection, including (improved) heuristics and anomaly detection.
Threat Intelligence
The current goal level for Threat Intelligence usage and understanding has not been reached according to the answers submitted during the assessment. Consider investing in technology, people or processes to improve the usage of Threat Intelligence.
SOC Staffing
Based on the assessment, the target level for SOC Staffing is close to being reached. Hiring more employees is one of the ways to improve at this stage. Also investing in improving your current employees can improve this facet.
SOC Infrastructure
The goal level for the SOC Infrastructure is, based on the answers submitted during the assessment, close to being reached. Consider improving by assessing what SOC functions are not adequately being supported and invest in people and processes to mitigate or remove bottlenecks.
Situational Awareness
The goal level of Situational Awareness has been reached based on the answers submitted during the assessment. This means the organization has a sufficient view on and understanding of what is currently going on in the IT environment in real-time. Specific aspects may fall short however, which may need investments.
Response & Investigation
Your organization has reached the target level for the Reponse & Investigation facet. This means the current technologies, processes and staff adequately handle incident response and digital investigation. When the functions of the SOC change, all of these need to be reassessed to continue supporting the Response & Investigation facet.
Figure 6.4: Example of the management summary part of a report rendered using a web browser. In a browser the radar chart is created dynamically using JavaScript and incor- porates an item legend that appears when hovering over a facet. The colored blocks of text summarize the radar chart. A complete example report can be found on https://hermanslatman.nl/ddsoqs/.