Evaluation of Information Elements in a Cyber Incident Report
Patrik Lif
∗, Stefan Varga
†, Mikael Wedlin
∗, David Lindahl
∗and Mats Persson
∗∗Swedish Defence Research Agency, Department of C4ISR, Link¨oping, Sweden, fi[email protected]
†KTH Royal Institute of Technology, Swedish Armed Forces Headquarters, Stockholm, Sweden, [email protected]
∗Swedish Defence Research Agency, Department of C4ISR, Link¨oping, Sweden, [email protected]
Abstract—An analyst’s main task within cyber security is to monitor, analyze, and decide on appropriate remedies for detected attacks. Analysts should ideally produce incident reports for record keeping and traceability, but also for enabling information sharing.
This paper investigates which information elements that need to be included in an incident report to support incident management in a time-critical and high mental workload situation. Background information on existing reporting templates were drawn from the literature. Five different templates, a data exchange format, as well as two cyber situational awareness frameworks were analyzed. Then a novel reporting template from the military domain was scrutinized during a live cyber defense exercise.
The results show that the information elements in the various existing templates differ, probably due to the differ- ent areas of use. The proposed military template was found to be useful to a high degree, even for incident reporting in the civilian sector. The new template can be improved by introducing additional fields, such as, e.g. descriptions of victims, attackers and assessments of attackers’ motives.
Index Terms—incident reporting, cyber security, cyber de- fense exercise, information elements, CDX.
1. Introduction
Cyber attacks are common and many large companies are attacked daily. In order to respond, organizations need to have cyber analysts who have both theoretical training and practical experience in dealing with attacks. An ex- cellent way to train the practical aspects in a realistic and manageable setting, is by using a so-called cyber range. In this paper we collect data from a cyber defense exercise, CDX, that was held in a cyber range.
The rationale for having cyber situational awareness and maintaining a cyber security stance is to prevent the degradation of one’s cyber network, including deliberate exploitation of it, as such events weaken the owner of the network [1]. In this respect, there are several ideas about what the cyber security field encompasses. It is clear that a range of tasks need to be accomplished to maintain control of a network. This paper focuses on a set of tasks that are carried out by the specific professional role of Cyber Defense Analyst, in the terminology of the U.S. National Initiative for Cybersecurity Education framework, NICE [2].
The analysts’ work in general consists of information collection, as well as both automatic and manual analysis [3]. The main tasks, more specifically, are to monitor (the state of the network), analyze (deviations from the normal), and decide on appropriate remedies for detected attacks and other threats. In other words, analysts need to have a good understanding of their own system, an ability to detect and identify threats, and be able to find and decide on appropriate courses-of-actions that eventually address those threats. Such actions may include measures such as system reconfigurations, software updates, and the disconnecting of computers and network equipment.
The outcome of a well performed analysis serves to inform other decision makers about the state of one’s own network, perceived threats against it, suggested remedies for detected threats, as well as assessments of the implica- tions of cyber incidents, which is why analysts also should produce documentation. Incident reports are important for the purpose of record keeping and traceability, but also for enabling information sharing within the own organization, or to external parties. The amount of data and level of detail needed in the reporting, naturally depends on the purpose of the reports, but regardless of the requirements it is still good practice to create a record of detected incidents shortly after detecting them, to reduce the risk of forgetting important information.
This paper investigates what information elements should be included in an incident report to support incident management in a time-critical and high mental workload situation. The appropriateness and suitability of informa- tion elements were examined by analyzing the frequency of the existence of specific information elements in various existing templates, as well as by asking the rapporteurs in a military-civilian CDX called SAFE Cyber, about the perceived usefulness of different information elements. In addition, one novel template proposal from the military sector was used and evaluated during the CDX, although the exercise also sought to determine relevant information elements for the civilian sector.
In evaluating what information elements were consid- ered important and relevant, the proposed template was compared and contrasted to a few existing templates, the cyber situational awareness frameworks by Barford et al.
[4], and Dressler et al. [5], as well as with the threat information exchange framework STIX [6]. Beside our research effort, the CDX was further evaluated by how the proposed template was used, both by Cyber Defense Analysts and cyber management professional exercise par-
2020 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW)
ticipants. The aim with this, was to develop the format and content of future SAFE Cyber exercises.
The remainder of this paper is organized as follows. In the next section2, a brief overview of relevant research, e.g,. analyst’s work situation, information elements for cyber situation awareness, a framework for incident han- dling, and some incident handling templates are described.
In section 3, a military incident report template is de- scribed, and in section4this template is evaluated during the exercise SAFE Cyber. Results from the evaluation of the military template are presented and compared to other templates in section5. Finally, a discussion of the findings and their implications are presented in section6, together with a description of proposed future work.
2. Related Work
To provide a better understanding of the information elements that are relevant for incident reports, this section describes the role of Cyber Defense Analyst, various information elements and their connection to analysts’
cyber situation awareness. It also lists some established frameworks for incident reporting.
2.1. Cyber Defense Analysts
Monitoring an information system for cyber security incidents is normally a rather monotonous task, which involves identifying discrepancies, irregular patterns and other signs of real threats among many alerts. The work typically involves manual checking of event logs in mul- tiple systems and correlating them with each other. It has been shown that cyber analysts need to maintain sufficient levels of (cyber) vigilance to perform this task well [7].
Tasks are often divided so that some people prioritize alerts from security sensors, while others perform deeper analysis tasks that involve manual correlation of data from multiple systems [8]. A goal directed task analysis [9] and a hierarchical task analysis [1], both in security operations centers, confirmed that cyber defense analysts work tasks may essentially be divided into three roles: team leaders, scouts and analysts.
The team leaders’ tasks are to prioritize between possi- ble events, monitor possible escalations of events, retrieve external information (e.g., from open sources on the inter- net), and keep a journal of the incidents. The scout should identify incidents and potential threats, choose appropriate software for analysis and notify team leaders and analysts.
The analyst receives cases (work) from the team leader or the scout, selects analysis software, conducts a thorough analysis and reports back to the team leader. Traceability is essential for all roles. Hence, systematic record keeping and incident reporting are major parts of any cyber defense analyst’s work, regardless of their specific role, although the purpose of reporting and recipients may vary. There is plenty of research about cyber incident handling, but it often has a narrow technical focus [10] [11] [12].
2.2. Cyber Defense Analysis, Cyber Situation Awareness and Information Elements
A literature review was conducted in order to under- stand the field of cyber defense analysis with a focus
on cyber situational awareness and information elements.
Barford et al. [4] argued that there are at least seven aspects to consider to obtain cyber situational awareness, CSA: the current situation, the impact of attacks, how the situation evolves, the (threat) actor behavior, why and how the current situation was caused, the quality (trustworthi- ness) of the collected information, and plausible future developments of the situation. Dressler, Bowen, Moody and Koepke [5] proposed that six classes of informa- tion should be taken into account in order to establish operationally relevant situation awareness: the threat en- vironment, anomalous activity, vulnerabilities, key cyber terrain, operational readiness, and a grip of ongoing oper- ations. Mahoney et al. [13] presented preliminary findings from a cognitive task analysis conducted on a subject matter expert. Their analysis highlighted some preliminary categories of requirements and questions that could affect the design and development of a situation awareness tool.
Detection and management of threats presume a good understanding of the systems that are monitored, including awareness of vulnerabilities, the capability of available tools (sensors) for detecting anomalies, and the signifi- cance of detected anomalies [14] [15].
As for the data sources for situation awareness, Le Blanc et al. [16] systematically listed the different sources, e.g., the cyber security tools that provide analysts with data, in three categories. The first category is network- based tools, such as firewalls, intrusion detection or pre- vention systems (IDS/IPS), network mapping and visual- ization tools, and traffic capture tools. The second cat- egory consists of host-based sources, such as IDS/IPS, antivirus/malware detection software, and file system in- tegrity monitoring tools. The third category can be found at the system level (both host-, and network- based tools), and includes security information and event management tools (SIEM). Shiravi, Shiravi and Ghorbani [17] similarly sought to provide six potential data sources for network security visualizations in a review that listed the sources as: network traces, network events, application logs, secu- rity events, context related to network activity, and users and assets.
To better understand the adversary, work has been conducted on threat intelligence, which is an important but immature research area. To spread information and increase the understanding of the meaning of threat in- telligence, work has been carried out by the Centre for the Protection of National Infrastructure, describing strate- gic, tactical, technical and operational subtypes of threat intelligence [18]. Moreover, the financial sector has an interest in protecting its assets by better understanding its adversary, and has therefore carried out work about intelligence sharing and defining roles and skills in a threat intelligence team [19]. Gutzwiller, Fugate, Sawyer and Hancock [20] discussed links between cyber defense chal- lenges and major human factors research, e.g., situation awareness. They stated that technology has a critical role in the fight against cyber-attacks, but also emphasized that technology does not exist in isolation from human factors.
Brynielsson, Franke and Varga [21] reported that cyber situation awareness is generally vaguely defined, and from our review the literature describing what information ele- ments provide situation awareness is limited. Even though there are some papers about human factors and the cyber
domain, it is obvious that research within this domain is lacking [22], [23].
In sum, several studies about CSA and related topics provide direct or indirect clues about what information fields to include in an incident report.
2.3. Frameworks for Incident Handling
There are several suggestions on how to describe cyber incidents. The Structured Threat Information Expression (STIX) [6], the Vocabulary for Event Recording and Inci- dent Sharing (VERIS) [24], and NIST’s Computer Secu- rity Incident Handling Guide [25] are three interesting and valuable frameworks. We chose to discuss STIX due to its versatility, and neither VERIS’, nor NIST’s approaches are discussed further in this paper. The STIX framework is particularly interesting because it offers a standardized vocabulary and data exchange format for cyber informa- tion threats and therefore covers many data types that can be of interest in a cyber incident reporting template.
STIX [6] includes information about diverse areas that covers multiple aspects of what cyber defense ana- lysts need to observe when analyzing incidents, hence:
the analyst monitors the computer network and searches (sighting) for suspected malicious code. The adversaries are described by specifying who is behind the current attack (threat actor), whether the attack is part of larger attack scheme (campaign), and the opponent’s resources and behaviors (intrusion set). Based on knowledge of the attack, the person, group or organization behind the attack is searched for (identity). The analyst also determines what needs to be done to prevent attacks or stop an ongoing attack (course of action). This often means ensuring that vulnerabilities do not exist in the own network. To detect and identify who is behind an attack it is also necessary to collect information about the opponent’s tactics, meth- ods and procedures, e.g., attack patterns, tools and mal- ware. This information specifies how to detect the attack (indication) and what information collection is required (observed data).
The twelve domain objects in STIX [6], some italized in the paragraph above, are described by content and relationships to other objects (lines and arrows). Domain objects hold ten common properties; type, id, created-by- ref, created, modified, revoked, labels, external-references, object-marking-refs, and granular-markings. In addition, each domain object has specific characteristics. For ex- ample, attack patterns require the user to enter name, text description and where in the kill-chain the attack is located [26]. With 58 specific properties (3-10 per object) and 120 common properties (12 objects with 10 properties each, as specified above), the total number of properties is 178. Add to that, the many relationships between objects, and STIX must be described as a rich model with high granularity. The model can be complicated and time- consuming to use due to the amount of information needed to fully populate it, although the framework itself does not enforce strict usage of all attributes. In sum, the STIX framework broadly outlines potentially useful information elements for cyber reporting templates as well as their relations.
2.4. Report Templates for Incident Handling
This subsection outlines the context in which incident reporting occurs, discusses the perceived value of specific information elements, and the rationale for including vary- ing information elements in reporting templates.2.4.1. Instances of Incident Reporting. In the United Kingdom, the National Cyber Security Centre (UK NCSC), that was inaugurated in 2016, acts as a bridge between industry and the government [27]. The center provides advice, guidance and support for cyber security, including management of cyber security incidents. The UK NCSC uses a template that includes four categories with a total of 18 information elements. The categories are your details, organization details, incident details, and additional information.
The European NIS Directive1 was launched in 2018.
From this point onwards, member states need to imple- ment specific measures, e.g., setting up NIS authorities and central points of contact, specifying security mea- sures, agreeing on reporting obligations, and selecting and communicating essential services to operators. In Sweden, providers of critical services must report incidents to The Swedish Civil Contingencies Agency (In Swedish:
Myndigheten f¨or Samh¨allsskydd och Beredskap, MSB).
MSB is responsible for issues concerning civil protection, public safety, emergency management and civil defense.
An incident report template has been developed for this purpose [28], including nine categories with a total of 36 information elements (and some sub-categories). The categories in the template are: basic information, af- fected clients, assessed secrecy, report to the police, description of the event, time, incident category and sub-category, extent of the consequences, and other2.
Locked Shields is an annual exercise organized by the NATO Cooperative Cyber defense Centre of Excellence (CCDCOE). Cyber security experts train their skills in defending IT systems and critical infrastructure during real-time attacks in a CDX. In 2019, more than 1,200 experts from about 30 nations took part in the exercise.
Several reporting templates were used during the CDX.
In order to accommodate for the technical level threat reporting, a simple template with five categories was used:
description (free text), source (IP), destination/targets (IP), zone (subnet), and observables (IOC)3.
2.4.2. Relevance of Information Elements. Lif et al.
[9] developed and validated a freeze-probe technique to measure cyber defense analysts’ situation awareness. Two questionnaires were designed separately for the scout and analyst roles and evaluated in a realistic setting during an exercise involving five professionals. The questionnaires were well received by the analysts. The results suggested that the technique can be used to evaluate cyber situation awareness for analysts and function as a tool in the ana- lysts’ daily work to keep track of incidents. Based on these findings [9] and the reviewed work in the literature, a pro- posal was developed regarding what information elements
1. NIS = Network and Information Security 2. Author’s translation from Swedish 3. IOC = Indicator Of Compromise
should be included in rapid cyber incident reports. It was found that several questions used to measure cyber situa- tion awareness can also be used as information elements in an incident report. The proposal contained five categories with a total of 16 information elements. The template used in iPilot exercise 2018 was similar to other estab- lished standards, but was significantly simplified. As a result, it should be useful for information sharing activities where there is limited time to write full incident reports, e.g., in communication between scouts and analysts. An evaluation was conducted during the exercise iPilot, and the results showed a variation in the extent to which the information elements were used. The information ele- ments rapporteur, observation or indicators, victim node, description of suspected attack objective, asset criticality, attack impact, and response urgency were used in over 80
% of the incident reports on day two. Information about the source of the attacks and a description of this entity were reported far less frequently, which is not surprising as it is a much more difficult task than describing the indicators that the rapporteur reacted to. Although the participants could identify which node was attacked, they rarely reported what users were affected and did not give any further descriptions of the victim. According to Lif.
et al. [29] this was not surprising, because in several incidents entire systems and services were targeted, rather than individual users. The participants’ reactions to the selected information elements were positive and in that sense the template can be useful. The evaluation with participants working with cyber security indicated that these information elements can be used as a starting point for incident reports in different situations. The evaluated incident report template is a promising candidate for rapid reporting of critical information relating to cyber incidents in stressful situations, e.g., when a scout first detects a suspected incident and passes over the incident report to an analyst for deeper analysis. The information elements used by Lif et al. [30] in the exercise iPilot are essentially consistent with the contents of the VERIS and NIST frameworks. Both contain five categories, while STIX only contains background, attacker and description. As STIX was designed for information exchange, it focuses on generic elements of an attack and therefore does not include information tied to specific assets and their value.
2.4.3. Varying Number of Information Elements. There is limited consensus on what information elements are needed in an incident report. Some suggestions include, for example, abstract information elements such as health awareness (information that describes functional or po- tential capacity of the network) [13], while others suggest more concrete data elements such as IP addresses [31].
Some incident report templates are very brief, containing only a few data fields, while others include many details (often over 100 attributes), or require detailed textual descriptions of the incidents. Often neither are applicable in practical incident handling [30]. Due to workload and potential urgency to respond to critical events, incident handlers may require simplified reporting tools to keep track during their immediate response actions and preserve notes for post-handling incident reports.
3. Information Elements and the Proposed Military Reporting Template
The paper presented here evaluated an extensive inci- dent report template containing 29 information elements.
Although there were clear similarities with the previously used incident reports, this included more than twice as many information fields as were used during the iPilot exercise. Basic fields (e.g. rapporteur and date of the inci- dent) and information fields about damage consequences (e.g. time critically) were similar. Above all, the present incident report included more fields about the impact on other organizations and also included two fields about actions. The aim of the paper presented here was to evaluate the incident report template regarding what fields are relevant to use for civil organizations. In addition, two researchers analyzed the differences between the two incident report templates with regard to cyber defense analysts’ experiences of the templates in the iPilot and SAFE Cyber exercises. There was also an interest to evaluate the exercise in order to develop the content of a future cyber defense exercise (CDX) and table-top exercise (TTX) exercise. To investigate this, the template was evaluated during the exercise SAFE Cyber.
4. Evaluation of Information Elements
The information categories in the incident report tem- plate (see Section5.1, Table 3), were evaluated during the incident handling exercise SAFE Cyber.
4.1. The SAFE Cyber Exercise
The SAFE Cyber exercise was conceived by the Swedish Armed Forces for the overarching purpose of promoting inter-, and intra-agency-, as well as military- civilian cooperation, within the cyber defense realm. It was conducted in three phases over as many days: i) intro- ductory and training phase, ½ day, ii) the live exercise, 1 ½ day, and iii) exercise evaluation, ½ day. The exercise was furthermore divided into two separate parts with different formats, that ran simultaneously: a cyber defense exercise (CDX), and a table-top exercise (TTX).
The CDX aimed at practicing participants in detect- ing, reporting and managing (cyber) incidents, and more specifically to ensure the continuous operation of an IT- system in a simulated coal power plant, thus maintaining an adequate production level of electrical power. The CDX teams were to detect intrusions and misconfigurations in the computer networks, and remedy their effects.
The TTX personnel were tasked to handle strategic risk management and decision-making, or more specif- ically to evaluate and use incoming incident reports and other relevant information (injects) to carry out continuous risk management and perform strategic decision-making.
The TTX teams also had to perform (strategic) incident- handling.
The CDX organization during SAFE Cyber consisted of white, green, red and blue teams in accordance with in- ternational practice and terminology [32]. The white team had an exercise leader who had the overall responsibility of the exercise, a game leader who was responsible for
the game implementation, an evaluation team, as well as an exercise staff support function. The green team provided technical support to ensure that computers and networks were operational throughout the exercise. The red team was responsible for the planning and launch of incidents, i.e attacks, prior to, and during the exercise.
There were four blue teams, each containing five to six cyber defense analysts, that had cyber defense tasks as mentioned previously. The participant of the table-top exercise were split into four groups, that each had six to seven management professionals. The CDX and TTX teams did not exchange information directly. The flow of relevant information from the CDX to the TTX was relayed via the white team, with the exception of the incident reports that were communicated directly to the TTX teams.
In alignment with the purpose of this paper, neither the TTX itself, nor the broader interactions between the CDX and the TTX, are described in any detail.
4.2. SAFE Cyber Environment
This section describes the technical environment and characterizes the participants of the SAFE Cyber exercise.
4.2.1. Technical Environment. The CDX was executed in the Swedish Defence Research Agency’s Cyber RAnge and Training Environment (CRATE) [33]. Each exercise team was given a virtual environment to monitor and protect. The milieu was to mimic that of a small coal production plant. The network, that was identical to all teams, had two main segments: a typical office network that contained web servers, mail servers, file servers, net- work equipment and ten office computers with industrial control systems that were connected to, and controlled, the production process. The office network was separated from the industrial control system by firewalls.
The system was intentionally pre-configured with dated, unpatched software and some poorly configured services in order to generate enough vulnerabilities for the red team to attack the blue teams with widely known ex- ploits. The attacks were carried out using a tool developed in-house called Scanning, Vulnerabilities, Exploits and Detection (SVED) [34]. SVED attacks could be scripted and launched in parallel against all blue teams, which made it possible to compare the teams’ response efforts.
Example attacks included simple network scans, brute force attack on network services, malware-infected thumb drives, and denial of service attacks.
The participants had been provided with a description of the cyber environment before the exercise started. Each of the participants in a team was equipped with one laptop with various Linux tools for monitoring and examining their network. The teams used sensors to monitor network traffic at six distinct probe locations in their respective net- work. For analysis they had access to the tools Wireshark [35], Snorby [36] (graphical frontend to Snort [37], and Etherape [38]. System events such as Windows Event Log [39] were collected in a central system.
The TTX was performed in a regular office environ- ment with discussions being held around a table. However, as the focus of this paper is on the CDX participants’
use of the incident template the TTX will not be further discussed.
To facilitate if information handling during the exer- cise, the in-house tool Cyber Exercise Control (CEC), developed by the Swedish Defence Research Agency, was used. This tool was used to visualize all events along a timeline (such as attacks, detection of attacks and incident reports). Moreover, as part of the tool, the template for incident reports was implemented.
4.2.2. Participants. All exercise participants, 51 individ- uals, were professional IT administrators or cyber security specialists with experience from cyber incident handling practices. The 22 participants in the CDX exercise and 29 in the TTX, worked at 17 different organizations. Beside the Swedish Armed Forces, there were other governmental agencies, such as the Swedish Contingencies Management Agency (Swedish: MSB), the security police, the police etc., as well as commercial companies, e.g., banks, utility- , and telecom companies etc. The partakers were delib- erately assigned into mixed teams with members from different organizations, in line with the overall exercise goal of promoting inter-agency cooperation. Thus, most people did not know any other member(s) in their team when the exercise started.
This paper focuses on evaluating the information el- ements included in the incident report used during the SAFE Cyber exercise. The template with 29 information elements was evaluated with respect to the following research questions:
1) To what extent did the content of five exercise templates, STIX, and two scientific references with focus on incident management and situa- tional awareness for incident managers match each other?
2) How was the SAFE Cyber exercise experienced in general?
3) To what extent was each of the information ele- ments used by the cyber defense analysts during the exercise?
4) How did the cyber defense analysts assess possi- ble use of the information elements at their regu- lar work place and not just during the exercise?
4.3. Data collection
Data, both for the sake of the exercise and for research purposes, was collected from the exercise in multiple ways. The white team reviewed all incident reports and assessed their quality according to a predefined scoring system. This scoring system was developed prior to the exercise. Some information elements had a correct or incorrect answer, while others had several correct response alternatives. Information elements about the technical con- tent of the incident (e.g., detect defaced website) had one correct or incorrect answer, while some information fields entailed an assessment of the situation (e.g., a legal paragraph that was not applicable but partly used during SAFE Cyber), which meant that there was not always a correct answer. In cases where there was no right or wrong answer, participants received points if they appeared to have made an effort to fill in the incident
report regardless of the information quality. There was one background survey for all participants and one survey after the CDX and TTX exercise, respectively. The survey after the CDX exercise contained overall questions about the completed exercise (e.g., realism, learning and teamwork), and questions regarding whether the information elements in the template are applicable in the participants’ real working environment. The survey after the TTX exercise contained the same overall questions, but questions about information sharing between organizations were focused upon, rather than questions about the template.
5. Results
This paper focuses on information elements that should be included in an incident report to support incident management. That is why the four CDX teams, and not the TTX teams, were primary evaluated. Results are therefore solely based on data from the 22 participants in the CDX exercise. The four groups submitted a total of 107 incident reports. Group one submitted 23 reports; group two 32 reports; group three 36 reports; and group four 16 reports. The results are based on the aforementioned research questions. These were analyzed quantitatively and qualitatively with respect to information elements usage and applicability, as well as the perceived value of each of the 16 information elements. The four subsections below correspond to the four research questions previously outlined.
5.1. Quantitative Comparisons of Incident Re- port Templates
The contents of five existing incident templates were analyzed in order to gain a better understanding of infor- mation elements that can be included in incident reports.
The templates of interest were obtained from the iPilot [30], SAFE Cyber and Locked Shields [40] exercises. In addition, the MSB template that is used at the national level in Sweden was of interest, as was the template used by the UK NCSC. Furthermore, the STIX framework and two scientific papers [4], [5] were also analyzed to get further insights about possible information to include in incident reports.
The starting point was the five categories used in the exercise presented in this paper (SAFE Cyber). However, the names were somewhat altered to better match the content of the different templates. The five categories that emerged were: Basic information, Description of the incident, Damage consequences, Effect on operations and communication, and Actions taken or planned.
The four exercise templates evaluated were SAFE Cyber, henceforth:SC; iPilot (iP); Locked Shields (LS); MSB (MSB); and the United Kingdom National Cyber Security Centre (UK NCSC) (Table 1). These abbreviations are used in all tables.
Since the purpose of the templates, STIX, and the contributions of Barford et al. [4], and Dressler et al.
[5] overall focus differ, it is not surprising that they do not have the same information elements. However, the reviewed templates provide a good overview of possible fields to include. They can serve as a starting-point for
TABLE 1. CATEGORIES INSAFE CYBER TEMPLATE AND RELATIONS TO OTHER TEMPLATES,FRAMEWORKS AND REFERENCES.
SC iP LS MSB NCSC STIX Barford Dressler
Basic info. x x x x
Description x x x x x x x x
Consequences x x x x x
Effects x x
Actions x
the development and evaluation of information elements to be included in novel reporting templates. The scope of the content in each category varied between the dif- ferent references. For example, STIX described the in- cident extensively, but did not include other categories.
Moreover, MSB focused on the actual incident, but lacked information elements about, e.g., effects on operations and communications. The content of the incident description (Table 2) and consequences (Table 3) were of special interest as there must always be an incident description, and the consequences are of great importance regardless of organization. Basic information, effects on communication or management, and actions taken to solve the problem are important elements, although they are considered less interesting in the present paper.
TABLE 2. CONTENT OF INCIDENT DESCRIPTION INSAFE CYBER TEMPLATE AND RELATIONS TO OTHER TEMPLATES,FRAMEWORKS
AND REFERENCES.
SC iP LS MSB NCSC STIX Barford Dressler
Affected system x x x x x x
Time/date x x x
System owner x x
Category x x x x
Sub-category x x x
Incident status x x x x x x
How detected x x x x
Victim x x x x
Attacker x x x
Attacker motive x x
TABLE 3. CONTENT OF CONSEQUENCES INSAFE CYBER TEMPLATE AND RELATIONS TO OTHER TEMPLATES,FRAMEWORKS AND
REFERENCES.
SC iP LS MSB NCSC STIX Barford Dressler
Severity x x x x
Risk x
Exposure x Time critically x x
Reliability ass. x x
Success prob. x
Ongoing op. x x
Op. readiness x
5.2. Overall Subjective Evaluation of the Exercise
In order to answer whether the information elements in the incident report template were relevant in other contexts, the participants answered questionnaires on a seven-point grading scale. A low value was always neg- ative and a high value positive, that is: 1 = very low and 7=very high. The questionnaire was concluded with one open question where the participants could writeoverall subjective comments. The subjective ratings of the exercise (mean=5.5) and the work in mixed groups with participants from different organizations (mean=6.1) were positive. Exercise realism (mean=4.8) was rated to be at a medium level. The incident report template (mean=4.1), which is the main topic of this paper, received a medium level overall rating.
5.3. Quantitative Use of the Information Ele- ments
Here we present a description of the information el- ements that were used during SAFE Cyber. The results are based on the 107 submitted incident reports. The descriptions are given in five categories - named slightly different than the ones in Section 5.1. The categories contain a total of 29 information elements.
The analysis of the category About the report, showed that the information elements Date and Rappor- teur were often used (above 85%), while Secrecy level was used less frequently (66%), and Legal paragraph was used very rarely (7%) (Fig. 1). The reason why Secrecy level and particularly Legal paragraph were used to such a limited extent was not studied in detail. However, in discussions during and after the exercise participants expressed that they were not used to making these type of assessments in the civilian sector.
Figure 1. The About the report category.
The analysis of the category What happened showed that all information elements were used in 75% or more in the incident reports. This is an acceptable level, indicating that the participants did their best to gather information about ongoing incidents.
The information elements in the category Conse- quences were used in 75-77% of all incident reports, with the exception of Exposure, which was only used in 43%
of the reports (Fig. 2). The reason why Exposure was so rarely used is not clear, but a probable explanation is that it was difficult to make this assessment. In addition, the fact that the assessment criteria were 1-4 without any further explanations made the assessments more difficult for the participants to make.
The information elements in the category Impact were used in about 65% of all incident reports, with the excep- tion of External operator which was only used in 37%
of the reports. The information elements in this category were very difficult to assess. It needs to be further investi- gated whether this was due to technical reasons where the participants were not in control of the whole system and who they should communicate with in different situations,
Figure 2. The Damage consequences category.
or not. Furthermore, it should be investigated if these assessments should be conducted on a management level rather than on a technical level.
The category Actions that contained the elements Actions taken and Planned actions, were used in 66% and 75% respectively in the reports. It is not surprising that these values are at this level as the focus of the exercise was not to solve the actual problems, but rather to detect and report vulnerabilities and incidents to the (in-game) management teams.
5.4. Subjective Views on Information Elements for Civilian Use
The proportion of used information elements varied greatly between different elements (data from survey after the exercise). To analyze these results, an ANOVA re- peated measures and Tukey HSD post hoc test were used.
The ANOVA tell if the results are significant overall and the Tukey’s HSD find out which specific group’s means are different.
A comparison was made between the five categories included in the incident report template (Fig.3). Overall, four of the five categories were rated high (mean 4.5-5.5), and one category was rated more modestly. The ANOVA showed a significant effect (F(4, 172)=47.1, p<.001), and the post hoc test showed that the categories What happened and Actions were rated higher than the three categories About the report, Consequences and Impact (p<.001). Moreover, the analysis showed that the cate- gories About the report and Consequences were rated higher than Impact (p<.001).
Figure 3. Subjective rating of information categories (1).
Next, each category’s information elements were an- alyzed to obtain an understanding of what elements the participants experienced to be most relevant in their own organizations.
First, the four information fields in the category About the report were analyzed (Fig.4). The ANOVA showed a significant effect F(3, 54)=37.0, p<.001, and the post hoc test showed that the information fields Date and Rapporteur were rated significantly higher than Secrecy level and Legal paragraph (p<.001). In addition, Secrecy level was rated significantly higher than Legal paragraph (p<.005).
Figure 4. Subjective rating of information categories (2).
Second, the seven information fields in the category What happened were analyzed. Overall, these infor- mation fields were rated high (mean values above 5).
The ANOVA showed a significant effect (F(6, 120)=15.9, p<.001), and the post hoc test showed that the information fields Affected system and Incident time/date were rated significantly higher than System owner, Category, Sub- category and Status ; (p<.05). Furthermore, Text descrip- tion was rated significantly higher than Sub-category (p¡
.001), which had the lowest rating within this category.
Third, the five information fields in the category Dam- age consequences were analyzed. Overall, these fields were rated at a medium level (with mean values about 5), except for Severity, which was rated higher. The ANOVA showed a significant effect (F(4, 68)=13.5, p<.001), and the post hoc test showed that the information field Severity was rated significantly higher than the four other informa- tion fields in this category (p<.05).
Fourth, the ten information fields in the category Impact were analyzed. These information fields were generally rated at a medium level (mean values between 4 and 5). The ANOVA showed a significant effect (F(9, 144)=5.1, p<.001), and the post hoc test showed that Communication with pendant organizations was rated sig- nificantly higher than most of the other information fields within this category, but not higher than Operation man- agement and Communication with third party (p<.001).
Even though there are statistical differences it should be noted that these are small in this category.
Fifth and finally, the analysis of the two information fields in the category Actions showed that they were rated at a very high level (mean over 6). The ANOVA showed no significant effects ; (F(1, 21)= .7, p>.05).
6. Discussion
Experiences from previous exercises show that cyber defense analysts prefer to work with actual incidents and find it less stimulating to write incident reports. They generally prefer simplified templates, especially when the incident handling is focused on the technical level.
6.1. The Various Templates
It is clear that the required information in the five analyzed templates (SAFE Cyber, iPilot, Locked Shields MSB, and UK NCSC) differ. One reason for this is that the purpose of the templates differ. The template used during SAFE Cyber was designed to report incidents in a military setting. It therefore includes information fields about anticipated or experienced effects on operations and (tele)communications, in a way that none of the other templates do. The template for iPilot [30], by contrast, was developed by researchers, where a one goal was to produce a simplified, manageable and useful template especially for initial phases of incidents. It was optimized to meet requirements in environments where factors like time criticality and stress affect the working conditions of cyber defense analysts to a high degree. The template used during Locked Shields contains only five information elements. It was probably designed to function as a basis for further reporting and analysis, rather than serve as a full-fledged reporting tool in real settings. This template can consequently also be seen as a simplified template for the initial phase of incident reporting. More infor- mation elements are needed for reporting by companies or authorities in real situations. MSB’s template [28] is used to obtain information from, e.g., suppliers of critical infrastructure services and other governmental agencies.
MSB’s overarching task to this respect, is to support societal efforts at large to manage and prevent IT-related incidents, and their template was developed to further that goal.
6.2. Frameworks
The STIX framework focuses on, and is quite useful for, data exchange about cyber threats. STIX is therefore cyber incident-centric. The possibility to describe inci- dents, however, are comprehensive and exhaustive, and the template includes significantly more information elements than any other template analyzed in this paper.
Barford et al. [4] focus broadly on cyber situation awareness, which is why it not only asks for information elements that describes an incident in isolation. Barford’s contribution is interesting because it addresses several contextual factors that the templates do not include. Be- sides describing incidents and opponents, Barford et al.
[4] emphasize factors such as how situations evolve, trust- worthiness of collected information, and possible future scenarios. Furthermore, research show that information about all aspects put forth in Barford’s model, with the possible exception of information about adversaries, is wanted on the managerial level [41]. Moreover, Dressler, et al. [5], also have a more comprehensive view of what cyber defense analysts need to be aware of, than the templates. They argue that six classes of information are essential, in a similar fashion to Barford et al. [4].
6.3. Analysis of the SAFE Cyber Template
The evaluation of the SAFE Cyber template clearly pointed out that respondents considered four of the five categories relevant for incident reporting in the civilian sector too. The only category not deemed relevant, was Impact. This category focused on the reported incident’s impact on the ability to manage both national and inter- national (military) operations. The category also sought to clarify the ability to communicate within the own orga- nization, as well as to external parties. These kinds over- arching managerial issues, thus appear to be of limited interest for a technical cyber defense team, especially in the civilian sector.It appears that Cyber defense analysts focus on the technical level and find it difficult to assess the con- sequences of an incident on, e.g., whether communica- tion between different sections of an organization are affected or not. At the same time, these types of assess- ments can also be difficult to conduct at the management level. It is fair to assume that the management has lim- ited understanding of technical details and relevant inter- dependencies. To solve this problem, increased collabora- tion between the technical and the management level can be proposed.
The category About the report, was regarded as basic and unproblematic to fill in, it provides essential information. In a military organization it is important to immediately determine the required level of secrecy, e.g.
whether to classify information or not. The results from SAFE Cyber showed that this information element re- quirement was less relevant for the civilian sector. This is interesting, because this type of assessment is traditionally not carried out at the technical level. However, in Sweden a new piece of legislation that came into effect on April 1st, 2019 [42], states that this type of information has to be reported also in the civilian sector.
Within the category What happened, all informa- tion elements were considered to be important. Such a basic description of incidents is obviously relevant for both the military and civilian sector. Although the SAFE Cyber template already contains seven elements, there are reasons to ponder whether supplementing those with additional elements would be useful. For example; more elaborate descriptions of attackers, attacker motives and their modus operandi could be added.
The category Consequences, was also rated relatively high (on average above five on the seven-grade scale) during SAFE Cyber. This can not be considered surprising since it is of the utmost importance for all organizations to understand the consequences of a cyber incident. Also, Severity was rated higher than Risk, Exposure, Time criti- cality, and Assessment reliability. Still, all elements were considered to be important.
The category Actions, i.e., measures taken, was con- sidered very relevant during SAFE Cyber. This clearly shows that measures already taken or planned, should be included as information elements in an incident template.
It is obvious that information elements in the category Impact were rated lower than information elements in other categories. However, this does not necessarily imply that these elements should be excluded from template altogether. In discussions with participants in the CDX it
was noted that they rated Impact as important, but not in the way it was asked for in the template. To understand the impact on communications was seen as important, but they felt that it was not relevant to them, given their civilian context. It is possible that these kinds of assessments should be done by a management team and not by a technical team. However, this was not investigated during the exercise, but should be further investigated in future work.
7. Conclusions and Future Work
Here we answer our research questions and outline possible future work.
The analysis of five reporting templates, a data ex- change format and two situational awareness frameworks shows that the required information elements differ. This is probably due to their different areas of use. When it comes to the evaluation of the exercise, the participants ranked it to be good, although moderately realistic. The report template, which is the main subject of this paper, was regarded to be of medium use. Some information categories were used more than others, such as the basic About the report category. The more complicated ones, especially e.g. Consequences and Impact categories, that require extensive analysis were used to a lesser extent.
Some individual elements were used very rarely, e.g.
Secrecy level and Legal paragraph. The overall results, however, showed that most of the information elements were considered to be very useful both in the military and civilian sector. Some modifications were suggested, such as adding descriptions of victims, attackers, and assessments of attackers’ motives.
Future work needs to further explore the importance of key definitions (e.g., severity, risk, exposure, time crit- icality etc.), and whether all participants perceive them in the same way.
Acknowledgments
The work of Stefan Varga was sponsored by the Swedish Armed Forces.
References
[1] P. M. Salmon, N. A. Stanton, G. H. Walker, D. Jenkins, D. Ladva, L. Rafferty, and M. Young, “Measuring Situation Awareness in complex systems: Comparison of measures study,” International Journal of Industrial Ergonomics, vol. 39, no. 3, pp. 490–500, 2009.
[2] NIST, “Framework for Improving Critical Infrastructure Cybersecurity,” Gaithersburg, Tech. Rep., 2014. [On- line]. Available: https://www.nist.gov/sites/default/files/documents/
cyberframework/cybersecurity-framework-021214.pdf
[3] T. Sommestad and A. Hunstad, “Intrusion detection and the role of the system administrator,” Information Management & Computer Security, vol. 21, no. 1, pp. 30–40, mar 2013. [Online]. Available:
http://www.emeraldinsight.com/doi/10.1108/09685221311314400 [4] P. Barford, M. Dacier, T. G. Dietterich, M. Fredrikson, J. Giffin,
S. Jajodia, S. Jha, J. Li, P. Liu, P. Ning, X. Ou, D. Song, L. Strater, V. Swarup, G. Tadda, C. Wang, and J. Yen,
“Cyber SA: Situational Awareness for Cyber Defense,” in Cyber Situation awareness, S. Jajodia, P. Liu, V. Swarup, and C. Wang, Eds. London: Springer, 2010, pp. 3–14. [Online]. Available:
http://link.springer.com/10.1007/978-1-4419-0140-8 1
[5] J. Dressler, C. L. Bowen, W. Moody, and J. Koepke, “Operational data classes for establishing situational awareness in cyberspace,”
in 2014 6th International Conference On Cyber Conflict (CyCon 2014). IEEE, jun 2014, pp. 175–186. [Online]. Available:
http://ieeexplore.ieee.org/document/6916402/
[6] STIX, “STIX Whitepaper — STIX Project Documentation,” 2017.
[Online]. Available: http://stixproject.github.io/getting-started/
whitepaper/
[7] B. D. Sawyer, V. S. Finomore, G. J. Funke, V. F. Mancuso, M. E. Funke, G. Matthews, and J. S. Warm, “Cyber Vigilance,”
Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 58, no. 1, pp. 1771–1775, sep 2014. [Online]. Avail- able: http://journals.sagepub.com/doi/10.1177/1541931214581369 [8] C. Zimmerman, Ten Strategies of a World-Class Cybersecurity
Operations Center. Bedford, MA: The Mitre Corporation, 2000.
[9] P. Lif, M. Gran˚asen, and T. Sommestad, “Development and validation of technique to measure cyber situation awareness,” in 2017 International Conference On Cyber Situational Awareness, Data Analytics And Assessment (Cyber SA). IEEE, jun 2017, pp. 1–8. [Online]. Available: http://ieeexplore.ieee.org/document/
8073388/
[10] S. Miserendino, C. Maynard, and J. Davis, “ThreatVectors: Con- textual workflows and visualizations for rapid cyber event triage,”
in Proceedings of the 2017 International Conference On Cyber Incident Response, Coordination, Containment & Control (Cyber Incident), London, 2017.
[11] T. Sommestad, M. Ekstedt, and P. Johnson, “Cyber Security Risks Assessment with Bayesian Defense Graphs and Architectural Mod- els,” in Proceedings of the 42nd Annual Hawaii International Conference on System Sciences, Big Island, 2009, pp. 1–10.
[12] S. J. Yang, H. Du, J. Holsopple, and M. Sudit, “Attack Projection,”
in Advances in Information Security 62: Cyber Defense and Sit- uational Awareness, A. Kott, C. Wang, and R. F. Erbacher, Eds.
Cham: Springer, 2014, pp. 239–261.
[13] S. Mahoney, E. Roth, K. Steinke, J. Pfautz, C. Wu, and M. Farry,
“A cognitive task analysis for cyber situational awareness,” in Proceedings of the Human Factors and Ergonomics Society, vol. 1, 2010.
[14] J. R. Goodall, W. G. Lutters, and A. Komlodi, “Developing expertise for network intrusion detection,” Information Technology
& People, vol. 22, no. 2, pp. 92–108, jun 2009. [Online]. Available:
http://www.emeraldinsight.com/doi/10.1108/09593840910962186 [15] R. Werlinger, K. Muldner, K. Hawkey, and K. Beznosov,
“Preparation, detection, and analysis: the diagnostic work of IT security incident response,” Information Management & Computer Security, vol. 18, no. 1, pp. 26–42, mar 2010. [Online]. Available:
http://www.emeraldinsight.com/doi/10.1108/09685221011035241 [16] K. Le Blanc, A. Ashok, L. Franklin, J. Scholtz, E. Andersen, and
M. Cassiadoro, “Characterizing cyber tools for monitoring power grid systems: What information is available and who needs it?”
in 2017 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2017, vol. 2017-January. Institute of Electrical and Electronics Engineers Inc., nov 2017, pp. 3451–3456.
[17] H. Shiravi, A. Shiravi, and A. A. Ghorbani, “A Survey of Visualization Systems for Network Security,” IEEE Transactions on Visualization and Computer Graphics, vol. 18, no. 8, pp.
1313–1329, aug 2012. [Online]. Available: http://ieeexplore.ieee.
org/document/6007132/
[18] D. Chismon and M. Ruks, “Threat intelligence: Collecting, analysing, evaluating,” Centre for the protection of national in- frastructure, Tech. Rep., 2015.
[19] B. of England, “Cbest intelligence-led testing - understanding cyber threat intelligence operations,” Tech. Rep., 2016.
[Online]. Available: https://www.bankofengland.co.uk/-/
media/boe/files/financial-stability/financial-sector-continuity/
understanding-cyber-threat-intelligence-operations.pdf?la=en{&}
hash=4B42B0382D1D33402689478433E2E1E0FFA93055 [20] R. S. Gutzwiller, S. Fugate, B. D. Sawyer, and P. A. Hancock,
“The Human Factors of Cyber Network Defense,” Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 59, no. 1, pp. 322–326, sep 2016. [Online]. Available:
http://journals.sagepub.com/doi/10.1177/1541931215591067
[21] J. Brynielsson, U. Franke, and S. Varga, “Cyber Situational Awareness Testing,” in Combatting Cybercrime and Cybert- errorism. Advanced Sciences and Technologies for Security Applications., B. Akhgar and B. Brewster, Eds. Springer International Publishing, 2016, pp. 209–233. [Online]. Available:
http://link.springer.com/10.1007/978-3-319-38930-1 12 [22] A. Vieane, G. Funke, R. Gutzwiller, V. Mancuso, B. Sawyer, and
C. Wickens, “Addressing Human Factors Gaps in Cyber Defense,”
in Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 2016, pp. 770–773.
[23] B. J. Knox, R. G. Lugo, Ø. Jøsok, K. Helkala, and S. S¨utterlin,
“Towards a cognitive agility index: The role of metacognition in human computer interaction,” in Communications in Computer and Information Science, vol. 713. Springer Verlag, 2017, pp. 330–
338.
[24] VERIS, “The VERIS Framework,” Tech. Rep., 2017. [Online].
Available: http://veriscommunity.net/
[25] P. Cichonski, T. Millar, T. Grance, and K. Scarfone, “Computer Security Incident Handling Guide Recommendations of the Na- tional Institute of Standards and Technology,” U.S. Department of Commerce, Gaithersburg, MD. USA, Tech. Rep. Revision 2, 2012.
[26] LockheedMartin, “Cyber Kill Chain® · Lockheed Martin,”
2017. [Online]. Available: http://www.lockheedmartin.com/us/
news/features/2014/isgs-cyber-kill-chain.html
[27] NCSC, “Reporting a cyber security incident,” 2019. [Online].
Available: https://report.ncsc.gov.uk/
[28] MSB, “Formul¨ar f¨or it-incidentrapportering,” 2019. [Online].
Available: https://www.msb.se
[29] P. Lif, M. Gran˚asen, and T. Sommestad, “Development and validation of technique to measure cyber situation awareness,” in 2017 International Conference On Cyber Situational Awareness, Data Analytics And Assessment (Cyber SA). IEEE, jun 2017, pp. 1–8. [Online]. Available: http://ieeexplore.ieee.org/document/
8073388/
[30] P. Lif, T. Sommestad, and D. Gran˚asen, “Development and evaluation of information elements for simplified cyber-incident reports,” in Cyber SA 2018. C-MRIC.org, 2018. [Online].
Available: https://www.c-mric.com/cs2018/
[31] A. D’Amico, K. Whitley, D. Tesone, B. O’Brien, and E. Roth,
“Achieving Cyber Defense Situational Awareness: A Cognitive Task Analysis of Information Assurance Analysts,” Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 49, no. 3, pp. 229–233, sep 2005. [Online]. Available:
http://pro.sagepub.com/lookup/doi/10.1177/154193120504900304 [32] N. Wilhelmson and T. Svensson, “Handbook for planning, running
and evaluating information technology and cyber security exer- cises,” Stockholm, Tech. Rep., 2013.
[33] T. Sommestad, “STO-MP-IST-133 Experimentation on operational cyber security in CRATE,” Tech. Rep., 2015.
[34] H. Holm and T. Sommestad, “SVED: Scanning, Vulnerabilities, Exploits and Detection,” in MILCOM 2016 - 2016 IEEE Military Communications Conference. IEEE, nov 2016, pp. 976–981.
[Online]. Available: http://ieeexplore.ieee.org/document/7795457/
[35] Wireshark, “Wireshark · Go Deep.” 2017. [Online]. Available:
https://www.wireshark.org/
[36] Github, “Snorby,” 2017. [Online]. Available: https://github.com/
Snorby/snorby
[37] Snort, “Snort - Network Intrusion Detection & Prevention System,” 2017. [Online]. Available: https://www.snort.org/
[38] Etherape, “EtherApe, a graphical network monitor,” 2017.
[Online]. Available: http://etherape.sourceforge.net/
[39] Microsoft, “Windows Event Logs,” 2017. [Online]. Avail- able: https://technet.microsoft.com/en-us/library/cc722404(v=ws.
11).aspx
[40] CCDCOE, “Locked Shields,” 2019. [Online]. Available: https:
//ccdcoe.org/exercises/locked-shields/
[41] S. Varga, J. Brynielsson, and U. Franke, “Information requirements for national level cyber situational awareness,” in 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), Barcelona, Spain, 2018, pp. 774–781.
[42] Regeringskansliet, “S¨akerhetsskyddslag,” 2019. [Online]. Avail- able: https://svenskforfattningssamling.se/doc/2018585.html