Case study

The Delphi study delivered a risk map that needed to be tested in a real context. Hasson and Keeney (2011) discuss that the results of a Delphi study do not offer indisputable fact and that, instead, they offer a snapshot of expert opinions, for that group, at a particular time, which can be used to inform thinking, practice or theory. As such, Delphi findings should be compared with other relevant evidence in the field and verified with further research to enable findings to be tested against observed data to enhance confidence (Hasson & Keeney, 2011). In order to test the quality of the results, a real organisation would have to be involved in the testing.

Three possible methods to test the HI-risk output were considered. The first option was to hold a survey amongst healthcare organisations. Organisations could be presented with the scenarios and indicate how much they agree with the expected frequency. A survey would have the advantage that many organisations in different locations could be involved, but it is limited in investigating the organisational context. Furthermore, it

would not provide an opportunity to test the quality of the map against a list of real incidents that had occurred.

The second option was an experiment with the model in a controlled environment. This option could deliver a concrete product: the prototype of an expert system for incident and risk monitoring, but it might be difficult to compare the results with real data in their context. Extending this prototyping to a real situation would require a longitudinal study within multiple healthcare organisations that would use the expert system for a longer period of time. This would have been the preferred option if the research had been sponsored. However, this study was self-funded, so unfortunately this was not feasible.

Considering the circumstances, a case study using multiple research techniques was chosen as the best possible strategy to validate the method. A case study suits the type of ‘how’ research questions (Yin, 2009), looking to find out how staff behaves and how information security risks are identified and controlled. On top of that it provides the opportunity to have in-depth conversations with employees in healthcare to gain a better understanding of the socio-technical context of information security. Furthermore, it is possible to actually observe people in their working environment to test some of the risk scenarios. Finally, it provides an opportunity to run a simulation with the risk map, using real information security incident data.

The case study was held at a large NHS hospital. A sponsor was found in the Speech and Language Therapy department. The case study proposal was given to Edinburgh Napier University’s ethics committee within the School of Computing for consideration in September 2012. Although patients were not involved in this study and there was no need to access patient records, and thus formally this type of study would only need approval from the institution where the research will be conducted (NHS, 2012), it would have been possible that –as a visitor to a healthcare organisation- personal information was overheard or patients could be seen. Furthermore, during observations of staff and their security behaviour, members of NHS staff might feel uncomfortable during observations as they might see the researcher taking notes of non-compliant behaviour. Finally, the registers of incident data that were analysed could potentially contain sensitive information as well. For these reasons, ethical approval was requested from the university’s ethics committee.

The School of Computing ethics committee decided to refer the request for approval of the case study to the ethics committee within the faculty of Life Sciences. This second committee did not approve of the research and decided that the researcher needed a NHS research passport. This passport is provided by the university having undertaken all the appropriate disclosures and checks on the student and confirms this to the NHS partner. Unfortunately, at that time, the university did not have a process in place to provide research passports and the case was taken to the University degrees committee and to the University Integrity committee in December 2012.

After completing several forms to allow a criminal records check of the researcher, and spending several weeks waiting for feedback or progress, the research coordinator within the case organisation advised in January 2013 that, after all, no disclosure approval nor research passport was needed (NHS, 2012). Furthermore, the university’s ethics committee approved the research proposal on 14 February 2013. After that, the correct approval process ran through online forms, which needed completion in the Integrated Research Application System (IRAS). After the necessary authorizations within the case organisation were signed off on 25 March 2013, the case study took place in April 2013.

Semi-structured interviews were held with the IT Security Manager and with the two Information Governance Leads of the Speech and Language Therapy Department.

These persons were selected because of their knowledge of information governance and risk management processes and their leading role in promoting secure behaviour amongst staff. The interviews were guided by a list of open-ended questions and more questions were created during the interviews. The set-up was face-to-face and the interviews were voice recorded and transcribed. The interviewees were asked general questions about information governance and information security, about their approach to risk assessment and their opinion about the most important risks. During the interviews, new potential risk factors were identified and these were added to the classification of risk factors. Furthermore, the researcher gained more knowledge about daily information security routines, policies, risk assessment methods and organisational culture.

Observations were held in two locations of the Speech and Languages Therapy department. The aim of the non-participative observations was to test if any risk scenarios could be spotted and if they would fit in the classification. The aim was specifically not to audit staff or to report any potential incidents, as was pointed out to

the staff in a preliminary briefing. Staff were observed in their daily routines, without disturbing them. Any potential information security risks were noted and matched against the classification. This led to adjustments of the classification where risk factors were recognised that had not been listed yet. The aim was to test the classification, not to test the security of the case organisation. The observation form to take notes on was the classification itself (presented in Table 5.6 in this thesis), and observed categories were ticked and missing categories were noted and added.

On behalf of the researcher, the IT Security Manager forwarded a survey to his colleagues who regularly participate in risk assessments. The survey was created online in SurveyGizmo with the aim of surveying the business requirements for the HI-risk method. Unfortunately, only three responses were received and the IT Security Manager indicated that it was unlikely to receive a better response, as only a few members of staff perform risk assessments. Therefore, the results of this survey cannot be used for generalisation, but they still provide a useful indication of opinions about risk assessment methods within the IT department. The survey contained nine questions about risk assessment methods, frequently occurring risks and risk management.

The ultimate test of the quality of the forecasts was the analysis of the incident register.

The incident register data was copied into the HI-risk database and benchmarked against the risk map, using the same scenario analysis technique as before with the primary data. This led to conclusions about the quality of the forecast shown in the risk map.