Lack of Experience or Knowledge

For the participants, the lack of experience or knowledge was related to both management and the security professionals being managed. However, this lack of experience was not limited to those working in the security field but was extended to those security professionals engaged in their reporting roles. For security professionals, this lack of experience was typically related to being new to the field or when changing to a new security domain. One participant recounted a case in which new security analysts were writing reports after validating their findings:

That expertise wasn’t there…yet to ensure if it’s being vetted correctly or not; a couple of analysts would not report everything, and it turned out that they were missing out [on] true positive findings.

Another participant who at the time was a manager described a similar issue they experienced in their own team from new security analysts whom they supervised. The team was responsible for triaging security issues that were sent to them through a ticketing system. The analysts would quickly close out a security ticket, and the participant began questioning whether the issue was actually resolved:

A lot of times you would actually question the amount of visibility that the team had [or] how well they can actually deep dive into an issue. So, you would [repeatedly] see an issue get closed pretty quickly and you’d scratch your head a little bit and say, okay, is that…really addressed, or are we just moving right along because we’re a little bit blind

With new security managers, the lack of experience was portrayed as typically resulting in a lack of confidence that leads to security reports being submitted in an automated fashion instead of having the findings vetted for fear of the team missing a false positive.

A more experienced female participant stated that she disagreed with her team supervisor on what should be included in the security report. She argued that false positives should be removed to make the security report as accurate as possible and to avoid the false positives causing confusion in the final report. As she explained, “Initially, the person…whom I was reporting to was not as experienced.” This lack of experience prompted the supervisor to require the team to report everything to the client because she was not adequately experienced to be confident that all the false positives were being properly vetted before being removed from the report. The same participant later indicated that a new, more experienced supervisor came in, and the team began vetting the findings and generating more accurate security reports.

Another experienced female participant was hired as a security manager for an organization. During the interview sessions, she was advised that her department would be audited in two months after she started; just two weeks into her new role, the participant received an email from the security auditor advising her that he would be there the next day. In this instance, the participant described going into the security audit completely unprepared and with very limited knowledge of the organization on which she was being audited. During the security audit, she was joined by her chief information security officer (CISO) who was also not knowledgeable in the areas being audited. As the participant recalled,

We were noncompliant in so many areas; I was two weeks [in], and I didn’t know who to go to. [Even] my CISO [claimed that] he didn’t

know who to go to[stating]…the manager who was here before you did all that.

The participant described answering to the best of her ability with the limited knowledge she had and not directly answering other questions, with the CISO seated behind the assessor shaking his head yes or no depending on the question. Although this situation was not directly related to the altering of security reports, it demonstrated the types of inexperience that the security professionals must deal with, which affect the accuracy of the reports.

Similar to the previous situation, several participants mentioned working with stakeholders to gather the information they needed to generate the security reports. When asked if the stakeholders ever provided false information, an inexperienced female participant advised that such cases frequently occurred yet proved difficult to catch:

You can tell when they don’t give you the [entire] story; instead they just offer a small piece of the pie and try to evade.

When asked to provide an example of this evasion, the participant cited a client who was queried about a very specific encryption setting:

I remember [an incident when] we, as an audit team, became convinced that it was not [encrypted], and they were not showing us and not really understanding…[that] we needed to see a specific functionality; it’s a screen that’s a check box [indicating whether]

it’s encrypted or not. They sort of avoided showing us that screen.

Many of the participants described similar issues and confirmed the difficulty in catching the stakeholders providing false information. This evasion and false information have a direct effect

on the accuracy of security reports. The participants repeatedly depicted these stakeholders’

actions as being due to a perception of the security teams, which leads them to fear security.