Study of Security Awareness Training

11  Download (0)

Full text


Study of Security Awareness Training

Steve Kruse,Security Principal @ RSA

Bill Pankey, Consultant @Tunitas Group





Innovation Norway Feb 04, 2010


l t d

f i f

t &


CSI / FBI Computer Crime Survey

Annual study of infosec events & security

practice in US

Estimates losses from computer crime

Type and frequency of exploits

Survey of security practice / response to threats

Commonly used baseline for

Risk assessments


% f


b d


2007 CSI Finding: Minimal UAT* Budgets

% of security budget spent on UAT

48% 22%


2007 Security budgets average between 3-5% of total IT budget

*UAT: Security Awareness Training



i d



Counter Indications

Convention wisdom among US security

practitioners is that end users are the

greatest security threat:

85% of security breaches involve end users ~ Forester

End user web surfing primary source of network infection ~ Symantec

Email borne compromise of end user accounts has been has been the initial entry vector for 100% of ‘advanced persistent threats’(APT) ~ Mandiant



I th





i l



The Paradox

“You can’t service pack the end user” - Mike Nash, Microsoft

Is there an inverse Pareto principle at work:

spend the least on the biggest problem?

High recognition of end user involvement with

security breaches

Relatively low spending on user security

awareness training awareness training

Is the security industry guilty of ‘risk

management’ malpractice?

Study resolved to understand / explain

this apparent paradox

57 ti

2009 Survey of Security Awareness & Training

57 questions

UAT budgeting & rationale

UAT metrics & accountability

UAT practices

Respondents recruited from:

UseNet / yahoo UAT interest groups

Metricon audience


No sampling control

Suspect bias in self-report survey results:

More, rather than less interest in UAT than among typical security practitioners

Natural tendency to over-estimate the quality of their security program


Survey sample places only slightly greater emphasis on UAT than CSI study

Sample validates CSI Finding

emphasis on UAT than CSI study

55% less than 2% of security budget

13% greater than 8% of security budget

Difference are either not significant or indicate a

very slight increased emphasis on UAT

True cost of UAT typically masked:

Security budgets typically are not burdened with the cost of

Security UAT budget in context

Security budgets typically are not burdened with the cost of the users’ time spent on UAT

Only 1 respondent reported that UAT training time is charged against security budget

Total cost of user training time is likely to be the most costly component of UAT

93% of respondent companies require UAT for all93% of respondent companies require UAT for all employees at hire; 75% annual refresher training


Security program’s UAT expenditure is either

very efficient or very wasteful of company

Security UAT budget in context (2)

very efficient or very wasteful of company

(human) resources

UAT budget includes preparation, delivery and

management costs for a significant expenditure of corporate resources

Normalized UAT management cost metric: Normalized UAT management cost metric:

security program UAT budget / total # of user training hours * loaded labor rate

Measure of ‘efficiency’ ? or ‘inattention’?

I th


l d

t ti


ti i

t d

Questions about the rationale for UAT?

Is there formal documentation of anticipated

UAT benefits

? If so, where?

Security plan: 65%

Security policy: 42% (pro-forma?)

UAT business case: 35%

Individual campaign proposal: 5%

But, often there is no accountability

for benefit statements:

60% of respondents report no management review or approval of the above


Wh t i

d t j

tif th


t f

UAT Rationale

What is used to justify the commitment of

users’ time to UAT

Regulatory requirement: 73%

Unspecified ‘security benefit’: 65%

External Auditor report: 45%

Expected increase in user accountability: 15%

Expected increase in user accountability: 15%

Basis for rejection of requests for increase in

mandated UAT (100 % of respondents)?

Unspecified management priorities: 70%

Weak business case: 16%

Few companies tracked meaningful

Lack of UAT Effectiveness | Performance Measurement



measures of UAT performance

Training completion / compliance rate: 100%

(User) Behavioral \ attitude measures: 13%

Correlation w/ security incident metrics: 7%

Management satisfaction determined by:

CSO \ CISO: 67%

CIO: 25%


60% l i

f UAT t

Illusory UAT Management

“you can’t manage what you don’t measure” Drucker

60% claim success of UAT to

Reduce security incidents

Address root causes of security breaches

Increase compliance with security policy

While focusing on cost

While focusing on cost,

Rate of training completion

But Avoiding collection of UAT performance /

effectiveness metrics

Focus on activity rather than benefit realization.

Few Meaningful UAT Metrics


Security Metrics, Jacquith

• % of staff completing security awareness training? (6)

Correlation of tailgating rates w/ training latency

Metrics for IT Service Management, ITSM

• % of staff not at optimal training level? (10)

Poll of callers to service deskPoll of callers to service desk

SP800-55 Performance Measurement Guide for

Information Security, NIST

• % of employees in security roles receiving specialized security training (1)


Common UAT metrics available to establish industry

Do UAT metrics obscure the security objective?

baselines may ‘miss the point’

Implicitly assume the effectiveness of

training, ie, the results of management

– W/o the relevance, credibility & appropriateness of the training, the ‘completion rate’ indicates nothing about the security value of the UAT program.

Currently these measures are primarily cost

metrics reflecting the scale of resource (end

user time) consumption

Security role of the end user is specific to company and its security strategy Is the user a:

Meaningful UAT Metrics imply Strategy

its security strategy. Is the user, a:

– Threat to be engineered around?

– An actor whose behavior needs the constraint of policy? – A source of detective or corrective control?

– All of the above? Some of the above?

The view of user determines the objectives of UAT j

(improve user performance in security role)

– What and how much risk awareness – Will vary with industry and company culture


Effectiveness of UAT has to be measured

against expectations of security strategy

Standardization of Security Awareness

against expectations of security strategy

Some user roles are expected to have more than

less general security / risk awareness

Some use roles expected to take specific action

in response to events

Standardization of user expectation facilitates

development of appropriate metrics and

establishing meaningful industry baselines

• Blissfully unaware

– Little recognition or acceptance of most information security threats At thi l l l t i i th t i f ti it i t f IT t

Maturity Model for User Security Awareness

– At this level, prevalent view is that information security is a property of IT systems and largely a matter of architecture and configuration. Security largely independnet of user behavior.

• Consciously incompetent

– Some recognition that there is a information security threat, but: • Poor risk assessment skill and intuition

• Uncertain of action needed to protect company information assets. Will do nothing rather than create further harm

• Compliant

– Aware of risks identified in company policy – Aware of risks identified in company policy

• Will take action identified in company security policy • Risk aware

– Considers information security risk in performance of company duties, but • Unsure of appropriate action; sometime will report incidents

• Competent & Practiced


Blissfully unaware

User: ‘heads down’ routine data processing roles

UAT Goals re Targeted Maturity Level



User: Industry w/ little discretionary access control (e.g. banking). Users ‘locked down’ by restrictive policy.

UAT: Policy training

Metric: Correlate policy violations with training latency

Metric: Correlate policy violations with training latency

Risk aware

User: Industry w/ significant discretionary access (e.g. health)

UAT: Policy training + Threat identification

Metric: # of anomalies reported by users


End user sees what could be a company owned


End user sees what could be a company owned laptop in an unlocked car in the facility parking lot. What is the end user expected to do?

Blissfully unaware: Where the company security tolerates

the ‘unaware user’, e.g. where whole disk encryption has been implemented for all company laptops ~ p p y p p nothingg

Compliant: Where company policy prescribes all security

obligations ~ only what is described in policy

Risk aware: Where company security model depends upon


Little accountability for UAT beyond compliance with regulatory mandates


regulatory mandates

Few, if any performance metrics

Focus on cost

Where there is no accountability, the optimal strategy is to reduce the absolute UAT expenditure

UAT management requires new UAT performance metrics

Correlate UAT with specific security benefits

Lack of industry UAT performance benchmarks




Related subjects :