Session 190 PD, Model Risk Management and Controls Moderator: Chad R. Runchey, FSA, MAAA

(1)

Session 190 PD, Model Risk Management and Controls

Moderator:

Chad R. Runchey, FSA, MAAA

Presenters:

Michael N. Failor, ASA, MAAA

Michael A. McDonald, FSA, FCIA

(2)

System Access and Change Control

SOA 2014 Annual Meeting

(3)

Agenda

Systems vs. Models

Reasons for System Access and Change Control Change Control Components

Leading Practices

Systems Development Life Cycle Spreadsheet Specific Issues

(4)

System Access and Change Control

(5)

Systems vs. Models

 Characteristics of Actuarial Systems

 Systems often contain a myriad of options and settings in which to build and run actuarial models

 Systems provide the underlying code (actuarial formulas) needed to run projection models

– “Open systems” provide users the ability to modify system code / formulas – “Closed systems” prevent user-level changes to underlying system code

 System coding errors can affect every model built on the system (similar to DNA coding errors)

(6)

Systems vs. Models (cont.)

 Characteristics of Actuarial Models

 Often run on projection systems

 Defined by selecting options, settings and inputs accommodated by the underlying projection system (similar to genes in DNA)

 Models may be independent of each other

(7)

System Access and Change Control

Reasons for System Access and Change Control

Change Control Components Leading Practices

(8)

Reasons for System Access and Change Control

 Modeling Demands (“givens”)

 Models will continue to evolve

 Models must be quickly modified to remain viable

 Model flexibility cannot be at the expense of controls

(9)

Reasons for System Access and Change Control (cont.)

 Modeling Demands → Increased Systems Development

More code development in more applications

– Vender provided “open code” actuarial projection systems – Use of Excel (VBA)

– SQL interfaces – C++

– SAS

Some major drivers

– Stochastic modeling – Solvency II

– VM-20

– System consolidations – Product creativity

(10)

Actuarial Modeling Controls:

A Survey of Actuarial Modeling Controls

in the Context of a Model-Based Valuation

Framework

(December 2012)

Reasons for System Access and Change Control (cont.)

 Sponsored by the SOA

 Research and Analysis by Deloitte

 SCOR Participated in the Project Oversight Group

 Compared Current State of Controls Against

Those Expected To Be in Place for Model-Based Valuation (MBV) Approaches

(11)

Modeling Governance Theme Score Current State Synopsis Governance Standards 3 While many companies employ a variety of model governance policies, few companies have a holistic, formal and documented model governance structure. General Modeling Process 3 Many companies have multiple models and modeling platforms and few companies incorporate a model steward role in the modeling processes. System Access and Change Control 4 Model changes are not generally governed by a formal change process. Model Assumption Management 3 Assumptions are regularly reviewed and updated, but with few controls in place to ensure assumptions are approved and input appropriately. Model Input Management 2 Many companies use automated feeds from admin systems for model inputs of liabilities. Other model inputs are often less automated.

Model Output Management 2 Model output used for financial reporting purposes is generally well controlled, while model output for analysis and other purposes is generally less controlled.

On a scale of 1 to 5 (1 being the best), system change control received a “4” – the worst score of all the categories. Note that system coding changes fall under this category.

(12)

Reasons for System Access and Change Control (cont.)

 Majority of the survey respondents had no formal change control process

 Under system access and change control, the SOA sponsored report

recommends two key next steps to move from the current state to leading practices:

 “Implement a formal change management process for governing model code changes and model updates”

 “Determine the calendar for internal model releases to ensure consistency of the model of record across the organization”

(13)

System Access and Change Control

Reasons for System Access and Change Control

Change Control Components

(14)

Change Control Components

 Code Changes Are Often Performed by Actuarial Staff

 IT involvement beneficial

 Adoption and modification of existing IT control processes

 Code Comparison Tools

 May be provided in modeling system software

 Third party code comparison tools work with VBA (Excel and MS-Access)

 Code Changes Should Be Peer Reviewed

 Rigor varies from company to company

 SOA 2013 Valuation Actuary Symposium, Session 14 (Reviewing and Validating Actuarial Models), “Systems Peer Review” presentation

(15)

Change Control Components (cont.)

 Model Steward

 Must have top-down support

 Roles vary from organization to organization

 Secures production models

 Archives models

– When changes are made – Periodic scheduled archival

 Applies version controls

 Assures that procedures have been properly followed

(16)

Change Control Components (cont.)

 Clearly Defined System Access Controls

 Levels

– Run model – Read only

– Change model inputs – Change code

 Locations

– Production grids – Testing servers

 Models and Systems

– Actuarial projection systems – Specific models

– Spreadsheets – Data

(17)

Change Control Components (cont.)

 Test Beds (or Test Packs)

 Need to be kept up to date

 Test beds have their limitations

 Management / Departmental Approvals

 Valuation

 Pricing

 Risk Management

 Others

 Communication and Documentation

 Start to finish

 High-level descriptions down to comments in the code

 Archived and linked to: – Production models – Test results

(18)

Change Control Components (cont.)

 Different Modeling Environments

 Centralized models

– Model of record is maintained

– Simultaneous changes by different coders

– Requires a formal code aggregation process

– Integration testing

 Non-Centralized models

– No single model of record

– Pricing models vs. valuation models

– Potentially divergent modeling results

(19)

System Access and Change Control

(20)

Leading Practices*

 Typically Have Four Different Steps in the Control Process

1. Establish procedure to identify and prioritize model changes (change request process)

2. Evaluate coding changes in a test environment and analyze impact on financial results

3. Perform additional testing on the model code changes (depending on nature of model change)

– Regression testing – Sensitivity testing – Peer reviews

4. Document and seek formal approval

– Document changes and rationale, results and discussion, any sensitivity tests – Appropriate parties review documentation and either approve or deny release – If approved, schedule system release for production use (try to coordinate

(21)

Leading Practices* (cont.)

 Modeling Teams Responsible for Managing Change Requests

 Determine change request procedures

 Maintain and publish logs of all change requests

 Prioritize change requests

 IT Involvement in the Code Change Process

 For open code systems and centralized models

(22)

System Access and Change Control

Systems Development Life Cycle

(23)

Systems Development Life Cycle

 What Is a Systems Development Life Cycle (SDLC)?

 Clearly defined process for planning, creating, testing, and deploying systems and systems modifications.

 SDLC Benefits

 Risk reduction

 Well defined roles and responsibilities

 Improved communication and documentation

 Auditability

 Are Actuaries Employing SDLC Methods?

(24)

Systems Development Life Cycle (cont.)

 Change Request Log

 Vital for any SDLC process

 Formalized documentation of requested system changes – Reasons for the change?

– Is it a system correction or a system enhancement? – Date when change needs to be in production?

– Consequences if change is not made?

– Other relevant details (i.e., change request ID, date, etc.)

 Each change request is assigned a priority level based on the above

(25)

Regression T est BUILD Unit Test Technical Specification Functional and

System Design System Test

Business

Requirements User Acceptance _Test

(26)

Systems Development Life Cycle – ‘V’ Methodology* (cont.)

 Business Requirements (BR)

 Produced by the actuaries requesting the change

 Details the required changes

 Includes calculation methodologies (i.e., reserve requirements)

 Functional and System Design

 Describes model design and structure as a solution to the BR

 Can be a source for an actuarial programmer to go directly to code

 Technical Specification

 Documentation at a more detailed “coding” level

(27)

Systems Development Life Cycle – ‘V’ Methodology* (cont.)

 Unit Testing

 Ties back to technical specification

 Low-level testing of code segments in isolation

 System Testing

 Ties back to functional and system design

 Testing on a single policy or model point

 Scenario testing

 End-to-end testing

 Stress testing

 User Acceptance Testing (UAT)

 Ties back to business requirements

 Run in production mode

 Checks for reasonableness of results

 Regression Testing

(28)

System Access and Change Control

Systems Development Life Cycle

(29)

Spreadsheet Specific Issues

 At the Far End of the “Flexibility vs. Control” Paradigm

 Reduce the Number of Spreadsheets

 Periodic Archival

 Inventory and Document Workbook Components

 VBA / SQL code

 How to run VBA / macros and use spreadsheet

 Input data sources and links

 Description of outputs

 Microsoft add-ins

– Random number generators – @Risk

(30)

Spreadsheet Specific Issues (cont.)

 Apply Workbook Access Controls

 Need to lock down models

 Limit directory access

 Protect Workbooks

 Apply Change Controls

 Workbook versioning

 VBA

– Code management methods apply – Third party code comparison tools

(31)

Spreadsheet Specific Issues (cont.)

 Additional Testing Concerns

 Workbook testing should be Excel version specific

 VBA testing may be affected by Excel session settings

 Common Risks

 Difficult to analyze large complex spreadsheets

 Key person risk

(32)

(33)

Model Risk Management and

Controls

Model Risk Management

Chad Runchey, FSA MAAA

Ernst & Young LLP

(34)

Model risk management

• Background

• Drivers of model risk

• Model risk management framework

(35)

What is model risk?

… possible adverse consequences (including financial loss) of

decisions based on models that are incorrect or misused. – Federal

Reserve Supervisory Letter 11-7

• Can have many causes:

• Design errors

• Inappropriate or incorrect use • Model calculation errors

• Poor data

• Misunderstood or poorly communicated assumptions or results

• Increases with greater model complexity, input/assumption

(36)

What is driving the increased

focus?

• More model-driven financial reporting and

analysis

• Regulatory environment

• Solvency II – internal capital models

• Federal Reserve – comprehensive scope

• NAIC – limited requirements

(37)

Model risk management framework

Pr oc es s & C on tr ol s Enterprise Governance M ode l G ov er na nc e Model Lifecycle

Model Review Components

1st _{Line of Defense}

(Model Owners) 2

nd _{Line of Defense} (Model Governance & Validation)

.

Roles and responsibilities

• 1stline: Owns model

lifecycle and related activities

• 2ndline: Establishes

model risk management framework and

standards, provides review and challenge, and can perform independent model testing

• 3rdline: Performs

independent testing and verification of 1st

and 2nd_{line activities}

Co nc ep tua l S oundne ss Da ta & IT In fr as tr uc tu re Model Development Model Implementation Model Operation Performance Monitoring Model Change Management

Business Purpose Inde pe nde nt T es ting

(38)

Model risk policy

• A policy formalizes the framework and clearly

states the roles and responsibilities of each

group within the organization.

• Other key components of a policy include:

• Model definition

• Risk-rating approach

• Documentation standards

(39)

First line of defense: Model owners

• Business purpose

• Model development and implementation

• Ensure design, theory and logic underlying model is supportable and documented

• Ensure that model components work as intended, are appropriate for intended business purpose, and are conceptually sound and mathematically and statistically correct

• Compare with alternative theories and approaches

• Demonstrate data is suitable for model and consistent with theory behind approach • Test model’s accuracy, robustness and stability, assessing potential limitations and

documenting results

• Model operation

• Performance monitoring

• Assess performance over time under various conditions and as model applications change • Gather insights from observation of outcomes and incorporate into feedback loop

(40)

Second line of defense:

Independent risk management

• Model Governance

• Assess model access rules

• Assess change management protocols

• Conceptual Soundness

• Ensure model methodology is theoretically correct and fit for purpose • Ensure applicability of assumptions and other inputs

• Data & IT Infrastructure

• Assess applicability, accuracy, completeness and availability of all input data, including data mapping and transformation procedures

• Assess the sufficiency of the IT infrastructure

• Process and Controls

(41)

Common challenges

• Governance and policy

• Model definition

• Model validation

• Management expectations

• Independence

• Sustainability

• Model documentation

(42)

Contact information

Chad Runchey

(212) 773-1015

(43)

Session 190 PD, Model Risk Management and

Controls

Michael McDonald

SOA 2014 Annual Meeting

(44)

Overview

• Model Validation

• Model Controls

• Current Trends in Valuation Transformations

• Useful Links

(45)

Overview

• _{Model Validation}

• Model Controls

• Current Trends in Valuation Transformations

• Useful Links

(46)

Model Validation

Why is it needed?

• Reserve and capital valuations are moving to model-based

approaches

• Complexity of models has increased likelihood of material

modeling errors

• Increased scrutiny from regulators and senior management

• Basis for Actuarial analysis of inforce exposure

(47)

Modeling methodology

Calculations

Model Validation

Model Review

(48)

Model Validation

Model Review

• Identify model setup deficiencies vs. best practices

• Consistency

• Transparency

• Organization

• Intuitive object/variable names

• Hard coded values are stored in central location

• Well documented

(49)

Model Validation

Model Vetting

• Validate Model results

• Vetting tools should be independent of model

• Build simplified calculator

• Compare results of simplified scenario for reasonability

• Build robust calculator for interim calculations

• Projected mortality calculation

• Projected NAAR calculation

• Dynamic lapse assumption

(50)

Model Validation

Sensitivity Analysis

• Review against pricing model and/or historical

sensitivities

SENSITIVITY RECONCILIATION Pricing Sensitivity _x Differences Inforce Distribution _a Inforce Aging _b Actuarial Assumptions _c Economic Assumptions _d

(51)

Model Validation

Sensitivity Analysis

• Review against pricing model and/or historical

sensitivities

SENSITIVITY RECONCILIATION Pricing Sensitivity _x Differences Inforce Distribution _a Inforce Aging _b Actuarial Assumptions _c Economic Assumptions _d Unknown Differences _e

(52)

Model Validation

Case Testing

• Roll forward inforce – compare reserves vs baseline

projected

time

Extract aged for 5 years Inforce Aging Process

(53)

Model Validation

Case Testing

• Stress-test dynamic assumptions

• Single policy – unique attributes

(54)

Overview

• Model Validation

• _{Model Controls}

• Current Trends in Valuation Transformations

• Useful Links

(55)

Model Controls

• Documentation

• Model User manual/Documentation manual

• Internal model documentation

• Issue log

• Version Controls

• Linear development

• Standardized model naming

• Model comparison tools

(56)

Model Controls

• Model Development Validation

• Change summary

• Testing plan with expected impact

• Change management checklist

• Peer review

• Impact quantification

(57)

Model Controls

• Inclusion control

• User access

• Important for both production and development

environment

• Consolidate calculations into one model

• Eliminate spreadsheet models!

• Too flexible

(58)

Overview

• Model Validation

• Model Controls

• _{Current Trends in Valuation Transformations}

• Useful Links

(59)

Valuation Transformation

• Legacy Design

• Extracts and assumptions reside in business function

area

• Model output consolidated in back-end spreadsheets

• Next Generation Design

• All model assumptions reside in external databases (pricing

and valuation)

• Models reference tables populated with admin data through

(60)

ADMIN EXPERIENCE MONITORING Policy Data Queries Analysis PRICING Model Hypothetical Policy Data Assumptions Product features Actuarial VALUATION Model Policy Data Assumptions Product features Actuarial Feedback Feedback FINANCE PRICING TEAM

Extract Extract Extract Extract

(61)

Historic Versioning FINANCIAL REPORTING SYSTEM • Consolidates, organizes, aggregates financial results DATA WAREHOUSE Policy Info Claims/Pre miums/Exp enses Pricing Distribution ADMIN FINANCE ASSUMPTION DATABASE Product Features Actuarial Assumptions Scenario s EXPERIENCE MONITORING Queries Analysis PRICING Model VALUATION Model PRICING TEAM

Next Generation Illustration

(62)

Next Generation Illustration

FINANCIAL REPORTING SYSTEM DATA WAREHOUSE Policy Info Claims/Prem iums/Expens es Pricing Distribution ASSUMPTION DATABASE Product Features Actuarial Assumptions Scenarios

(63)

Valuation Transformation

• Robust automated controls and processes

• Actuarial staff can focus time on analyzing results

• Non modeling staff can review model assumptions

• More granular model output available for analysis (ie.

seriatim projections)

• Financial statement info from single source

• Increased transparency

(64)

Overview

• Model Validation

• Model Controls

• Current Trends in Valuation Transformations

• _{Useful Links}

(65)

Useful Links

SOA Survey – Model Control

http://www.soa.org/Research/Research-Projects/Life-Insurance/Actuarial-Modeling-Control.aspx

EU Study – Model Processes and Controls

http://www.google.com/url?sa=t&rct=j&q=&esrc=s&frm=1&source=web&cd=6 &ved=0CDsQFjAF&url=http%3A%2F%2Fwww.actuaries.org.uk%2Fsystem%2F files%2Fdocuments%2Fpdf%2Factuarialprocessescontrolswplife200905rpt.pdf&ei =qpPzU_C7GrjLsQS29IDIAg&usg=AFQjCNHhVm8nlNxsrg5Hyjzwod0XTUS Sjw&sig2=mWsOJGTI1aq6un5CWkE43w

(66)