Session 190 PD, Model Risk Management and Controls Moderator: Chad R. Runchey, FSA, MAAA

66  Download (0)

Full text

(1)

Session 190 PD, Model Risk Management and Controls

Moderator:

Chad R. Runchey, FSA, MAAA

Presenters:

Michael N. Failor, ASA, MAAA

Michael A. McDonald, FSA, FCIA

(2)

System Access and Change Control

SOA 2014 Annual Meeting

(3)

Agenda

Systems vs. Models

Reasons for System Access and Change Control Change Control Components

Leading Practices

Systems Development Life Cycle Spreadsheet Specific Issues

(4)

System Access and Change Control

Systems vs. Models

Reasons for System Access and Change Control Change Control Components

Leading Practices

Systems Development Life Cycle Spreadsheet Specific Issues

(5)

Systems vs. Models

 Characteristics of Actuarial Systems

 Systems often contain a myriad of options and settings in which to build and run actuarial models

 Systems provide the underlying code (actuarial formulas) needed to run projection models

– “Open systems” provide users the ability to modify system code / formulas – “Closed systems” prevent user-level changes to underlying system code

 System coding errors can affect every model built on the system (similar to DNA coding errors)

(6)

Systems vs. Models (cont.)

 Characteristics of Actuarial Models

 Often run on projection systems

 Defined by selecting options, settings and inputs accommodated by the underlying projection system (similar to genes in DNA)

 Models may be independent of each other

(7)

System Access and Change Control

Systems vs. Models

Reasons for System Access and Change Control

Change Control Components Leading Practices

Systems Development Life Cycle Spreadsheet Specific Issues

(8)

Reasons for System Access and Change Control

 Modeling Demands (“givens”)

 Models will continue to evolve

 Models must be quickly modified to remain viable

 Model flexibility cannot be at the expense of controls

(9)

Reasons for System Access and Change Control (cont.)

 Modeling Demands → Increased Systems Development

More code development in more applications

– Vender provided “open code” actuarial projection systems – Use of Excel (VBA)

– SQL interfaces – C++

– SAS

Some major drivers

– Stochastic modeling – Solvency II

– VM-20

– System consolidations – Product creativity

(10)

Actuarial Modeling Controls:

A Survey of Actuarial Modeling Controls

in the Context of a Model-Based Valuation

Framework

(December 2012)

Reasons for System Access and Change Control (cont.)

 Sponsored by the SOA

 Research and Analysis by Deloitte

 SCOR Participated in the Project Oversight Group

 Compared Current State of Controls Against

Those Expected To Be in Place for Model-Based Valuation (MBV) Approaches

(11)

Modeling Governance Theme Score Current State Synopsis Governance Standards 3 While many companies employ a variety of model governance policies, few companies have a holistic, formal and documented model governance structure. General Modeling Process 3 Many companies have multiple models and modeling platforms and few companies incorporate a model steward role in the modeling processes. System Access and Change Control 4 Model changes are not generally governed by a formal change process. Model Assumption Management 3 Assumptions are regularly reviewed and updated, but with few controls in place to ensure assumptions are approved and input appropriately. Model Input Management 2 Many companies use automated feeds from admin systems for model inputs of liabilities. Other model inputs are often less automated.

Model Output Management 2 Model output used for financial reporting purposes is generally well controlled, while model output for analysis and other purposes is generally less controlled.

On a scale of 1 to 5 (1 being the best), system change control received a “4” – the worst score of all the categories. Note that system coding changes fall under this category.

(12)

Reasons for System Access and Change Control (cont.)

 Majority of the survey respondents had no formal change control process

 Under system access and change control, the SOA sponsored report

recommends two key next steps to move from the current state to leading practices:

 “Implement a formal change management process for governing model code changes and model updates”

 “Determine the calendar for internal model releases to ensure consistency of the model of record across the organization”

(13)

System Access and Change Control

Systems vs. Models

Reasons for System Access and Change Control

Change Control Components

Leading Practices

Systems Development Life Cycle Spreadsheet Specific Issues

(14)

Change Control Components

 Code Changes Are Often Performed by Actuarial Staff

 IT involvement beneficial

 Adoption and modification of existing IT control processes

 Code Comparison Tools

 May be provided in modeling system software

 Third party code comparison tools work with VBA (Excel and MS-Access)

 Code Changes Should Be Peer Reviewed

 Rigor varies from company to company

 SOA 2013 Valuation Actuary Symposium, Session 14 (Reviewing and Validating Actuarial Models), “Systems Peer Review” presentation

(15)

Change Control Components (cont.)

 Model Steward

 Must have top-down support

 Roles vary from organization to organization

 Secures production models

 Archives models

– When changes are made – Periodic scheduled archival

 Applies version controls

 Assures that procedures have been properly followed

(16)

Change Control Components (cont.)

 Clearly Defined System Access Controls

 Levels

– Run model – Read only

– Change model inputs – Change code

 Locations

– Production grids – Testing servers

 Models and Systems

– Actuarial projection systems – Specific models

– Spreadsheets – Data

(17)

Change Control Components (cont.)

 Test Beds (or Test Packs)

 Need to be kept up to date

 Test beds have their limitations

 Management / Departmental Approvals

 Valuation

 Pricing

 Risk Management

 Others

 Communication and Documentation

 Start to finish

 High-level descriptions down to comments in the code

 Archived and linked to: – Production models – Test results

(18)

Change Control Components (cont.)

 Different Modeling Environments

 Centralized models

– Model of record is maintained

– Simultaneous changes by different coders

– Requires a formal code aggregation process

– Integration testing

 Non-Centralized models

– No single model of record

– Pricing models vs. valuation models

– Potentially divergent modeling results

(19)

System Access and Change Control

Systems vs. Models

Reasons for System Access and Change Control Change Control Components

Leading Practices

Systems Development Life Cycle Spreadsheet Specific Issues

(20)

Leading Practices*

 Typically Have Four Different Steps in the Control Process

1. Establish procedure to identify and prioritize model changes (change request process)

2. Evaluate coding changes in a test environment and analyze impact on financial results

3. Perform additional testing on the model code changes (depending on nature of model change)

– Regression testing – Sensitivity testing – Peer reviews

4. Document and seek formal approval

– Document changes and rationale, results and discussion, any sensitivity tests – Appropriate parties review documentation and either approve or deny release – If approved, schedule system release for production use (try to coordinate

(21)

Leading Practices* (cont.)

 Modeling Teams Responsible for Managing Change Requests

 Determine change request procedures

 Maintain and publish logs of all change requests

 Prioritize change requests

 IT Involvement in the Code Change Process

 For open code systems and centralized models

(22)

System Access and Change Control

Systems vs. Models

Reasons for System Access and Change Control Change Control Components

Leading Practices

Systems Development Life Cycle

(23)

Systems Development Life Cycle

 What Is a Systems Development Life Cycle (SDLC)?

 Clearly defined process for planning, creating, testing, and deploying systems and systems modifications.

 SDLC Benefits

 Risk reduction

 Well defined roles and responsibilities

 Improved communication and documentation

 Auditability

 Are Actuaries Employing SDLC Methods?

(24)

Systems Development Life Cycle (cont.)

 Change Request Log

 Vital for any SDLC process

 Formalized documentation of requested system changes – Reasons for the change?

– Is it a system correction or a system enhancement? – Date when change needs to be in production?

– Consequences if change is not made?

– Other relevant details (i.e., change request ID, date, etc.)

 Each change request is assigned a priority level based on the above

(25)

Regression T est BUILD Unit Test Technical Specification Functional and

System Design System Test

Business

Requirements User Acceptance Test

(26)

Systems Development Life Cycle – ‘V’ Methodology* (cont.)

 Business Requirements (BR)

 Produced by the actuaries requesting the change

 Details the required changes

 Includes calculation methodologies (i.e., reserve requirements)

 Functional and System Design

 Describes model design and structure as a solution to the BR

 Can be a source for an actuarial programmer to go directly to code

 Technical Specification

 Documentation at a more detailed “coding” level

(27)

Systems Development Life Cycle – ‘V’ Methodology* (cont.)

 Unit Testing

 Ties back to technical specification

 Low-level testing of code segments in isolation

 System Testing

 Ties back to functional and system design

 Testing on a single policy or model point

 Scenario testing

 End-to-end testing

 Stress testing

 User Acceptance Testing (UAT)

 Ties back to business requirements

 Run in production mode

 Checks for reasonableness of results

 Regression Testing

(28)

System Access and Change Control

Systems vs. Models

Reasons for System Access and Change Control Change Control Components

Leading Practices

Systems Development Life Cycle

(29)

Spreadsheet Specific Issues

 At the Far End of the “Flexibility vs. Control” Paradigm

 Reduce the Number of Spreadsheets

 Periodic Archival

 Inventory and Document Workbook Components

 VBA / SQL code

 How to run VBA / macros and use spreadsheet

 Input data sources and links

 Description of outputs

 Microsoft add-ins

– Random number generators – @Risk

(30)

Spreadsheet Specific Issues (cont.)

 Apply Workbook Access Controls

 Need to lock down models

 Limit directory access

 Protect Workbooks

 Apply Change Controls

 Workbook versioning

 VBA

– Code management methods apply – Third party code comparison tools

(31)

Spreadsheet Specific Issues (cont.)

 Additional Testing Concerns

Workbook testing should be Excel version specific

VBA testing may be affected by Excel session settings

 Common Risks

 Difficult to analyze large complex spreadsheets

 Key person risk

(32)
(33)

Model Risk Management and

Controls

Model Risk Management

Chad Runchey, FSA MAAA

Ernst & Young LLP

(34)

Model risk management

Background

Drivers of model risk

Model risk management framework

(35)

What is model risk?

… possible adverse consequences (including financial loss) of

decisions based on models that are incorrect or misused. – Federal

Reserve Supervisory Letter 11-7

Can have many causes:

• Design errors

• Inappropriate or incorrect use • Model calculation errors

• Poor data

• Misunderstood or poorly communicated assumptions or results

Increases with greater model complexity, input/assumption

(36)

What is driving the increased

focus?

More model-driven financial reporting and

analysis

Regulatory environment

• Solvency II – internal capital models

• Federal Reserve – comprehensive scope

• NAIC – limited requirements

(37)

Model risk management framework

Pr oc es s & C on tr ol s Enterprise Governance M ode l G ov er na nc e Model Lifecycle

Model Review Components

1st Line of Defense

(Model Owners) 2

nd Line of Defense (Model Governance & Validation)

.

Roles and responsibilities

• 1stline: Owns model

lifecycle and related activities

• 2ndline: Establishes

model risk management framework and

standards, provides review and challenge, and can perform independent model testing

• 3rdline: Performs

independent testing and verification of 1st

and 2ndline activities

Co nc ep tua l S oundne ss Da ta & IT In fr as tr uc tu re Model Development Model Implementation Model Operation Performance Monitoring Model Change Management

Business Purpose Inde pe nde nt T es ting

(38)

Model risk policy

A policy formalizes the framework and clearly

states the roles and responsibilities of each

group within the organization.

Other key components of a policy include:

• Model definition

• Risk-rating approach

• Documentation standards

(39)

First line of defense: Model owners

• Business purpose

• Model development and implementation

• Ensure design, theory and logic underlying model is supportable and documented

• Ensure that model components work as intended, are appropriate for intended business purpose, and are conceptually sound and mathematically and statistically correct

• Compare with alternative theories and approaches

• Demonstrate data is suitable for model and consistent with theory behind approach • Test model’s accuracy, robustness and stability, assessing potential limitations and

documenting results

• Model operation

• Performance monitoring

• Assess performance over time under various conditions and as model applications change • Gather insights from observation of outcomes and incorporate into feedback loop

(40)

Second line of defense:

Independent risk management

• Model Governance

• Assess model access rules

• Assess change management protocols

• Conceptual Soundness

• Ensure model methodology is theoretically correct and fit for purpose • Ensure applicability of assumptions and other inputs

• Data & IT Infrastructure

• Assess applicability, accuracy, completeness and availability of all input data, including data mapping and transformation procedures

• Assess the sufficiency of the IT infrastructure

• Process and Controls

(41)

Common challenges

Governance and policy

Model definition

Model validation

• Management expectations

• Independence

• Sustainability

Model documentation

(42)

Contact information

Chad Runchey

(212) 773-1015

(43)

Session 190 PD, Model Risk Management and

Controls

Michael McDonald

SOA 2014 Annual Meeting

(44)

Overview

Model Validation

Model Controls

Current Trends in Valuation Transformations

Useful Links

(45)

Overview

Model Validation

Model Controls

Current Trends in Valuation Transformations

Useful Links

(46)

Model Validation

Why is it needed?

Reserve and capital valuations are moving to model-based

approaches

Complexity of models has increased likelihood of material

modeling errors

Increased scrutiny from regulators and senior management

Basis for Actuarial analysis of inforce exposure

(47)

Modeling methodology

Calculations

Model Validation

Model Review

(48)

Model Validation

Model Review

Identify model setup deficiencies vs. best practices

Consistency

Transparency

Organization

• Intuitive object/variable names

• Hard coded values are stored in central location

Well documented

(49)

Model Validation

Model Vetting

Validate Model results

Vetting tools should be independent of model

Build simplified calculator

Compare results of simplified scenario for reasonability

Build robust calculator for interim calculations

Projected mortality calculation

Projected NAAR calculation

Dynamic lapse assumption

(50)

Model Validation

Sensitivity Analysis

Review against pricing model and/or historical

sensitivities

SENSITIVITY RECONCILIATION Pricing Sensitivity x Differences Inforce Distribution a Inforce Aging b Actuarial Assumptions c Economic Assumptions d

(51)

Model Validation

Sensitivity Analysis

Review against pricing model and/or historical

sensitivities

SENSITIVITY RECONCILIATION Pricing Sensitivity x Differences Inforce Distribution a Inforce Aging b Actuarial Assumptions c Economic Assumptions d Unknown Differences e

(52)

Model Validation

Case Testing

Roll forward inforce – compare reserves vs baseline

projected

time

Extract aged for 5 years Inforce Aging Process

(53)

Model Validation

Case Testing

Stress-test dynamic assumptions

Single policy – unique attributes

(54)

Overview

Model Validation

Model Controls

Current Trends in Valuation Transformations

Useful Links

(55)

Model Controls

Documentation

Model User manual/Documentation manual

Internal model documentation

Issue log

Version Controls

Linear development

Standardized model naming

Model comparison tools

(56)

Model Controls

Model Development Validation

Change summary

Testing plan with expected impact

Change management checklist

Peer review

Impact quantification

(57)

Model Controls

Inclusion control

User access

Important for both production and development

environment

Consolidate calculations into one model

Eliminate spreadsheet models!

Too flexible

(58)

Overview

Model Validation

Model Controls

Current Trends in Valuation Transformations

Useful Links

(59)

Valuation Transformation

Legacy Design

Extracts and assumptions reside in business function

area

Model output consolidated in back-end spreadsheets

Next Generation Design

All model assumptions reside in external databases (pricing

and valuation)

Models reference tables populated with admin data through

(60)

ADMIN  EXPERIENCE  MONITORING Policy Data Queries Analysis PRICING Model Hypothetical  Policy Data Assumptions Product features Actuarial VALUATION Model Policy Data Assumptions Product features Actuarial Feedback Feedback FINANCE PRICING TEAM

Extract Extract Extract Extract

(61)

Historic  Versioning FINANCIAL REPORTING SYSTEM • Consolidates, organizes,  aggregates financial results DATA WAREHOUSE  Policy  Info Claims/Pre miums/Exp enses Pricing  Distribution ADMIN  FINANCE ASSUMPTION DATABASE Product Features Actuarial  Assumptions Scenario s EXPERIENCE  MONITORING Queries Analysis PRICING Model VALUATION Model PRICING TEAM

Next Generation Illustration

(62)

Next Generation Illustration

FINANCIAL REPORTING SYSTEM DATA WAREHOUSE  Policy  Info Claims/Prem iums/Expens es Pricing  Distribution ASSUMPTION DATABASE Product Features Actuarial  Assumptions Scenarios

(63)

Valuation Transformation

Robust automated controls and processes

Actuarial staff can focus time on analyzing results

Non modeling staff can review model assumptions

More granular model output available for analysis (ie.

seriatim projections)

Financial statement info from single source

Increased transparency

(64)

Overview

Model Validation

Model Controls

Current Trends in Valuation Transformations

Useful Links

(65)

Useful Links

SOA Survey – Model Control

http://www.soa.org/Research/Research-Projects/Life-Insurance/Actuarial-Modeling-Control.aspx

EU Study – Model Processes and Controls

http://www.google.com/url?sa=t&rct=j&q=&esrc=s&frm=1&source=web&cd=6 &ved=0CDsQFjAF&url=http%3A%2F%2Fwww.actuaries.org.uk%2Fsystem%2F files%2Fdocuments%2Fpdf%2Factuarialprocessescontrolswplife200905rpt.pdf&ei =qpPzU_C7GrjLsQS29IDIAg&usg=AFQjCNHhVm8nlNxsrg5Hyjzwod0XTUS Sjw&sig2=mWsOJGTI1aq6un5CWkE43w

(66)

Figure

Updating...

References

Related subjects :