www.pwc.com
Internal Audit Data Analytics
What’s driving the need for enhanced data analytics in the market
Analytics maturity model
• Capabilities are well developed and practiced with appropriate governance
Data so rces are readil
• Scale is achieved for department specific teams
• Continuous improvement methodologies are implemented
• Continuous monitoring occurring for metrics and controls
• Capabilities developed and adopted
• Capabilities used to drive audits
• Defined goals and standardized
• Data sources are readily available
• Activities begin to become repeatable and CM metrics are developed
• Teams are 10% of department
Insight
• Limited capabilities
• Ad hoc activities resulting in unpredictable and inefficient performance
• Defined goals and standardized processes and tools
estment, Effeciency, p
• Success based on individual competence
Defined
Advanced
Leading
e e ed
Drivers and value of analytics within internal audit
The risks highlighted as a result of the financial crisis has required organizations to develop a deeper understanding of the businesses they manage. They cannot simply rely on existing control structures. They must evaluate transactional activities for patterns, trends
d l b h i O f th t ffi i t t d l thi d t di d t t th t ti i b d l i and anomalous behavior. One of the most efficient ways to develop this understanding and test these transactions is by developing sound information management practices and conducting forensic testing.
Drivers Value
What is creating demand for analytics? Why is it so important for organizations to get it right?
What is creating demand for analytics? Why is it so important for organizations to get it right?
• Regulatory expectations to monitor business activities are increasing.
• Deeper business understanding &
focus on risk
Analytics allows specialists to link data to business processes allowing us develop a better understand of the business and associated risks.
• Lower existing and future costs Advanced analytical techniques are more cost efficient
• Data is growing exponentially, while the skill sets to use tools beyond access and excel is decreasing.
• Lower existing and future costs Advanced analytical techniques are more cost efficient than traditional methods. A recent study indicated that an analytical procedure costing $0.01 could cost as much as
$4 to perform similar procedures using traditional testing.
• End-to-end testing Analytics can track data across functions and units
ll i t id tif i i d b t diti l (
• Organizations are being forced to do more with less.
• Competitive pressures are forcing
allowing us to identify issues missed by traditional (more focused) testing.
• Increased population and control coverage
Analytics allows for 100% population testing as compared to traditional sample testing. In addition, analytics allows us to reverse engineer issues identified during testing.
organizations to innovate.
• Key technology, business and data hybrid skills are in high demand.
• Real-time evaluation of controls and data integrity - facilitates “what-if”
analysis
Clients can view actual mistakes in logic or processing errors rather than focus on hypothetical risks – this can accelerate root cause analysis and assist in the implementation of new controls.
3 August 14, 2012
Value of data analytics – Audit execution
Finding Follow Up
Risk Assessment Continuous Monitoring
Analytics within Audits will be focused on techniques to test the operating and design effectiveness of our control environment. The goal of the procedures outlined below will remain focused on allowing IA to give stakeholders recommendations on their control environments.
Area Type Description Examples
Reporting Audit Execution
Area Type Description Examples
Business Analytics
Analytics used to identify business control weaknesses
Profiling Data Mining
Statistically profiling the quantitative and qualitative parameters of a set of data to identify risk areas (anomalies, spikes, unusual patterns) within a business and/or process.
Using analytics to mine payments data to identify systems to focus on and identify patterns of money movements in high risk countries.
Stratified Conduct risk based analysis of data and target portions of Using analytics to choose sample of and recommend
design enhancements
Sampling sampling on higher risk dimensions of business areas. surveillance exceptions based on volume and value of trades, high volatility trading days, traders with the highest level of exceptions.
Logic Recreation/
Recalculation
Utilizing business requirements or independent sources of data to recreate selected portions of the logic of data based control processes (reports, business processes).
Using analytics to re-perform risk calculations performed by vendor systems which aggregate data from multiple sources.
Data Governance Analytics
Analytics used to test IT controls within technology systems
Reconciliations Comparing sets of data between hand off’s and/or independent sources to measure completeness and accuracy.
Using analytics to compare data sent to regulators in reports vs. source systems.
Data Quality Examining completeness, validity, consistency, timeliness and accuracy of enterprise data as it moves from source to reporting.
Using analytics to isolate data quality issues within source systems which may cause downstream report breaks.
technology systems
Value of data analytics – Continuous monitoring
Finding Follow Up
Risk Assessment Continuous Monitoring
Analytics within continuous monitoring will be used to assist in the analysis of identifying risk metrics to be used to quantitatively compare and cross-rough risk in our audit universe. The outline below is a potential framework.
Framework Development • Currently the department engages in a qualitative
process to assess risk as part of our continuous Core Data S
Aggregation, comparison
Metrics and interactive
Reporting Audit Execution
process to assess risk as part of our continuous monitoring program.
• The potential natural evolution of the program is to add a quantitative component that would drive risk analysis and comparison.
• To achieve this a framework for development would
d b d d b il Th f ll i
Core Technology Team
Internal Audit Analytics Team
Sources comparison
and analysis
interactive dashboards
not need to be agreed upon and built. The following is one proposal for a framework.
Pilot Metrics Development Once agreement and management support on a framework is developed analytics intelligence will work
CM Working Group
A proposed framework for continuous monitoring is to develop a technology environment where selected data can be ingested, pre-defined analysis can be conducted and interactive dashboards can be produced with metrics. The dashboards would allow auditors to filter and compare different dimensions of the metrics to gain a higher level or lower level analysis to gain insights providing our team the ability to conduct ‘what if’ and drill down analysis to drive unique conclusions.
OU 1 Trading Desk 1 PWM Office 1
framework is developed analytics intelligence will work with business and technology audit teams to begin piloting and conduct ‘behavioral analysis’ on data within the firm to assist in creating meaningful, impactful, risk based metrics.
Internal Audit Analytics Team
Business and Technology Auditors
CM Working Group
OU 1 OU 2 OU 3
Trading Desk 1 Trading Desk 2 Trading Desk 3
PWM Office 1 PWM Office 2 PWM Office 3
Comparable Entities Aggregation, Comparison and Analysis
Production Roll Out Once a metrics and a process to develop a metric has been approved by the supporting OU owners, IA analytics will working with core technology to put the metric and corresponding dashboards into production for continuous monitoring use.
Core Technology Team
5 August 14, 2012
Bank Regulatory Reporting Liquidity Risk Regulatory Reporting
The following case studies demonstrate practical examples of analytics work performed and the value driven by the analysis.
Bank Regulatory Reporting
Challenge: Report logic specifications were incomplete and required significant data transformation, increasing the complexity of report re-performance and root cause analysis of discrepancies.
Liquidity Risk Regulatory Reporting
Challenge: Determine if any unusual variances exist in two months worth of 4G reports, consisting of 523,000 individual data points spread throughout 40 different
excel spreadsheets. of discrepancies.
Analytics Leveraged: Re-performance Tools: SQL Server with BIDS
Benefit Analytics Leveraged: Visual Analytics, Data Profiling
Tools: Spotfire, SQL Server with BIDS Benefit
L i i l l ti i ifi tl l t d d t Leveraging SQL Server with BIDS enabled a the development of a logical process flow, accelerating logic development and simplifying root cause analysis.
Time Savings*: 80 hours
Assurance: 100% population coverage, 1 day Leveraging visual analytics significantly accelerated data
ingestion and aggregate analysis to identify significant variances over time for sample selection and review.
Time Savings*: 120 hours
Assurance: 100% population coverage, 2 months
Internal Audit analytics maturity model
PWC uses the Maturity Model below to assess Internal Audit Analytics performance and identify priorities for PWC uses the Maturity Model below to assess Internal Audit Analytics performance and identify priorities for improvement opportunities
improvement opportunities
Below industry practices At industry practices
Priorities dictated by surprises Priorities dictated by immediate business needs Clearly defined strategy and roadmap
y
Leading practices
Duplicated & fragmented Unclear roles and responsibilities Small span of control
Centers of Excellence leveraged Integrated coordination & management Roles and responsibilities clearly defined Priorities dictated by surprises
Reactive to business needs
Limited measurement of processes, etc.
Clearly defined strategy and roadmap Metrics in place to track performance Integrated on a global basis
Priorities dictated by immediate business needs Moderate anticipation of analytics needs
Limited measurement of processes, targets not set Organizational structure overlaps
Roles and responsibilities understood but not documented Moderate span of control
Organization/ structureStrateg
Narrow skill base
Limited development; ad-hoc training Limited analytics understanding
Strong analytics mindset
staff rotations & leadership programs Deep technical skills & understanding Narrow skill base
Functional training and development
Uneven expertise across functional audit areas
dit Analytics PeopleOnology Limited integration / no data governance
Manual data collection processes
Automatic data capture Strong data governance Multiple and redundant systems and data feeds
Limited data acquisition automation
ts of a Internal Aud ProcessTechn Manual data collection processes
Extensive Excel-based Access ACL Analytics
g g
Analytics specific tools and data warehouses
Limited advanced tools and information sharingq Minimal data governance
Fragmented & non-standardized Process inefficiency & duplication
Business criticality drives process evolution
End-to-end process ownership and documentation
Standardized process flows and leverage of workflow and mapping tools Optimized by function and audit area
Quality Control Processes
Process inefficiency and duplication Minimal use of workflow and mapping tools
No formal measurement strategy Limited use of metrics Key Component Risk AssessmentAudit lanning
Ratings defined through interviews Limited on-going analysis
Audit Universe only qualitatively analyzed
Independent Risk Metrics On-going Risk Analysis Shared dashboards Metrics from business leveraged
Some on-going analysis
Universe linked to profit and cost centers
Scope defined through interviews
Data not leveraged to make scope decisions
Data used for all applicable areas of business
Outside metrics leveraged for scoping Some data utilized to understand business
Data used to guide scope discussions Excel and ACL modeling
Manual data acquisition
Controls TestingPous ng
g p g p g
Shared dashboards Excel and ACL modeling
Analytics use driven by requests Most Analytics used for reconciliations No historical testing libraries
Analytics discussion as part of annual planning
Spread of multiple techniques Analytics planed quarter by quarter
Multiple analytics techniques utilized Some historical work papers
Monitoring defined through interviews Metrics from business leveraged Independent analysis and metrics
7 Continuo Monitorin Monitoring defined through interviews
Limited on-going analysis Manual processes
Independent analysis and metrics Shared dashboards
Use of external data Metrics from business leveraged
Some on-going analysis Excel trending of metrics
Appendix D: Case study – Internal Audit analytics
analytics
AML Internal Audit analytics | Internal Audit analytics
Analytics is a powerful auditing tool given the significant dependency controls have on data.
Business driven controls: T h l d i t l
Business-driven controls:
Risk Assessment Customer Risk Scoring
Technology-driven controls:
s Know Your Customer (KYC)
Investigations
OFAC/Watch List Screening
Transaction Monitoring
Processe s
SAR Filing SAR Reporting
Transactional Data Entity Data
a ta Entity Data
Other Referential Data (Alerts, SARs, Watch Lists, Risk Scores, etc.)
Analytics enables a bottom-up approach to reviewing and evaluating the design and operating effectiveness of AML- related control environments by leveraging data
D a
PwC March 2012
related control environments by leveraging data.
ACAMS
AML Internal Audit analytics | Auditing life cycle
Analytics can be leveraged as a tool within each phase of the AML auditing lifecycle.
Risk Assessment
• Assessing transaction/business risk
• Assessing customer/counterparty risk 1
Analytics enables regular, periodic Analytics increases assurance through
Auditing Life Cycle
Analytics enables regular, periodic reporting and accelerates on-going risk identification and assessment.
Analytics increases assurance through
improved population coverage and more
intelligent, risk-based sample selection.
AML Internal Audit analytics | Auditing life cycle – Risk assessment
The following highlights the benefits of analytics applications within AML risk assessments.
Objectives
I. Identify measurable risks II. Evaluate critical controls III. Refine audit plan
Techniques/enablers Profiling Data Mining
• Statistically profiling the quantitative and qualitative parameters of a data set to identify risk areas
p
IV.Define audit scope (high risk areas) (anomalies, spikes, unusual patterns) within a business and/or process.
Sample activities
• Assess business risk
- Measure activity (transactions, accounts, customers) by line of business
- Identify anomalies
• Assess customer risk Assess customer risk
- Measure behavioral profiles (volumes and values)
- Relate customers to risk factors (geographies, products etc )
PwC March 2012
products, etc.)
ACAMS
AML Internal Audit analytics | Auditing life cycle – Audit testing
The following highlights the benefits of analytics applications within AML audit testing.
Objectives
I. Increase testing effectiveness
II. Increase assurance (through increases in population coverage)
Techniques/enablers Logical Recreation
• Independently develop logic and execute
with actual data to identify implementation gaps
i t i t
III. Identify the “needle in the haystack”
IV.Validate data conditions (completeness/quality)
against requirements Data Quality
• Test actual data against quality checks to evaluate data conditions which may impact control
effectiveness Sample activities
Stratified Sampling
• Profile data to identify higher risk transactions or events for sample selection
• Test screening effectiveness
- Leverage fuzzy matching algorithms to mimic watch list screening
- Stratify and sample matches at varying levels
Identify near-miss alerts
8 10
of similarity Identify near miss alerts
and conduct sample
4 6 8
AML Internal Audit analytics | Auditing life cycle – Continuous monitoring
The following highlights the benefits of analytics applications within continuous monitoring.
Objectives
I. Improve availability and frequency of information II. Understand and react to changes in risk profiles III. Leverage prior audit work or risk profiling for go-
Techniques/enablers Reporting
• Providing on-demand or high frequency key risk indicators for continuously monitoring risk profiles
g p p g g
forward reporting
IV.Forecast future risk areas
Predictive Analysis
• Leverage current trends to forecast future results and assess “what-if” scenarios
Sample activities
• Transaction monitoring dashboards - Monitor changes in transactional activity
- Measure ongoing transaction monitoring alerts and SARs by scenario
• Capacity forecasts
- Estimate future alert volumes given observed
PwC March 2012
activity trends and current output
ACAMS
AML Internal Audit analytics | Data validation techniques
Data validation is a fundamental component of effective AML control auditing.
D t l t D t lit R f ti l i t it
A Data completeness B Data quality C Referential integrity
• Verify that the source data reconciles with an independent system of record.
Example:
• Verify that elements critical for surveillance
are in line with expected data conditions (standardized formats
• Verify that relationships between core data (transactions and customers) accurately and completely map to referential
A B C
Example:
Source file control totals (i.e. record counts) reconcile with volumes from the control file (i.e. system of
record) or tie back to management
conditions (standardized formats or value ranges).
Example:
Validate values of transaction amounts and dates. Validate
completely map to referential data
(risk scores, jurisdictions).
Example:
Validate jurisdictions exist for all
reporting. availability of counterparty fields
(no null or blank values).
counterparties in transactional data.
Validate risk scores are available for
all customers.
AML Internal Audit analytics | Sample testing approach
The following is an example of the application of analytics techniques within an AML audit.
Test step Results
1 Validate data completeness Wire data in the compliance data mart was reconciled against feeds extracted from Treasury payment processing systems.
2 Validate quality of source data Customer addresses were analyzed to identify state and country values out of range or in an inappropriate format (i.e. ‘Canada’ rather than expected
‘CA’ abbreviation).
3 Validate implementation of surveillance detection logic
Surveillance scenarios selected for testing were modeled in an analytics platform (recreating each of the logical conditions defined within system
i t ) d th lt il d
requirements) and the results reconciled.
4 Validate the productivity of surveillance results
Alert data was profiled to identify areas of high productivity and low productivity to assess effectiveness of coverage.
5 Assess risk of targeted
ill t l i
Raw transactional data was quantitatively modeled to identify potentially
i i tl i ti it d tt d i t t l l t
surveillance typologies suspicious, outlying activity and vetted against actual alerts.
PwCACAMS March 2012
