STATE OF NORTH CAROLINA

S

TATE OF

N

ORTH

C

AROLINA

P

ERFORMANCE

A

UDIT

D

EPARTMENT OF

H

EALTH AND

H

UMAN

S

ERVICES

NCT

RACKS

-

P

OST

I

MPLEMENTATION

D

ECEMBER

2013

O

FFICE OF THE

S

TATE

A

UDITOR

B

A.

W

,

CPA

P

This audit was conducted to determine the magnitude of defects in NCTracks - North Carolina’s Medicaid payment system - and to assess the Department’s corrective action plan.

B

The North Carolina Department of Health and Human Services replaced its old Medicaid Management Information System and went live with NCTracks on July 1, 2013. NCTracks is expected to process and pay more than $12 billion a year in health care claims for the more than 77,000 medical providers serving approximately 1.5 million North Carolina citizens. Since going live, the Department has encountered a large number of NCTracks system defects. The audit period covered July 1, 2013, through November 6, 2013.The audit fieldwork was conducted from October 29, 2013,toNovember 6, 2013.

K

F

• Since going live, the NCTracks system has encountered more than 3,200 defects.

• The Department has an inadequate framework for the timely resolution of NCTracks defects. • The state lacks a comprehensive action plan to address all NCTracks issues.

• Federal and state government mandated changes have not been implemented within their target or mandatory implementation dates.

• The state government ‘revolving door’ creates a perception of bias or conflict of interest.

K

R

• The Department should establish official guidelines, metrics, and a methodology for tracking the timely resolution of defects. The Department should monitor CSC performance against these metrics.

• The Department should develop a comprehensive master action plan to fix NCTracks issues. • The Department should provide the General Assembly a follow-up report that includes

implementation dates and cost data for all changes that have yet to be implemented.

• The General Assembly should consider a narrow change to state law to limit the ability of senior level state government employees to go directly to work for a vendor they directly managed while employed by the State.

O

I

• The NCTracks system certification timeline is uncertain.

The key findings and recommendations in this summary are not inclusive of all the findings and recommendations in the report.

Beth A. Wood, CPA

State Auditor

Office of the State Auditor

2 S. Salisbury Street 20601 Mail Service Center

Raleigh, NC 27699-0601 Telephone: (919) 807-7500 Fax: (919) 807-7647 Internet http://www.ncauditor.net December 9, 2013

The Honorable Pat McCrory, Governor

Members of the North Carolina General Assembly Mr. Chris Estes, State Chief Information Officer

Dr. Aldona Wos, Secretary, Department of Health and Human Services

Mr. Joseph Cooper, Jr., Chief Information Officer, Department of Health and Human Services

Ladies and Gentlemen:

We are pleased to submit the results of our performance audit of NCTracks titled Department

of Health and Human Services, NCTracks - Post Implementation.

The audit objectives were to determine the magnitude of NCTracks system defects and whether the corrective action plan addressed all NCTracks issues.

The Office of the State Auditor initiated this audit as part of an effort to keep the General Assembly informed on the progress of NCTracks.

The Department was presented in advance with the findings and recommendations on November 7, 2013, and reviewed a draft copy of this report. The Department’s written comments are included after each recommendation section and in Appendix F.

We wish to express our appreciation to the staff of the Department and the Office of Medicaid Management Information System Services for the courtesy, cooperation, and assistance provided us during the audit.

Respectfully submitted,

Beth A. Wood, CPA State Auditor

TABLE OF CONTENTS

PAGE

INTRODUCTION

BACKGROUND ... 1

OBJECTIVES,SCOPE, AND METHODOLOGY ... 2

FINDINGS AND RECOMMENDATIONS 1) THE DEPARTMENT HAS AN INADEQUATE FRAMEWORK FOR THE TIMELY RESOLUTION OF NCTRACKS DEFECTS ... 3

2) THE DEPARTMENT LACKS ACOMPREHENSIVE MASTER ACTION PLAN TO ADDRESS NCTRACKS ISSUES. ... 6

3) NCTRACKS GOVERNANCE CHANGES PRESENT BUDGETARY AND SYSTEM CAPABILITY RISKS TO THE STATE. ... 7

4) STATE GOVERNMENT’S ‘REVOLVING DOOR’CREATES A PERCEPTION OF BIAS OR CONFLICT OF INTEREST ... 9

OTHER INFORMATION  NCTRACKS SYSTEM CERTIFICATION TIMELINE IS UNCERTAIN ... 10

APPENDICES A. NCTRACKS DEFECT SEVERITY FRAMEWORK ... 11

B. SUMMARY OF NCTRACKS SYSTEM DEFECTS (NOVEMBER) ... 12

C. SUMMARY OF NCTRACKS SYSTEM DEFECTS (DECEMBER) ... 13

D. LIST OF TOP PRIORITY CSRS ... 14

E. AUDITOR’S RESPONSE ... 15

F. DEPARTMENT RESPONSE ... 18

1

BACKGROUND

On July 1, 2013, the North Carolina Department of Health and Human Services (the Department) went live with NCTracks – the replacement Medicaid Management Information System. NCTracks is a multi-payer system1 that facilitates provider enrollment, consolidates claims processing activities, and supports healthcare administration for multiple divisions of the Department. NCTracks is expected to process and pay more than $12 billion a year in health care claims for the more than 77,000 medical providers who serve approximately 1.5 million North Carolina citizens.

NCTracks was designed, developed, and is currently operated based on a $484 million contract awarded to Computer Sciences Corporation (CSC) in 2008.2 CSC is expected to serve as the state fiscal agent until 2020.

During the NCTracks development phase, the federal government funded up to 90% of the associated costs. Since going live on July 1, 2013, the federal government has paid 50% of the NCTracks operational costs. The State pays the other half. The federal government cost share will increase to 75% once the system is certified by the federal Centers for Medicare and Medicaid Services (CMS). Per the existing contract, the vendor is expected to assist the State in achieving federal certification for NCTracks within one year of the operational start date, or by July 1, 2014.

On May 22, 2013, the Office of the State Auditor released an NCTracks pre-implementation audit report titled Department of Health and Human Services, NCTracks (MMIS Replacement) –

Implementation.3 The report indicated that the Department had failed to fully test the system and that the production testing process had flaws.

Since going live, the Department and CSC have encountered a large number of system defects. On October 8, 2013, senior leaders from the Department and CSC appeared before the Joint Legislative Oversight Committees on Health and Human Services and Information Technology to answer questions from the General Assembly about NCTracks issues and its impact to medical providers throughout the state.

1 _{NCTracks is used by the Division of Medical Assistance (DMA), the Division of Mental Health, Developmental} Disabilities, and Substance Abuse Services (DMH/DD/SAS), the Division of Public Health (DPH), the Migrant Program for the Office of Rural Health and Community Care (ORHCC). Providers enrolled in DMA, DPH, and ORHCC health plans submit claims for covered health care services to NCTracks Provider Portal. NCTracks coordinates processing among the payers to ensure the proper assignment of the payer, benefit plan, and pricing methodology for each service on a claim. (Source: http://ncmmis.ncdhhs.gov/faq.asp)

2

The contract value is allocated as follows: $186,604,862 for design and development; $27,338,679 for Provider enrollment and Retro DUR Operations (conducted from April 2009 through June 2018); $15,277,760 for the “Health Information Technology” initiative; $179,485,713 for five years of operations; and $76,173,784 is reserved for two years of operations that remain optional to DHHS. (Source: DHHS)

2

O

,

S

,

M

The audit objectives were to determine the magnitude of NCTracks system defects and whether the corrective action plan addressed all NCTracks issues.

The Office of the State Auditor initiated this audit as part of an effort to keep the General Assembly informed on the progress of NCTracks.

The audit scope included a review of NCTracks system defects, corrective action plan documentation, the call center process, the Change Service Request (CSR) approach, and the NCTracks certification timeline. The audit period covered July 1, 2013, through November 6, 2013. The audit fieldwork was conducted from October 29, 2013, to November 6, 2013.

To accomplish the audit objectives, auditors gained an understanding of post-implementation activities, analyzed NCTracks system documentation, interviewed Department officials and CSC staff, observed call center operations, and accessed appropriate technical systems.

As a basis for evaluating NCTracks governance processes, guidance contained in the COBIT 5 framework issued by ISACA was applied. Per North Carolina General Statute §143D-6, the State Controller has directed state agencies to adopt COBIT as the IT internal control standards for the state. COBIT 5 is a comprehensive framework that assists enterprises in achieving their objectives for the governance and management of enterprise information and technology assets (IT). This framework helps enterprises create optimal value from IT by maintaining a balance between realizing benefits and optimizing risk levels and resource use. COBIT 5 enables IT to be governed and managed in a holistic manner for the entire enterprise, taking in the full end-to-end business and IT functional areas of responsibility, considering the IT-related interests of internal and external stakeholders.

This performance audit was conducted in accordance with generally accepted government auditing standards. Those standards require that auditors plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for findings and conclusions based on the audit objectives. The Office of the State Auditor believes that the evidence obtained provides a reasonable basis for the findings and conclusions based on the audit objectives.

This audit was conducted under the authority vested in the State Auditor of North Carolina by

3

F

#1:

T

D

H

A

I

F

F

T

T

R

NCT

D

Since going live, the NCTracks system has encountered more than 3,200 defects. More than 600 defects remain unresolved at the time of the audit. Auditors also found:

(1) The number of new ‘High’ and ‘Medium’ defects reported each month has not decreased (2) There is no contractual requirement related to the timely resolution of defects

(3) Medicaid providers are bypassing the call center for assistance NCTracks Defects

The number of ‘Critical’ NCTracks defects reported each month has decreased over time. However, the number of new defects reported each month for issues considered ‘High’ and ‘Medium’ has not decreased and remains consistent.

Anytime a new NCTracks system defect is discovered, a severity level is assigned to it. According to DHHS, a ‘Critical’ defect indicates a system-wide failure, a ‘High’ defect means inconsistent results within the system, and a defect classified as ‘Medium’ indicates that a work-around exists. See Appendix A for the framework used by the Department to assign severity levels to defects.

Between August 3, 2013, and November 5, 2013, the number of new reported defects classified as ‘High’ has ranged between 449 to 555 a month, while the number of new defects classified as ‘Medium’ has ranged between 144 to 230 a month.

As of November 5, 2013, 19% of all reported NCTracks defects remain open. A review of the total number of NCTracks defects shows that 71% of all defects have been of a ‘High’ severity. ‘High’ severity defects also account for 74% of all open issues.

The Number of NCTracks Defects Reported

(Monthly Snapshot By Severity) Total

Defects (As of Nov 5) TOTAL OPEN (As of Nov 5) On July 2* July 3 to Aug 2 Aug 3 to Sept 4 Sept 5 to Oct 2 Oct 3 to Nov 5 Critical 11 130 35 14 13 203 11 High 76 697 555 449 553 2330 474 Medium 34 163 230 144 169 740 152 Grand Total 121 990 820 607 735 3273 637

Source: Production Daily Status report and SILK computer-generated report

Notes: *Per DHHS, on June 30, 2013, there were no ‘Critical’ defects in the system (as required prior to going live).

4

Auditors also found that, as of November 5, 2013, the top three Medicaid business processes that have been affected with the most defects since NCTracks went live are:

• Provider Portal: 1025 defects (31% of all system defects).

Serves as entry point for Medicaid providers to perform necessary actions such as: o Check Recipient Eligibility

o Enter and Check Prior Approvals o Create and check the status of a claim o Enter and check referrals

o Check for Managed Care overrides

o Search and verify procedure codes o Search payment history

o Search Consent Forms

o Update their own provider data o View messages, letters and reports • Batch Processing: 559 defects (17% of all system defects)

Jobs executed on the system without manual intervention. Examples of these types of jobs include: overnight reports and the processing of claims and checkwrites.

• Operations Portal: 525 defects (16% of all system defects)

Serves as entry point for Department and CSC fiscal agent staff to support the following functions: o Claims o eCommerce o Managed Care o Financial o Provider o Recipient o Reference o Prior Approval

o Third Party Liability (TPL) o Admin/Other

Together, these three Medicaid business processes have accounted for 64% of all NCTracks system defects. See Appendix B for a full breakdown on the number and types of business processes affected by defects.

Inadequate Framework for Timely Defect Resolution

Auditors found that NCTracks defects are being resolved, however, lack of formal goals to resolve defects in a timely manner indicates that the Department and CSC may not be managing all NCTracks defects efficiently. Auditors recognize that defects differ in complexity and other characteristics, so the time to resolve defects will vary and therefore trying to set hard and fast resolution timeframes is difficult.

The Department has established “target response times” for each defect severity that should be used to ensure timely resolution of defects. The target response times are listed in the table below.

Defect Severity Target Response Time

Critical

(System Wide Failure) 24 hours High 5 business days

5 (Inconsistent Results) Medium

(Work-Around Exists) 10 business days

However, the Department has not documented what the target response timeframe means. When auditors asked what constituted a response, the Department gave conflicting answers. It remains unclear what a response timeframe means.4 Without clarification of what is included in a

response and without the tracking of CSC responses to NCTracks defects, the Department

cannot effectively ensure that system defects are resolved timely.

Auditors noted that the Daily Production Status reports, used by Department and CSC senior management, provides an overview of the latest number of defects and their current status but does not track response times or the timeliness of resolving defects.

Auditors conducted an analysis of “Critical” defects. The table below indicates the resolution times reported by DHHS and CSC.

‘Critical’ Defect Resolution

(As of October 31, 2013)

Timeframe Fixed within _{24 hours} Fixed between _{24 - 48 hours} Fixed between _{48 - 72 hours} Fixed between _{72 - 96 hours} Fixed after _{96 hours} Total

Number 6 8 7 9 71 101*

Source: SILK computer-generated report

Notes: *As of November 5, 2013, the total number of reported ‘Critical’ defects was 203. This table accounts for 101

of those defects. The total represented here indicates the number of ‘Critical’ defects that auditors found labeled as fixed. Auditors found that the other 102 defects are labeled as: working as designed, duplicate, or old issue.

During the audit, the Department indicated that the timely resolution of defects is not part of the contractual service level agreements (SLAs) with CSC. Auditors found that over 200 SLAs exist related to the management and performance of NCTracks. However, none are related to the timely resolution of defects.

Providers Bypassing the Call Center

Inconsistencies exist in the way provider issues are logged and prioritized. According to CSC, calls to the call center5 are handled based on a ‘first in, first out’ approach. When a provider calls the call center, they must first go through the Tier1 group for support. If the Tier1 group cannot resolve an issue, it is escalated to Tier2.6 However, during the audit, call service agents informed

4 _{The Department did not provide auditors with a written definition of “target response timeframe” until December} 4, 2013. Auditors provide an assessment of this definition in Appendix E, State Auditor’s Response to DHHS

Response.

5 _{The NCTracks call center is responsible for assisting providers with inquiries regarding enrollment, claim status,} recipient eligibility, and other information needed to support their service to Medicaid recipients. It is staffed by approximately 100 call service agents who serve as Tier1, the first line of customer service.

6

6

auditors that some providers and clearinghouses have become familiar with Tier2 personnel and now bypass the call center.

When calls are not handled and recorded according to established procedures, NCTracks issues are not logged effectively to maintain full historical records. Furthermore, as a result of prioritization occurring outside of the ‘first in, first out’ approach, there is a risk that the Tier2 group, which is trained to handle more difficult issues, is spending time on issues that the Tier1 group could handle. The ISACA COBIT 5 Framework states that organizations should follow established procedures and log all service requests and incidents.7

Recommendations:

1. The Department should establish official guidelines, metrics, and a methodology for tracking the timely resolution of defects. The Department should monitor CSC performance against these metrics.

2. The Department should work with CSC to ensure that all call center service requests are processed and logged per established procedures.

Agency Response:

In addition to its existing means of managing defects described in DHHS’ response above [Appendix F], DHHS will establish formal guidelines, metrics and a methodology for tracking defect resolution. Target resolution times will represent guidelines and not contractual commitments by the vendor, as the NCTracks program managers cannot predict how long future/unknown defects will take to remediate. Responsible entity: Program Management Office. Target implementation date: January 1, 2014.

The Department will direct the vendor to log all incoming calls in a common historical database, regardless of whether a call is first received by Tier1 or Tier2 personnel. Thereafter, the service request, incident or other call subject matter will be prioritized and handled in accordance with established procedures. Responsible entity: Operations and CSC. Target implementation date: January 1, 2014.

F

#2:

T

D

L

A

C

M

A

P

T

A

NCT

I

The Department does not have a comprehensive and cohesive master action plan to direct the remediation of technical and operational NCTracks issues.

Throughout the audit, documentation covering some post-implementation activities and defect resolution processes was obtained from various sources. However, auditors did not find one document that could be considered a master action plan to address all NCTracks issues.

7

During the audit, the Department indicated that on October 29, 2013, it established a new committee8 to evaluate NCTracks’ work priorities against the current business needs of the Department and provider and recipient communities.

Regardless of approach, the lack of a comprehensive master action plan increases the risk that NCTracks defect resolution processes are not being managed in the most effective and efficient way, and that strategic objectives may be misunderstood or misinterpreted by the business or IT personnel. The ISACA COBIT 5 Framework states that organizations should agree on and implement an action plan to address identified issues, and business process owners and IT technical management should be engaged in the development of the action plan.9

Recommendation:

1) The Department should develop a master action plan and integrate, in a traceable manner, all documents related to addressing NCTracks issues.

Agency Response: [The response below is a portion of the Department’s full response which can be found in Appendix F.]

DHHS is currently implementing processes to address technical and operational NCTracks issues. DHHS is developing a single document outlining its master action plan to present all NCTracks issues for consideration and disposition by the existing “Executive Change Control Body” (ECC). A graphic representation of the “Management Framework” for the ECC’s prioritization and approval of work items is attached to this audit response document as Attachment 2 [Appendix F]. The integration of all related documents will be completed by a target date of January 1, 2014.

F

#3:

NCT

G

C

P

B

S

C

R

S

Since July 1, 2013, there have been major updates to the approach the Department will take to implement required capabilities. These changes present budgetary and system functionality risks to the State.

Government Mandated and Top Priority Changes Have Not Been Implemented

Federal and state government mandated capabilities have not been implemented in the NCTracks system. According to the Department’s list of top priority Change Service Requests (CSRs), 12 out of 14 (85%) legislative or regulatory mandated CSRs were not in place by their target or mandatory implementation dates. Of the 12 CSRs that were not implemented, two had a mandatory implementation date and 10 had a target implementation date.

Mandated and top priority changes that have not been implemented include business processes related to, among others, rehabilitation-pricing rules and the Medicaid pharmacy program. See

Appendix D for a breakdown of the top priority CSRs.

8

NC MMIS Executive Change Control

8

Prior to NCTracks going live on July 1, 2013, the Department had planned two rounds of major updates to take effect in October 2013 and January 2014 that would provide the system required capabilities that were not available at go-live. According to the Department, the General Assembly was informed that not all required capabilities would be available at go-live and that a two-round rollout plan had been established. However, based on the large number of defects in the NCTracks system and its need to focus efforts and resources on prompt resolution of the issues, the Department decided to change its two-round approach and instead develop and address, in the short term, a list of top priority CSRs, and address future CSRs on a rolling basis.10

A November 1, 2013, NCTracks progress report from the Department to the General Assembly indicates that CSR costs and estimated completion dates have yet to be determined.11

Overall, the Department’s new approach to implement required capabilities introduces uncertainty, since it is unknown when required functionality will be available and when the State will benefit from mandated capabilities.

Recommendations:

1) The Department should provide the General Assembly a follow-up progress report that includes implementation dates and cost data for all CSRs that have yet to be implemented.

2) The Department should continually update providers and end users on the status of CSRs, to ensure an adequate understanding of the functionality of NCTracks.

Agency Response:

The Department is working diligently to develop new estimated implementation dates for the outstanding CSRs along with accurate cost estimates. The Department will provide the General Assembly with a progress report that contains the forecast and cost information. Responsible entity: Contracts Administration. Target implementation date: March 1, 2014.

The Department currently sends all providers a weekly email (or more frequently, if required) that contains detailed pertinent information about changes to NCTracks. This same communication is also distributed internally to management and staff associated with the NCTracks program. These messages address, among other things, functionality changes that may be directly related to CSRs.

While some CSRs may be directly relevant to providers and end users, others may relate only to internal system management. Additionally, the list of CSRs and the scope of each is somewhat fluid. Because of these limitations, the Department agrees to periodically update providers and end users on the status of CSRs, but only those CSRs that directly affect functionality that is relevant to providers and end users in their respective roles.

10 _{DHHS advises that it will implement future CSRs based on Department priorities and using the most economical} approach.

9

F

#4:

S

G

’

‘R

D

’

C

P

B

C

I

A former DHHS employee who served for more than four years as the NCTracks Senior Program Manager and Associate Program Director now works for the NCTracks vendor, CSC, as the NCTracks Executive Account Director.

The established contract between the Department and CSC requires the vendor to obtain approval from the State for all individuals considered ‘Key Personnel’.12 CSC asked the Department for approval of this appointment in August 2013. On August 21, 2013, the Department approved CSC’s request. Auditors found that DHHS and CSC followed established contract procedures.

Without a “cooling off” period, the North Carolina State Ethics Commission has stated that, “even if there is no legal or “ethical” impediment to doing so, leaving public service to work for

a private entity that will do business with the employee’s former state employer, may create a perception of bias or conflict of interest. This potential for apparent bias attaches not only to the former employee’s subsequent dealings with his former state employer but also to their actions and decisions immediately prior to leaving state employment.”13

Recommendation:

1) The General Assembly should consider a change to state law to address public servants separating from employment with the State and accepting employment with a business entity that contracts with or does business with the State in matters in which the public servant was directly and significantly (emphasis added) involved during their employment with the State.

12

RFP Section 30.43.2, Key Personnel

10

NCTracks Certification Timeline Uncertain

A potential NCTracks certification delay by the federal Centers for Medicare and Medicaid Services (CMS) raises concerns that the State will not begin to save approximately $9.6 million a year at the earliest possible point, July 2014. The State currently pays 50% of NCTracks operational costs. Once the system is certified the federal government cost share will increase to 75% and the State will have to cover only 25% of the costs, resulting in $9.6 million in annual savings. The Department has indicated that once the system is certified the state will also receive a federal retroactive reimbursement effective as of July 1, 2013.

On October 8, 2013, the Department’s senior leadership presented a timeline to the General Assembly that indicated NCTracks would be certified in December 2014. During the audit, the Department indicated that the timeline presented to the General Assembly was an educated guess and that CMS is ultimately responsible for setting the certification timeline.

It is important to note that regardless of potential retroactive reimbursements from the federal government, the contract states that the Vendor shall apply its best efforts to help the State achieve federal certification for NCTracks within one year of the operational start date [July 1, 2014]. It also states, “Should certification fail to be achieved within one year of the operational

start date, the Vendor shall be liable for damages to the extent they result from its actions or inactions relating to the lack of certification.”14

According to the Department, the General Assembly’s biennial budget assumes that it will take DHHS two years to achieve certification and that CMS will fund the system at 50% until the third year of NCTracks operations, which begins in July 2015. The Department indicated that as a result of this contingency plan there will not be a budget shortfall related to system certification for SFY 2014/2015. Additionally, the Department indicated that CSC is on track to deliver their completed checklist of CMS certification requirements by the end of February 2014.

11

NCTracks Defect Severity Framework

Defect Severity Examples Target Response Time

Critical

(System Wide Failure)

 No workaround  Loss of data  Corruption of data  System crash

 Network Connectivity unavailable

 HIPAA violation

24 hours

High

(Inconsistent Results)

 Missing/failed functional requirements  Eligibility denial

 Incorrect payment status  Wrong payment amount

 Edit/audit incorrect disposition

5 business days

Medium

(Work-Around Exists)

 Business function can be completed with manual

intervention or an alternate system method 10 business days Source: DHHS

12

NCTracks Defect Summary

Summary Defect Severity

Business Process Affected Critical High Medium Total

Point of Sale 4 13 7 24

User Access 6 28 10 44

Provider Portal 51 857 117 1025

Operations Portal 24 345 156 525

Electronic Data Interchange / Trading Partner 9 32 16 57

Inbound Interfaces 5 21 2 28

Claims Adjudication 9 270 59 338

Prior Approval/Authorization 11 110 53 174

Call Center Monitoring 2 9 9 20

Batch Processing 59 338 162 559

Pega Ticket System 12 148 78 238

Recipient Portal 1 24 6 31

Outbound Interfaces 4 18 19 41

Remittance Advice / Checkwrite 4 62 28 94 Performance Engineering Monitoring 0 0 1 1

Provider Data Issue 2 26 6 34

SLA 0 29 11 40

Total Defects 203 2330 740 3273

Total Fixed 192 1856 588 2636

TOTAL OPEN 11 474 152 637

Source: Production Daily Status report and SILK computer-generated report

13

NCTracks Defect Summary

The information below was obtained from the Department after audit fieldwork and updates Appendix B, thru December 3, 2013. The information in the table below is unaudited.

Summary Defect Severity

Business Process Affected Critical High Medium Total

Point of Sale 5 13 7 25

User Access 6 31 12 49

Provider Portal 55 1049 143 1247

Operations Portal 25 399 174 598

Electronic Data Interchange / Trading Partner 9 32 19 60

Inbound Interfaces 5 26 4 35

Claims Adjudication 10 307 61 378

Prior Approval/Authorization 11 123 60 194

Call Center Monitoring 2 9 9 20

Batch Processing 60 352 184 596

Pega Ticket System 12 192 90 294

Recipient Portal 1 23 6 30

Outbound Interfaces 4 21 20 45

Remittance Advice / Checkwrite 4 67 27 98 Performance Engineering Monitoring 0 0 1 1

Provider Data Issue 3 39 11 53

SLA 0 35 14 49

Total Defects 212 2718 842 3772

Total Fixed 203 2234 682 3119

TOTAL OPEN 9 484 160 653

Source: Production Daily Status report

14

Top Priority Change Service Requests (CSRs)

CSR# Title and Brief Description of Change Implementation Date

Mandated by: (e.g., CMS, Legislature)

CSR 1366 Changes to MCO Encounter Processing 1/1/2014* CMS compliance CSR 1356 Infant Toddler Program Recipients Options for Not Billing

Insurance or Medicaid N/A N/A

CSR 1077 RHC/FQHC changes to audit summary (FO09.230/13422) 11/30/2013 DMA policy compliance CSR 1357 Addition of Codes and Changes to the Be Smart Family

Waiver Planning Program to SPA 11/1/2013

CMS waiver compliance CSR 1091 DMA Rehab Pricing Rule revision. 10/31/2013 Medicaid State Plan CSR 1088 Billing Lower Level of Care beds. 10/31/2013 Medicaid State Plan CSR 1078 Pharmacy program / edit changes from legacy HPES CSRs 3/1/2013* Legislative mandate CSR 1079 NC Medicaid Pharmacy legacy program and edit

changes from HPES CSRs 3/1/2013* Legislative mandate CSR 1080 NC Medicaid Pharmacy Program Edit updates 3/1/2013 Legislative mandate CSR 1148 Recipient Lock-In. Include Recipient Management (for

Controlled Substance) Lock-in info in Eligibility Requests N/A N/A CSR 1134 Changes to reimbursement of Home Health T1999 billing 10/31/2013 Medicaid State Plan CSR 1355 Request to modify the PA entry functionality to allow the

ability to enter a PA for a specific NDC/GCN/GSN 10/31/2013

DMA policy compliance CSR 1365 Re-Establish Provider Enrollment for Behavioral Health

Providers 7/1/2013

DMA Policy compliance CSR 1200 Infant Toddler Program Payment and Fees federal

requirement through POMC N/A N/A

CSR 1364 Produce New DMH/DMA Global Provider Extract File for

the LME/MCOs N/A N/A

CSR 1210 Direct Enrollment 10/31/2013 DMA policy

compliance CSR 763 Modifications Required to Support Cost-Sharing for

Health Coverage for Workers With Disabilities 10/12/2012

Federal/State legislative mandate CSR 862

Change Application Process Flow for

Rendering/Attending Providers and New Edit for Manage Change Request CCNC/CA Provider Terminations

3/1/2013 DMA policy compliance CSR 1527 Office Administrator, User Provisioning and New

Administrative Role Access N/A N/A

Source: DHHS on November 4, 2013.

15

Auditor’s Response

This audit was conducted to provide the General Assembly immediate information about the magnitude of NCTracks defects and DHHS’ response to problems with the system. We appreciate DHHS’ effort to assist our office in providing the General Assembly with information that is timely enough to be useful.

The Department has generally agreed with the recommendations made in our report.

However, we are required to provide additional explanation when an agency’s response could potentially cloud an issue, mislead the reader, or inappropriately minimize the importance of our findings.

Generally Accepted Government Auditing Standards state,

When the audited entity’s comments are inconsistent or in conflict with the findings, conclusions, or recommendations in the draft report, or when planned corrective actions do not adequately address the auditor’s recommendations, the auditors should evaluate the validity of the audited entity’s comments. If the auditors disagree with the comments, they should explain in the report their reasons for disagreement.

To ensure the availability of complete and accurate information and in accordance with Generally Accepted Government Auditing Standards, we offer the following clarifications.

In its response to the Background information section of the audit report, the Department states:

“After the auditor’s release of its May 22, 2013 report, the Department received the

approval of the Centers for Medicare and Medicaid Services (CMS) to go-live on July 1st.

The Department also received favorable opinions about the July 1st implementation date

from a consultant, Susan Young, and from a CMS-mandated IV&V vendor, Maximus Consulting Services, Inc.”

For clarification, the reader should not correlate the approval from CMS with its review of the May 22, 2013, NCTracks report. While this information appears in the same sentence, they are not related. The CMS approval was based on information provided by the Department.

The Department also states that it received a favorable opinion about going live on July 1, 2013, from Maximus, the CMS-mandated IV&V vendor. The Department’s response fails to take into account that our May 22, 2013, report expressed serious concerns about Maximus. Our audit found that Maximus did not provide independent oversight of the NCTracks testing process. Our NCTracks Pre-Implementation report states:

“During the user acceptance testing and production simulation testing phases, Maximus, the independent verification and validation (IV&V) vendor, relied exclusively on the test results reports of other vendors to conduct its own test case analysis. By relying on the test result reports of other vendors, Maximus did not help minimize system implementation risks as required by the Centers for Medicare and Medicaid Services (CMS).”

16

“By not conducting independent verification and validation analysis during user testing, the monitoring and high-level auditing of test activities by Maximus is questionable, especially considering that they were not aware of key details and issues regarding the user acceptance test case environment.”

For the Department to consider Maximus as a source for its go-live decision is questionable. In its response to Finding #1, ‘NCTracks Defects’ section of the audit report, the Department states:

“To put a volume of 3,273 defects into proper perspective for a system the size of NCTracks, the Department offers the following. In the software development industry, the complexity of a software system is commonly measured by determining the number of “function points” within the software’s code. A widely accepted technique for gauging the success of a software development effort is to determine the ratio of defects to function points at various stages of the software’s development. According to software development experts at Software Productivity Research LLC, the average defects per function point ratio at go-live for a software system is 0.75 across all U.S. software development projects in the survey. For projects greater than 10,000 function points, such as NCTracks, the average is 1.67 defects per function point. As of November 5, 2013, NCTracks is operating at 0.19 defects per function point, which is significantly better than the industry average for a software system of such size and complexity at this stage.”

The objectives of our audit were to determine the magnitude of NCTracks issues in terms of the number of ‘Critical, High, and Medium’ defects in the system, whether the corrective action plan addressed all NCTracks issues, and assess the Department’s approach for timely resolution. The Department’s response does not consider either of these objectives and the perspective offered in its response deals only with the number of defects and not the severity.

Additionally, since the objective of this audit was not to assess NCTracks defects based on the number of its function points, we express no opinion on the number of defects in terms of function points.

In its response to Finding #1, ‘Inadequate Framework for Timely Defect Resolution’ section of the audit report, the Department’s response is misleading as it attempts to inappropriately minimize the importance of our finding. In its response the Department states:

“The Department has been managing NCTracks defects efficiently since go-live. Formal target response times are in place and defect metrics have been tracked. The State Auditor’s Appendix A serves as evidence that the Department has a formal hierarchy of defect response times in place. Further, as acknowledged by a footnote to tables entitled “The Number of NCTracks Defects Reported” and “Critical Defect Resolution," the State Auditor derived the tables from data that the Department was already tracking and reporting in “Production Daily Status” reports and in reports generated by the Department with its “SILK” development management tool. The Department also tracks defects in other publications not named in the audit finding, including the weekly management presentation, as illustrated in Attachment 1 to this audit response document. The target response timeframe is defined as follows: it begins when a prospective defect is logged in the tracking system and ends when the vendor formally acknowledges the

17

issue. Acknowledgement occurs when the vendor indicates that it understands the problem, has no outstanding questions of the State, and assigns the defect to a specific person/developer for resolution.”

While the Department states that it has been managing NCTracks defects efficiently since go-live, it has not provided sufficient and adequate documentation to demonstrate this. Evidenced by the Department’s own response to the audit recommendation, they agree to “establish formal guidelines, metrics, and a methodology for tracking defect resolution.” Additionally:

1) The report does not dispute that the Department has a formal hierarchy of defect response times. The issue is that the Department does not effectively use the defect response times as a means for ensuring that defects are resolved timely.

2) While there are formal target response times, the Department had not defined what CSC’s responsibilities were for a response nor was the Department tracking them. Furthermore, based on its own definition, once acknowledgement is received from CSC, the Department does not have, and is not provided, an estimate of time to complete or a target date for resolution. Without this information, the Department cannot track in its systems, reports, and management meetings that the vendor is resolving issues in a timely manner.

3) The report does not dispute that the Department has a database where NCTracks defects are logged. The issue demonstrated in the audit is that the information within the database is not formalized into a meaningful report to be used by management during the weekly “DHHS/CSC NCTracks Executive Briefing” to ensure that defects are resolved in a timely manner. The “Critical Defect Resolution” table on page 5 of the report, created by the State Auditor staff, is an example of a meaningful way for management to use its own data to analyze the timeliness of defect resolution.

The purpose of the Auditor’s Response is to provide the reader with an explanation when an agency’s response could potentially cloud an issue, mislead the reader, or inappropriately minimize the importance of our findings.

18

North Carolina Department of Health and Human Services

Pat McCrory Aldona Z. Wos, M.D.

Governor Ambassador (Ret.)

Secretary DHHS

December 9, 2013 The Honorable Beth A. Wood, State Auditor

Office of the State Auditor 2 South Salisbury Street 20601 Mail Service Center

Raleigh, North Carolina 27699-0601 Dear Auditor Wood:

We have reviewed your performance audit entitled Department of Health and Human Services, NCTracks –

Post Implementation. The NCTracks program is part of a much larger transformational process to better serve

Medicaid patients and ensure that providers are paid for their services in an efficient and timely manner. This transformation involves patients and their families, service providers and a new technology. NCTracks is the technology component and we appreciate the auditor’s look at this new component in its first few months of operation.

The following is our response, by section, to the Report of the performance audit that was provided to us at 1:32 p.m. on Friday, December 6, 2013.

EXECUTIVE SUMMARY

The responses outlined herein are likewise applicable to the corresponding comments in the Executive Summary.

BACKGROUND

Paragraph 4 states: “On May 22, 2013, the Office of the State Auditor released an NCTracks pre-implementation audit report titled Department of Health and Human Services, NCTracks (MMIS Replacement)

– Implementation. The report indicated that the Department had failed to fully test the system and that the

production testing process had flaws.”

DHHS Response: After the auditor’s release of its May 22, 2013 report, the Department received the approval of the Centers for Medicare and Medicaid Services (CMS) to go-live on July 1st. The Department also received favorable opinions about the July 1st implementation date from a consultant, Susan Young, and from a CMS-mandated IV&V vendor, Maximus Consulting Services, Inc.

CMS further advised that it would only provide funding for the operation of one system. Therefore, the operation of parallel systems was not feasible.

19

In addition to having favorable opinions in support of going live effective July 1st, delaying the go-live date beyond July 1st would have resulted in either converting to a new system during a fiscal year or implementing it effective July 1st of 2015. Implementing during a fiscal year would have complicated the State’s closing its books at year end due to the need to consolidate reporting from two very different claims systems. Implementing July 1, 2014 was not a feasible option because providers would have only 3 months to adapt to the new system prior to the mandatory adoption of ICD-10, which is mandated for use by all payers and providers in the U.S. effective October 1, 2014. Delaying implementation until July of 2015 would have resulted in additional significant costs for the State.

FINDINGS, RECOMMENDATIONS, AND RESPONSES

FINDING #1: THE DEPARTMENT HAS AN INADEQUATE FRAMEWORK FOR THE TIMELY RESOLUTION OF NCTRACKS DEFECTS

NCTracks Defects

DHHS Response: To put a volume of 3,273 defects into proper perspective for a system the size of

NCTracks, the Department offers the following. In the software development industry, the complexity of a software system is commonly measured by determining the number of “function points” within the software’s code. A widely accepted technique for gauging the success of a software development effort is to determine the ratio of defects to function points at various stages of the software’s

development. According to software development experts at Software Productivity Research LLC, the average defects per function point ratio at go-live for a software system is 0.75 across all U.S. software development projects in the survey. For projects greater than 10,000 function points, such as NCTracks, the average is 1.67 defects per function point. As of November 5, 2013, NCTracks is operating at 0.19 defects per function point, which is significantly better than the industry average for a software system of such size and complexity at this stage.

Although NCTracks has fewer defects per function point than industry average for a comparable software system, we understand that some providers have not received payment for their claims for services. We have and will continue to work diligently to ensure that every provider is paid timely and accurately for covered services rendered.

Inadequate Framework for Timely Defect Resolution

Paragraph 1 states: “Auditors found that NCTracks defects are being resolved, however, lack of formal goals to resolve defects in a timely manner indicates that the Department and CSC may not be managing all NCTracks defects efficiently.”

DHHS Response: The Department has been managing NCTracks defects efficiently since go-live.

Formal target response times are in place and defect metrics have been tracked. The State Auditor’s Appendix A serves as evidence that the Department has a formal hierarchy of defect response times in place. Further, as acknowledged by a footnote to tables entitled “The Number of NCTracks Defects Reported” and “Critical Defect Resolution," the State Auditor derived the tables from data that the Department was already tracking and reporting in “Production Daily Status” reports and in reports generated by the Department with its “SILK” development management tool. The Department also tracks defects in other publications not named in the audit finding, including the weekly management presentation, as illustrated in Attachment 1 to this audit response document.

The target response timeframe is defined as follows: it begins when a prospective defect is logged in the tracking system and ends when the vendor formally acknowledges the issue. Acknowledgement occurs when the vendor indicates that it understands the problem, has no outstanding questions of the State, and assigns the defect to a specific person/developer for resolution.

20

Recommendations:

1. The Department should establish official guidelines, metrics, and a methodology for tracking the timely resolution of defects. The Department should monitor CSC performance against these metrics.

DHHS Response: In addition to its existing means of managing defects described in DHHS’ response

above, DHHS will establish formal guidelines, metrics and a methodology for tracking defect resolution. Target resolution times will represent guidelines and not contractual commitments by the vendor, as the NCTracks program managers cannot predict how long future/unknown defects will take to remediate. Responsible entity: Program Management Office. Target implementation date: January 1, 2014.

2. The Department should work with CSC to ensure that all call center service requests are processed and logged per established procedures.

DHHS Response: The Department will direct the vendor to log all incoming calls in a common

historical database, regardless of whether a call is first received by Tier1 or Tier2 personnel. Thereafter, the service request, incident or other call subject matter will be prioritized and handled in accordance with established procedures. Responsible entity: Operations and CSC. Target implementation date: January 1, 2014.

FINDING #2: THE DEPARTMENT LACKS A COMPREHENSIVE MASTER ACTION PLAN TO ADDRESS NCTRACKS ISSUES

Recommendation:

1. The Department should develop a master action plan and integrate, in a traceable manner, all documents related to address NCTracks issues.

DHHS Response: DHHS is currently implementing processes to address technical and operational

NCTracks issues. DHHS is developing a single document outlining its master action plan to present all NCTracks issues for consideration and disposition by the existing “Executive Change Control Body” (ECC). A graphic representation of the “Management Framework” for the ECC’s prioritization and approval of work items is attached to this audit response document as Attachment 2. The items for the ECC’s consideration include project “Customer Service Requests” (CSRs), “System Defects” and “File Maintenance Requests” (FMRs). Each division stakeholder in the Department independently initiates work items that require prioritization and approval for implementation in NCTracks’ multi-payer system. In addition, vendors (CSC and Truven) identify needed work items that generally promote operational efficiencies, such as software or hardware upgrades. This framework provides for the examination of all work associated with the multi-payer system, giving proper consideration to the impact across divisional stakeholders and vendors.

A review of the proposed body of work begins with a ground floor analysis performed by the “Interdivisional Work Group,” which is comprised of division “Subject Matter Experts” (SMEs). This work group evaluates each work request by assessing the impact across divisional boundaries, identifying opportunities to bundle related work items across work request types and noting dependencies to defect resolutions. A recommended list of work items is submitted to the “Joint Review Committee” (JRC). The JRC develops a multi-payer implementation priority of all work items after a review of the “Basis of Effort” (BOE, a form of cost estimate) that is submitted by the vendor for each item. After the JRC reviews the BOE cost in relation to the business value, the JRC informs the division’s ECC representative of the work prioritization and sequencing necessary to meet the goals, objectives and time frames of the Department.

21

On behalf of the divisions, the ECC approves the work items and the associated priority for implementation. The pertinent vendor assesses the work effort against its staff capacity and identifies workload efficiencies. The vendor establishes a release plan with a commitment to deliver the requested work. The Department is managing the full body of work by continuous review from all Department levels, to ensure that work effort approvals maintain a balance among business needs, legislative initiatives and value to the Department.

The integration of all related documents will be completed by a target date of January 1, 2014.

FINDING #3: NCTRACKS GOVERNANCE CHANGES PRESENT BUDGETARY AND SYSTEM CAPABILITY RISKS TO THE STATE

Government Mandated and Top Priority Changes Have Not Been Implemented

Paragraph 1 reads:

“Federal and state government mandated capabilities have not been implemented in the NCTracks system. According to the Department’s list of top priority Change Service Requests (CSRs), 12 out of 14 (85%) legislative or regulatory mandated CSRs were not in place by their target or mandatory implementation dates. Of the 12 CSRs that were not implemented, two had a mandatory implementation date and 10 had a target implementation date.”

DHHS Response: The targeted implementation dates were subjectively established by the

Department based on the best information available at the time. The management of CSR implementation dates is being addressed as part of the ECC process described in the Department’s response to Finding #2.

Recommendations:

1. The Department should provide the General Assembly a follow-up progress report that includes implementation dates and cost data for all CSRs that have yet to be implemented.

DHHS Response: The Department is working diligently to develop new estimated implementation

dates for the outstanding CSRs along with accurate cost estimates. The Department will provide the General Assembly with a progress report that contains the forecast and cost information. Responsible entity: Contracts Administration. Target implementation date: March 1, 2014.

2. The Department should continually update providers and end users on the status of CSRs, to ensure an adequate understanding of the functionality of NCTracks.

DHHS Response: The Department currently sends all providers a weekly email (or more frequently,

if required) that contains detailed pertinent information about changes to NCTracks. This same communication is also distributed internally to management and staff associated with the NCTracks program. These messages address, among other things, functionality changes that may be directly related to CSRs.

While some CSRs may be directly relevant to providers and end users, others may relate only to internal system management. Additionally, the list of CSRs and the scope of each is somewhat fluid. Because of these limitations, the Department agrees to periodically update providers and end users on the status of CSRs, but only those CSRs that directly affect functionality that is relevant to providers and end users in their respective roles.

22

We appreciate the time and resources the State Auditor’s Office invested in this review as we work toward the common goal of better serving North Carolina Medicaid patients and medical providers. If you need any additional information, please contact Joe Cooper, Jr. at (919) 855-3060.

23 DHHS Response - Attachment 1

24 DHHS Response - Attachment 2

_____________________________________________________________________________________________ This audit was conduct ed in 609 hours at an approximat e cost of $46,284. The t ot al cost of t he audit

represent s .009% of t he t ot al NCTracks cost of $497,017,478.

ORDERING INFORMATION

Copies of this report may be obtained by contacting the: Office of the State Auditor

State of North Carolina 2 South Salisbury Street 20601 Mail Service Center Raleigh, North Carolina 27699-0601

Telephone: 919-807-7500 Facsimile: 919-807-7647 Internet: http://www.ncauditor.net

To report alleged incidents of fraud, waste or abuse in state government contact the: Office of the State Auditor Fraud Hotline: 1-800-730-8477

or download our free app

https://play.google.com/store/apps/details?id=net.ncauditor.ncauditor

https://itunes.apple.com/us/app/nc-state-auditor-hotline/id567315745

For additional information contact: Bill Holmes

Director of External Affairs 919-807-7513