EHR Conformance Testing for Enhanced
Syndromic Surveillance System Interoperability
2013 International Society for Disease Surveillance Conference
Translating Research and Surveillance into Action, New Orleans.
Robert Snelick, National Institute of Standards and Technology (NIST)
Charlie Ishikawa, International Society for Disease Surveillance (ISDS)
Rebecca Zwickl, International Society for Disease Surveillance (ISDS)
Sheryl Taylor, Booz Allen Hamilton (BAH)
Objectives
•
Present an overview of the Meaningful Use Certification
process
–
Focus on the NIST/ISDS/CDC Collaboration to build the certification test
procedure, test cases, and test tool
•
Identify key aspects of the Syndromic Surveillance ONC
criterion and test procedure
•
Describe the Test Tool
–
Describe context-free and context-based testing approaches
–
Present the test scenarios and test cases
–
Illustrate the use of the certification test tool
•
Report Syndromic Surveillance certification testing results
Outcomes
•
Improved Syndromic Surveillance Standard
–
Feedback provided from extracting and encoding requirements
•
Test Scenarios and Test Cases
–
Real world scenarios and data that demonstrate the use of the standard
–
Examples messages that are conformance to the standard
•
Validation Suite
–
Syntactic and Semantic Testing
–
Is being used
•
Nationally by the ATLs to test EHR technologies for meaningful use certification
•
EHR vendors while developing their technologies
•
Locally by healthcare providers to validate electronic messages before sending
their Syndromic Surveillance system data to Public Health Authorities
Syndromic
Surveillance
Implementation
Guide
Test
Cases
ONC Criterion
Conformance
Requirements
Use Cases
TestStories Data Test
Test Script
Test
Procedure
ATLs
Post
CEHRT
on
CHPL
ONC
ISDS SMEs
NIST
Test
Tool
ATLs
Certification
Testing
ACBs
Certification
Test Tool
Local
Site
Testing
NIST
Test
Cases
NIST
Test
Procedure
ISDS/CDC
Addenda & Clarification Documents FeedbackNIST
ISDS – International Society for Disease Surveillance CDC – Centers for Disease Control and Prevention NIST – National Institute of Standards and Technology ATL – Accredited Testing Laboratory
ACB – Authorized Certification Body ONC – Office of the National Coordinator for Health Information Technology SME – Subject Matter Expert CEHRT – Certified Electronic Health Record Technology
CHPL – Certified Health IT Product List
Overview: Meaningful Use Certification Process
Syndromic
Surveillance
Implementation
Guide
Test
Cases
ONC Criterion
Conformance
Requirements
Use Cases
Test Stories Test DataTest Script
Test
Procedure
ATLs
Post
CEHRT
on
CHPL
ONC
ISDS SMEs
NIST
Test
Tool
ATLs
Certification
Testing
ACBs
Certification
Test Tool
Local
NIST
Test
Cases
NIST
Test
Procedure
ISDS/CDC
Addenda & Clarification Documents FeedbackNIST
NIST/ISDS/CDC
Collaboration
What we are
Overview: Meaningful Use Certification Process
Test Stories
Syndromic Surveillance Certification Criteria and
Testing
•
For certification, message creation function
must conform to standards
–
Capability Testing (not site testing)
•
2014 Certification: (
NIST Test Tool Target
)
–
“Inpatient” EHR Technology = Hospital (e.g., ED)
•
HL7 2.5.1 format
•
PHIN Guide for Syndromic Surveillance – Release 1.1
•
PHIN Addendum to Release 1.1 – Conformance Clarification
•
PHIN Testing Clarification Document
–
“Ambulatory” EHR Technology = Outpatient (e.g.,
EP’s, Urgent Care)
•
HL7 2.5.1 format
(Mandatory, but no Test Tool)
•
PHIN documents
(Optional, use Test Tool as above)
•
2011 Certification: HL7 2.3.1 or 2.5.1 format
–
Too broad; no conformance test tool)
•
Test Procedure
–
Scenario-based
–
Message focused (
A01, A03, A04, A08
)
–
“Inpatient” procedure
•
Three different scenarios, three test cases each
–
“
Ambulatory” procedure
Why is Testing Important?
•
Technology standards frequently implemented with
inconsistencies due to
–
Guidance ambiguities
–
Misinterpretations
•
Feedback Loop
•
Provides confidence that implementation is correct
•
Mandating a standard does not ensure
–
Use of standard as intended
–
Conformant Systems
–
Interoperability of EHR Syndromic Surveillance data among
systems
•
Conformance testing to the standard early in technology
development life cycle can
Key Capabilities
Context-free Testing
(No Test Cases - Test any Syndromic Surveillance message created by EHR
)
•
Context is unknown to validation tool
•
Provides a simple and convenient method for testing message structure and most vocabulary
Context-based Testing
(Test Cases - Test Syndromic Surveillance message associated with a specific test scenario)
•
Context is known to validation tool
•
All conformance requirements of the SS Implementation Guide can be assessed
•
Used for certifying 2014 Edition Meaningful Use EHR technology
Profile Viewer
Provides a browsable version of the conformance profile which encapsulates the requirements. Can
be used to assist in the interpretation of errors.
Vocabulary Browser
Provides a browsable view of the vocabulary requirements. Can be used to assist in the
interpretation of value set errors.
Documentation
Provides access to documents which will assist in using the tool (including test procedure, test
cases, profile descriptions, vocabulary descriptions and validation tool download).
Purpose: This tool validates Syndromic Surveillance messages created by Electronic Health
Records (EHR) technology.
Context-free Testing
•
No Test Cases provided
•
Context (Test Scenario, etc.) is
unknown to validation tool
•
May be used to test any syndromic
surveillance message created by an
EHR
•
Provides a simple and convenient
method for testing message
structure
and most
vocabulary
•
Typically is not used for certifying
EHR technologies for the ONC
certification criteria, but may be
used for certification testing in
specific instances (the Tester must
perform visual inspection to validate
content of message)
Syndromic Message Testing - Context-free
Validation
1. Click on the Context-free Validation tab.
2. Select profile to use to validate the message.
3. Browse for message to be validated and import it.
4. Validate the message. 5. Syndromic Test
Tool produces Validation Report.
6. Tester analyzes Validation Report.
Context-free Validation page
This page validates any Syndromic message. It is
disassociated from a test script, test case, or specific content.
Testing will include the technical requirements and
content-specific requirements specified in the selected profile.
Open Syndromic Surveillance Validation tool using link:
http://hl7v2-ss-testing.nist.gov.
1
Select conformance profile to use to validate the message.
3
Click on Browse Message button to load message into Message
Content window.
4
Click on Context-free Validation tab.
2
Message Uploader Dialog window displays.
Click on Select Message button.
5
Open test message file to upload it.
6
7
Validate test message and review message validation
errors
If message fails validation, errors will display in Message
Validation Result section of page.
2
Uploaded test message displays.
1
Description explains why error occurred. Format or table of data
element may be provided.
5
Line number and column of errors are
provided.
6
Location link displays the location of the data
element.
7
Validate test message and review message validation
errors
(cont’d)
Click on page numbers to view additional pages of errors.
8
Total number of errors displays.
4
Locate data element
5
Profile Viewer - look up data element information
Data elements may be filtered by Usage. Select R, RE, C (Only) to view only required, required
but may be empty and conditional elements. Select R, RE, C, O, X (All) to
view all data elements, including optional and not
supported elements.
Click on a tab to filter data elements by segment. The Full tab displays data elements of all segments.
3
4
Click on Profile Viewer tab.
1
Click on a tab to select the conformance profile.
2
Click on Vocabulary tab.
9
Locate and select table ID.
10
Valid values are listed.
11
Click on Context-free Validation tab to return to message.
13
Vocabulary - look up tables and valid data element values
(cont’d)
Vocabulary page
This page provides the ability to
browse the vocabulary requirements.
Search capabilities are provided and
include searching on value, table
name, table ID and description.
Use the search box to search for value code, value set, table name, table ID or description.
12
Documentation
The Documentation page provides the ability to download all
Documentation page
documents and files for testing, including Data Spreadsheet,
Test Cases, Profile and Vocabulary Description files, and
validation tool application file (.war file).
HL7 V2
Message
Context-based Testing (Sending Application)
Test
Story
Sets the clinical
backdrop
Test
Data
Typical data
Test Case
EHR
Validation
Test Tool
Validation
Report
Implementation
Guide
Requirements
Tester
Enters Test
Data
Context-based Testing
•
Test Cases provided
•
Context (specific Test Scenario, etc.) is known to validation tool
•
Messages created by the EHR technology are populated using the
test data provided in the Test Cases associated with specific Test
Scenarios
•
Provides a method for testing / assessing a syndromic surveillance
message for all conformance requirements of the Syndromic
Surveillance Implementation Guide
•
Is used for certifying EHR technologies for the ONC certification
criteria
•
Significantly expands the scope of testing
–
Usage: Required, but may be empty (RE), Conditionals (C)
–
Cardinality: Ranges
–
Length: Ranges
–
Vocabulary
Syndromic Test Scenario Matrix
UC Visit
Pt Death Pt Admit
ADT A04
– Registration
X
X
X
ADT A03
– Discharge
X
X
X
ADT A08
– Update
X
X
ADT A01
– Admit
X
Facility ID
is from the treating facility
X
X
X
Report Date/Time
is the date/time that information was sent from
the treating facility; will vary among messages for a visit
X
X
X
Visit Date/Time, Unique Visit ID,
and
Unique Patient ID
are
consistent among messages for a single visit
X
X
X
Patient Age
is sent in years
X
X
X
Free-text
Chief Complaint
can be sent
X
X
If patient presents unconscious,
patient demographics
(e.g.,
Race, Ethnicity, Zip Code, Age, and Patient's Chief Complaint)
are null or a default value
X
A coded
Reason for Visit
can be sent
X
When multiple
Diagnoses
are made, the primary is sent first
X
X
X
Diagnosis
ECODEs
can be sent
X
X
Test Scenario #1 – Routine Urgent Care
ü
ADT A04 and A03
ü
Free-text chief complaint
A.
Infant with influenza-like-illness
B.
Adult with a urinary tract infection
C.
Adult with broken arm
Test Scenario #1 Messages
Test Scenario #2 – Patient Death
ü
ADT A04, A08, and A03
ü
No demographic data or chief complaint in A04
ü
Diagnoses include ECODE
ü
Patient death sent in final message
A.
Skill fracture w/ intracranial injury from bike accident
B.
Acute respiratory failure due to house fire
C.
Subarachnoid hemorrhage due to fall
Test Scenario #2 Messages
Test Scenario #3 – Patient Admission
ü
ADT A04, A08, A03, and A01
ü
Free-text chief complaint
ü
Hospital admission
A.
CO-poisoning during severe whether power outage
B.
Critical dehydration due to cryptosporidium outbreak
C.
Pneumonia during flu season
Test Scenario #3 Messages
Syndromic Certification Testing - Context-based
Validation
Click on Context-based Validation tab.
2
Click on arrows to expand the first Test Scenario. Expand the Test Cases and select a
Test Case.
3
Test Case displays.
4
Test Case may be downloaded as a PDF file.
5
Select Test Case
Open Syndromic Surveillance Validation tool using link:
http://hl7v2-ss-testing.nist.gov.
1
Select Step 1 of Test Case.
1
Test Story displays.
2
Step 1 Test Story may be downloaded as a PDF file.
3
Step 1 Test Package may be downloaded as a PDF file.
4
Test Data Specification page
This page specifies the data that are entered
(automatically/manually) into the EHR and
included in the message that is created and
Click on Test Data Specification tab.
1
Review Test Data Specification and Message Content
Test Data Specification may be downloaded as a
PDF file.
4
Full tab is selected by default. To view a specific section, click on
a tab.
Relevant real-world clinical data is displayed.
2
Using the EHR technology, create the Syndromic Surveillance test message with the test data provided for the selected Test Step (step 3).
1
Review Test Data Specification and Message Content
(cont’d)
Create test message
Name of data element is provided.
7
Exact value of the data element for the selected Test Step is provided.
8
Categorization indicates if the data is fixed or can be changed.
9
Location specifies the location of a data element within the
message.
6
Click on Message Content tab.
5
Message Content page
This page provides details of the message
segment, as well as an example of a valid
instance of each data element.
Message Content may be downloaded
as a PDF file.
10
Validation page displays.
Click on Browse Message button.
2
Message Uploader Dialog window displays.
3
Click on Select button to load the Test Step.
Click on Select
4
5
Load Test Step and import test message
If message fails validation, errors will display in Message Validation
Result section of page.
2
Uploaded test message displays.
1
Click on location link to highlight the data element causing the error within
the Message Tree and Content. Note: Location link may not be available if the message element location does not map to a message
element in the message tree.
3
Validate test message and review validation errors
(cont’d)
Description explains why error occurred. Format or table of data
element may be provided.
5
Line number and column of errors are
provided.
6
Location link displays
7
Total number of errors displays.
4
Message Validation Report may be downloaded as a PDF, XML,
Word doc, HTML file.
2
Message Validation Report may be printed.
3
Click on Report Details icon to generate a report.
1
Results
•
As of November 2013
–
48 vendors/organizations certified EHR technologies for Syndromic
Surveillance messaging
–
ATLs tested these EHRs using NIST Syndromic Surveillance Test Tool
–
Certified EHR Technologies are listed on the ONC Certified Health
IT
Product List (CHPL) at:
http://oncchpl.force.com/ehrcert
•
Feedback about Test Tool indicates EHR technology vendors and
other groups
–
Have been motivated to think through the steps for capturing/reporting
Syndromic data in more detail than before
–
Are challenged by the new Syndromic Surveillance data requirements and
standards
–
Appreciate the subject matter expertise that generated the tools
Vendors/Organizations with Certified EHR
Technologies
As of November 2013
Agastha, Inc.
Allscripts
AmkaiSolutions LLC
Plexus Information Systems, Inc.
athenahealth, Inc
General Electric Healthcare IT
Dynamic Health IT, Inc.
Corepoint Health
CPSI (Computer Programs and Systems), Inc.
Henry Schein Practice Solutions, Inc.
DigiDMS, Inc.
eClinicalWorks LLC
EMD Wizard, Inc.
Epic Systems Corporation
T-System Technologies, Ltd.
gMed, Inc.
Glenwood Systems LLC
Greenway Medical Technologies, Inc.
Healthcare Management Systems, Inc.
Medflow, Inc.
SuccessEHS, Inc., a Division of Vitera Healthcare Solutions
LSS Data Systems
Advanced Data Systems Corporation
Medical Information Technology, Inc. (MEDITECH)
HealthFusion
Henry Schein Medical Systems, Inc.
Insight Software, LLC
NextGen Healthcare
Northwestern University
Bogardus Medical Systems, Inc.
NTT DATA, Inc.
Aprima Medical Software, Inc.
Iatric Systems, Inc
Pulse Systems
Orion Health
SRSsoft
Ulrich Medical Concepts, Inc.
