Scientific Acceptability of Measure Properties —Reliability, Validity, Disparities
3. Usability
Instructions: If the measure is to be submitted to NQF for endorsement, the measure must be judged to pass both
subcriteria 3a and 3b. A measure must be rated High or Moderate both public reporting and quality improvement usability to pass
Outcome Measures: An important outcome that may not have an identified improvement strategy still can be useful for
informing quality improvement by identifying the need for and stimulating new approaches to improvement and can be rated “High” or “Moderate”.
Subcriterion High Moderate Low
3a.
Public Reporting
Testing demonstrates that information produced by the measure is meaningful, understandable, and useful for
public reporting (e.g., systematic
feedback from users, focus group, cognitive testing).
Formal testing has not been performed, but the measure is in widespread use and you think it is meaningful and understandable for public reporting (e.g., focus group, cognitive testing)
OR
When measure is being rated during its initial development:
A rationale for how the measure performance results will be meaningful, understandable, and useful for public reporting.
The measure is not in use and has not been tested for usability;
OR
Testing demonstrates information produced by the measure is not
meaningful, understandable, and
useful for public reporting
OR
When measure is being rated during its initial development:
A rationale for how the measure performance results will be meaningful, understandable, and useful for public reporting is not provided.
3b.
Quality Improvement
Testing demonstrates that information produced by the measure is meaningful, understandable, and useful for
quality improvement (e.g.,
systematic feedback from users, analysis of quality improvement initiatives).
Formal testing has not been performed but the measure is in widespread use and accepted to be meaningful and useful for
quality improvement (e.g.,
quality improvement initiatives).
OR
When measure is being rated during its initial development:
A rationale for how the measure performance results will be meaningful, understandable, and useful for quality improvement.
The measure is not in use and has not been tested for usability;
OR
Testing demonstrates information produced by the measure is not
meaningful, understandable, and
useful for public reporting
OR
When measure is being rated during its initial development:
A rationale for how the measure performance results will be meaningful, understandable, and useful for quality improvement is
Guidelines for Summary Rating: Usability
Summary Rating: Usability
Pass: The measure rates “Moderate” to “High” on both aspects usability Fail: The measure rates “Low” for one or both aspects of usability
4. Feasibility.
Subcriterion High Moderate Low
4a.
Byproduct of care (clinical
measures only)
The required data elements are routinely generated concurrent with and as a byproduct of care processes during care delivery (e.g., BP reading, diagnosis).
The required data are based on information generated during care delivery; HOWEVER, trained coders or abstractors are required to use the data in computing the measure.
The required data are not generated during care delivery and are difficult to collect or require special surveys or protocols.
4b.
Electronic data
The required data elements are available in electronic health records or other electronic sources.
If the required data are not in electronic health records or existing electronic sources, a credible, near-term path to electronic collection is specified.
The required data elements are not available in electronic sources
AND
There is no credible, near-term path to electronic collection specified. 4c. Inaccuracies, errors, or unintended consequences Susceptibility to inaccuracies, errors, or unintended consequences related to measurement are judged to be inconsequential or can be minimized through proper actions
OR can be monitored and
detected
There is moderate susceptibility to inaccuracies, errors, or unintended consequences,
AND/OR
They are more difficult to detect through auditing.
Inaccuracies, errors, or
unintended consequences have been demonstrated,
AND
They are not easily detected through auditing.
4d.
Data collection strategy
The measure is in operational use and the data collection strategy (e.g., source, timing, frequency, sampling, patient confidentiality, etc.) has been implemented without difficulty.
The measure is not in operational use; HOWEVER testing
demonstrates the data collection strategy can be implemented with minimal difficulty or additional resources.
The measure is not in operational use,
AND
Testing indicates the data collection strategy was difficult to implement and/or requires
substantial additional resources.
Guidelines for Summary Rating: Feasibility
Summary Rating: Feasibility
High rating indicates: Three or four subcriteria are rated “High”.
Moderate rating indicates: “Moderate” or mixed ratings, with no more than one “Low” rating. Low rating indicates: Two or more subcriteria are rated “Low”.
Harmonization
Instructions: Measures should be assessed for harmonization. Either the measure specifications must be harmonized with
related measures so that they are uniform or compatible or the differences must be justified (e.g., dictated by the
evidence). The dimensions of harmonization can include numerator, denominator, exclusions, calculation, and data source and collection instructions. The extent of harmonization depends on the relationship of the measures, the evidence for the specific measure focus, and differences in data sources. Harmonization should not result in inferior measures—measures should be based on the best measure concepts and ways to measure those concepts. There is no presupposition that an endorsed measure is better than a new measure.
Completely Partially Not Harmonized
The measure specifications are
completely harmonized with related
measures; the measure can be used at multiple levels or settings/data sources
The measure specifications are
partially harmonized with related
measures, HOWEVER the rationale justifies any differences; the measure can be used at one level or setting/data source
The measure specifications are not
harmonized with related measures AND
the rationale does not justify the differences
5. Harmonization
Instructions: Measures should be assessed for harmonization. Either the measure specifications must be harmonized with
related measures so that they are uniform or compatible or the differences must be justified (e.g., dictated by the
evidence). The dimensions of harmonization can include numerator, denominator, exclusions, calculation, and data source and collection instructions. The extent of harmonization depends on the relationship of the measures, the evidence for the specific measure focus, and differences in data sources. Harmonization should not result in inferior measures—measures should be based on the best measure concepts and ways to measure those concepts. There is no presupposition that an endorsed measure is better than a new measure.
Completely Partially Not Harmonized
The measure specifications are
completely harmonized with related
measures; the measure can be used at
The measure specifications are
partially harmonized with related
measures, HOWEVER the rationale
The measure specifications are not
Completely Partially Not Harmonized
multiple levels or settings/data sources. AND
There are no competing measures (already endorsed, in development, or in use)
justifies any differences; the measure can be used at one level or setting/data source
AND
the rationale does not justify the differences
OR
There is a competing measure (same focus, same population)
4b.1 Measure Evaluation Criteria and Subcriteria for Composite Measures
Adapted from National Quality Forum Measure Evaluation Criteria7 and Composite Measure Evaluation Framework8
1. Impact, Opportunity, Evidence—Importance to Measure and Report:
Extent to which the specific measure focus is evidence-based, important to making significant gains in health care quality, and improving health outcomes for a specific high-impact aspect of health care where there is variation in or overall less- than-optimal performance.
If the component measures are determined to meet subcriteria 1a, 1b, and 1c, then the composite would meet 1a, 1b, and 1c. If a component measure does not meet 1a, 1b, and 1c and is not important enough in its own right as an individual measure, it could still meet the Importance to Measure and Report criterion if it is determined to be an important component of a composite and meets subcriteria 1d and 1e.
1a. High Impact
For each component measure of the composite measure, the measure focus addresses:
A specific national health goal/priority identified by the Department of Health and Human Services (HHS) or the National Priorities Partnership convened by NQF;
OR
A demonstrated high-impact aspect of health care (e.g., affects large numbers of patients and/or has a substantial impact for a smaller population, leading cause of morbidity/mortality, high resource use (current and/or future), severity of illness, and severity of patient/societal consequences of poor quality).
AND