4.1 Introduction
The Centers for Medicare & Medicaid Services (CMS) aim to develop quality measures of the highest caliber that will drive significant health care quality improvement and inform consumer choices. To gain CMS approval for measure implementation, the measure developer must first provide strong evidence that the measure adds value to existing measurement programs and that it is constructed in a sound manner. CMS gives preference to measures that are already endorsed by the National Quality Forum (NQF) or are likely to become endorsed for implementation in its programs. Therefore, measure contractors should develop measures that meet NQF evaluation criteria and are likely to be endorsed if they are submitted for endorsement.
Each proposed measure undergoes rigorous evaluation during the development process to determine its value and soundness based on a set of standardized criteria including—importance to measure and report, scientific acceptability of the measure properties, usability and use, feasibility, and in certain cases, harmonization. NQF requires measure harmonization as part of their endorsement and endorsement maintenance processes, placing it after initial review of the four measure evaluation criteria. Since harmonization should be considered from the very beginning of measure development, CMS contractors are expected to consider harmonization as one of the core measure evaluation
criteria. (Please refer to the Harmonization section for further information). Each criterion is comprised of a set of subcriteria. The measure developer uses an iterative evaluation process to build and
strengthen the measure justification and technical specifications to demonstrate the following: The aspects of care included in the specifications are highly important to measure and report
because the results can supply new, meaningful information to consumers and health care providers to drive significant improvements in health care quality and health outcomes where there is variation in or overall less-than-optimal performance.
The data elements, codes, and parameters included in the specifications are the optimum choices to quantify the particular measure because they most accurately and clearly target the aspects that are important to collect and report, and they do not place undue burden on resources in order to collect the data.
The calculations included in the specifications are optimum methodologies because they reflect a clear and accurate representation of the variation in the quality or efficiency of care delivered or the variation in the health outcome of interest.
This section (including its appendices) provides an overview of the measure evaluation criteria and subcriteria, and provides guidance for evaluating measures. A template for the measure evaluation report is included. The Measure Evaluation Report documents for CMS the extent to which the measure meets the criteria. It also documents any plans the measure contractor has to improve the rating, when a measure is rated low or moderate on any sub criterion. The measure contractor will use this report to document the pros and cons, cost benefit, and any risks associated with not further refining the measure. To facilitate efficient and effective development of high caliber measures, the materials in this section have been revised to reflect changes implemented by the National Quality
Forum (NQF) as of January 2012, as well as additional criteria developed by CMS for evaluating a measure set.
4.2 Deliverables
When recommending approval of candidate measures for further development—submit a separate measure evaluation report for each measure.
When recommending approval of fully tested and refined measures for implementation— submit a separate measure evaluation report for each measure.
4.3 Discussion of Measure Evaluation Criteria and Subcriteria
Importance to Measure and Report
High impact area
This criterion emphasizes that the specific measure focus (i.e., what is measured) should be considered important enough to expend resources for measurement and reporting according to CMS goals and priorities as well as recognized national health goals and priorities.
In addition to addressing national priorities, measures may be warranted for some other high impact aspects of health care. The subcriteria in the importance category address many issues related to these high impact aspects of care, such as: leading cause of morbidity/mortality, high resource use, severity of consequences of poor quality, number of people at risk, effectiveness of care, and opportunity for improvement. These aspects should be systematically evaluated by developing a business case for the measure. Not all topics are associated with financial savings to Medicare; however, a topic may be beneficial for society in general or may be of ethical value. The benefits derived from the interventions promoted by the quality measures should be quantified whether in terms of a business, economic or social case. (Refer to the Information Gathering section appendices for more information and
examples.)
Opportunity for improvement/gap in care
It is not enough that the measure is merely related to an important broad topic area. Evaluate whether the measure focus is a quality problem (i.e., there must be an opportunity or gap between actual and potential performance). Examples of opportunity for improvement data include, but are not limited to, prior studies, epidemiologic data, and measure data from pilot testing or implementation. If data are not available for review, the measure focus should be systematically assessed (e.g., expert panel rating) and evaluated relative to being a quality problem.
Generally, rare event outcomes do not provide adequate information for improvement or
discrimination. However, “never events” that are compared to zero are appropriate outcomes for public reporting and for quality improvement.
Evidence to support measure focus
Health outcomes are often the preferred focus of a measure because they integrate the influence of multiple care processes and disciplines involved in the care. Because multiple processes influence a health outcome, health outcomes generally do not require empirical evidence linking them to a known process or structure of care.1 For other (non-outcome) types of measures, there must be a high-to- moderate degree of certainty, as demonstrated by the evidence, that the measure focus is linked to positive outcomes. For intermediate outcome measures, process measures and structure measures, evidence should be evaluated on the quantity of studies, the quality of those studies, and the consistency in direction and magnitude of the net benefit of the body of evidence. When clinical practice guidelines are used to support the measure focus, the methodological rigor and review should be understood to ensure that the underlying evidence meets the quality, quantity and consistency requirements.
If the measure focus is a single step in a multi-step process, select the step with the greatest effect on the desired outcome. For example, although assessing immunization status and recommending immunization are necessary steps, they are not sufficient to achieve the desired impact on health status—patients must be vaccinated to achieve immunity. This does not preclude consideration of preventive screening interventions measures where there is a strong link with desired outcomes (e.g., mammography). This also does not preclude selecting a measure focus within a multi-step process that is consistent with the provider’s scope of practice.
NQF is currently restructuring its Consensus Development Process (CDP) to include a two-stage process.2 The proposed two-stage process will incorporate an early review and approval (1st stage) of measure concepts against the importance criterion before it moves to further stages of measure development (i.e. testing and specification). Potential related and competing measures would also be identified during this stage. If a measure is to be submitted for NQF endorsement, the importance and harmonization criteria are considered “must pass” before the measure will be given any further consideration, or allowed to advance to the 2nd stage. The 2nd stage of the CDP allows measure developers 18 months to complete testing and full specification of measures. At this stage, measures will be evaluated for scientific acceptability, feasibility and usability.