JAMA Forum: If You Can’t Measure Performance, Can You Improve It?

By Robert A. Berenson, MD

Robert A. Berenson, MD (Image: Urban Institute)

Robert A. Berenson, MD (Image: Urban Institute)

“If you can’t measure it, you can’t manage it” is an often-quoted admonition commonly attributed to the late W. Edwards Deming, a leader in the field of quality improvement. Some well-respected health policy experts have adopted as a truism a popular variation of the Deming quote—“if something cannot be measured, it cannot be improved”—and point to the recent enactment of the Medicare Access and CHIP Reauthorization Act of 2015 (MACRA) as a confirmation of “the broadening societal embrace” of this concept.

The problem is that Deming actually wrote, “It is wrong to suppose that if you can’t measure it, you can’t manage it—a costly myth” (my emphasis added)—the exact opposite. Deming consistently cautioned against requiring measurement to guide management decisions, observing that the most important data needed to manage often are unknown and unknowable.

Critics of policy makers’ infatuation with reliance on performance measures to support public reporting and “pay for performance” rewards or penalties to clinicians and health care facilities offer a quotation attributed to their own heavy hitter, Albert Einstein: “Not everything that can be counted counts, and not everything that counts can be counted.” If you Google this quote, you will find dozens of images of the learned professor linked with this quotation, but there is a problem here also: sociologist William Bruce Cameron apparently penned this in 1963, years after Einstein’s death.

So much for “evidence-based policymaking”—we can’t even get quotations right. No wonder there is such disagreement over the effect of Obamacare.

Many Routes to Improvement

The requirement for measurement as essential to management and improvement is a fallacy, not a self-evident truth and not supported by Deming, other management experts, or common sense. There are many routes to improvement, such as doing things better based on experience, example, as well as evidence from research studies.

Surely public reporting of performance has changed medical culture for the better, leading to a growing acceptance that the quality of clinical practice does not depend on the unmeasurable “art of medicine.” Comparative public performance using meaningful and accurate measures has led to quality improvements, as clinicians and hospitals reflect on their own comparative performance and seek to improve their public standing. Examples include improved hospital care for patients experiencing heart attacks and improved renal dialysis. In most clinical areas, however, we lack readily available measures to use as valid benchmarks to assess performance.

Not deterred, however, last year a rarely bipartisan Congress passed the MACRA legislation.  Its core element was repealing the unsustainable sustainable growth rate mechanism threatening huge payment cuts to physicians caring for Medicare patients. The law called for development of “value based” payment approaches that would pay for quality and cost outcomes, rather than just for the myriad services physicians provide or order, whether or not the services are needed or well performed. “Paying for value, not volume” has become the slogan du jour, itself assuming a mostly unchallenged position in health policy circles.

Now comes the hard part: actually achieving greater value, rather than fashioning an increasingly complex, intrusive, and likely doomed attempt to measure value.   

After the MACRA’s Merit-Based Incentive Payment System (MIPS) is fully phased in early in the next decade, a physician caring for Medicare patients under MIPS stands to lose up to 9% of their Medicare payments or conceivably gain 27%, based on their performance on measures of quality, their use of health care resources, the extent to which they have implemented electronic health records, and their participation in quality improvement activities.

MIPS is an outgrowth of a decade of smaller pay-for-reporting and pay-for-performance programs. Realizing that physicians basically ignored the small rewards and penalties limited to 2% of Medicare physician payments, Congress raised the financial stakes enormously, making sure physicians pay attention—an approach that brings to mind the Catskills-era quip, “The food here is terrible, and the portions are too small.”

Improving physician performance on particularly significant health problems amenable to accurate measurement would benefit from application of a few measures, such as physicians’ performance in controlling blood pressure in the millions of patients with inadequately controlled hypertension. But Congress in MACRA has a different purpose. Within a few years, MIPS will publish a performance scorecard for each physician participating in Medicare.

But performance on a few, random and often unreliable measures of performance can provide a highly misleading snapshot of any physician’s value. So it’s no surprise that only about half of physicians participate.

MACRA’s bipartisan consensus included the House GOP Doctors Caucus, where 17 of its 18 members voted for legislation requiring the Centers for Medicare & Medicaid Services to rank physicians in the country based on its calculation of their value. Having government rate physicians would be a step too far even if we had important and valid measures of physician performance.

A Bad Idea?

Practical challenges aside, pay for performance for health professionals may simply be a bad idea. Behavioral economists find that tangible rewards can undermine motivation for tasks that are intrinsically interesting or rewarding. Furthermore, such rewards have their strongest negative impact when they are perceived as being large, controlling, contingent on very specific task performance, or associated with surveillance, deadlines, or threats, as with MIPS.     

Another major problem with the current preoccupation with measurement as the central route to improvement is the assumption that if a quality problem isn’t being measured, it basically doesn’t exist. A prime example is diagnosis errors. Recently, an Institute of Medicine (IOM) committee, on which I was a member, issued Improving Diagnosis in Health Care, documenting serious errors of diagnosis in 5% to 15% of interactions with the health care system.

As the report emphasizes, we cannot now measure the accuracy of diagnoses, which means MIPS scores will not include performance on this core physician competency. Still, the IOM committee proposed numerous improvement strategies. These include development of immediate feedback programs to erring clinicians from patients and other health professionals when a serious misdiagnosis occurs (making errors memorable if not measureable), greater attention in medical education to the cognitive bias that commonly clouds clinicians’ judgment, improved systems to ensure that abnormal test results are promptly communicated to patients and diagnostic team members, and giving patients direct access to their medical records so they can introduce relevant, missing information and correct the misinformation that is common in clinical records.

These and other IOM recommendations represent better practices that might dramatically improve diagnostic accuracy, relying not on performance measures but on adopting better work processes and focused education. Measures would help, but substantial progress can be made regardless.

The overarching concern is that under MIPS and similar programs, physicians will focus on the money while their intrinsic motivation to make accurate, timely diagnoses as a core responsibility will be crowded out. If so, the worthwhile recommendations in the IOM report will likely sit on the shelf, gathering dust, thanks to the misguided supposition that “if you can’t measure it, you can’t manage it.”


About the author: Robert A. Berenson, MD, is an institute fellow at the Urban Institute. An internist who practiced 20 years, he has served in various government positions, including Assistant Director of the White House Domestic Policy Staff under President Carter, Director of the Center for Health Plans and Providers in the Centers for Medicare & Medicaid Services in the Clinton Administration, and Vice Chair of the Medicare Payment Advisory Commission. He graduated from Brandeis University and received his MD from the Mount Sinai School of Medicine.

About the JAMA Forum: JAMA has assembled a team of leading scholars, including health economists, health policy experts, and legal scholars, to provide expert commentary and insight into news that involves the intersection of health policy and politics, economics, and the law. Each JAMA Forum entry expresses the opinions of the author but does not necessarily reflect the views or opinions of JAMA, the editorial staff, or the American Medical Association. More information is available here and here.

Categories: Health Policy, The JAMA Forum

Tags: , ,