Part II: We Don’t Have to Incentivize Humanity
Finding a Path Away from the Perverse Incentives of Performance Measurement
By Ellen Schultz and Victoria Sale
In this article, the second of two parts, we build on co-author Victoria Sale’s personal experience shared in Part 1 to examine evidence showing that social sector performance measurement incentivizes a focus on efficiency over lasting relationships, and standardization over human caring, ultimately working against the people it is intended to benefit. We challenge policymakers, regulators, leaders, and innovators across the social sector to recognize the human cost of performance measurement and redesign how we measure performance in the social sector through shared power with service providers and recipients.
Driven by recognition of widespread and damning failures in the quality of healthcare and public education, the U.S. social sector has avidly embraced performance measurement as the accountability tool of choice. When used this way, performance measurement seeks to assess service providers such as doctors, teachers, and the hospitals and schools where they work by quantifying important aspects of their services.
The urge to incentivize better performance by holding service providers accountable for meeting quality standards has an appealing logic, particularly for those paying for services. Yet this logic rests on several questionable assumptions: that we know – and can agree on – what constitutes high quality service, that we can measure this quality standard through quantifiable metrics, and that tying these metric scores to penalties and rewards for service providers will ultimately benefit service recipients.
In Part 1 of this article, co-author Victoria Sale shared her experience caring for a patient we’ll call Angie within an intensive case management program. Struggling to provide the human connection and patience that Angie needed amid intense pressure to meet cost- and efficiency-focused metrics, Sale learned to recognize the human cost of performance measurement, for both service providers and recipients.
A Look in the Mirror
Angie’s case is one example of the unintended consequences of widespread performance measurement adoption in the U.S. social sector. Amid a national crisis of poor health outcomes and escalating healthcare costs, the American healthcare system has embraced performance measurement over the last 30 years. While sporadic efforts to measure healthcare quality in the U.S. date back as early as the mid-18th century, the view that measurement is essential to the delivery of high-quality care emerged in the U.S. amid the shift toward managed care in the 1990s.1 The adage, “you can’t improve what you don’t measure,” credited to renowned management consultant Peter Drucker, took on a reverence akin to gospel.
Today, the National Quality Forum’s catalog of performance measures boasts over 1100 metrics, of which more than 400 are endorsed for quality improvement, public reporting, and pay-for-performance uses. Public reporting initiatives such as the Federally-funded Hospital Compare website link institutional reputations to performance measure scores to try to incentivize better performance. Increasingly, the Centers for Medicare & Medicaid Services ties payment for healthcare services to performance measure scores, such as through Quality Payment Program.
Yet evidence raises many questions about the effectiveness, and negative consequences, of performance measurement in the social sector. Looking across the public service sector in the U.K., Eleanor Carter and Nigel Ball wrote in SSIR in June 2021 of the many perverse incentives they have observed in evaluating a range of Payment for Results contracts. These contracts pay social service providers based on performance metrics, typically outcome measures, selected by the U.K. national government. A 2017 review of healthcare-focused pay-for-performance programs in the U.S., U.K., and several other countries failed to show any substantial improvement in long-term patient outcomes associated with such programs, with only limited evidence that pay-for-performance improved some care processes.2 Although one recent large study showed significant improvements in the safety of U.S. hospital care over the last decade,3 a 2022 report from the U.S. Office of the Inspector General reported only marginal decreases in the rate of harm Medicare beneficiaries experienced during hospitalization in 2018 (12%), compared to 2010 (13.5%). Over the same time period, the U.S. made major investments in performance measurement: a 2016 study found that U.S. physician practices spent more than $15B annually on performance measure reporting.4
In public education, with an even longer history of systematic performance measurement efforts, the impact of measurement on education outcomes is similarly mixed. In a comprehensive review of decades of performance measurement efforts across the U.S. K-12 public education system, David Deming and David Figlio conclude that measurement is typically associated with modest gains in student achievement overall.5 While somewhat effective in narrowing achievement gaps in the lowest performing schools, the authors caution that the greater the incentives associated with performance measurement, the more likely for unintended negative consequences, including concentrating resources on students who are closest to the desired performance standard rather than on those who are lowest performing. They found that strong incentives also increase the incidence of gaming, like pushing lowest-performing students into disability classifications or suspending them from school on test days. The authors conclude that although schools generally respond strongly to performance measurement, the response is not always in line with policy intention. In other words, be careful what you incentivize.
When assessing the impact of performance measurement as an accountability tool, we must look not only at this modest improvement, but also at the human cost of such narrow focus on standardized measures of performance. It is time to look ourselves in the mirror and admit the mistake of assuming that incentives that work to improve outcomes in the business and manufacturing world would work for social systems, too. And we must acknowledge the ways that performance measurement has worked against the human connection, personal relationships, and caring that are an essential part of all social sector work.
Sharing Power in Performance Measurement Design
So what do we do instead? Measurement reform efforts often focus on implementing different metrics or relying on alternative data. But shifting only what we measure fails to address power imbalances in the measurement process itself. Implementing one-size-fits-all metrics that reflect the priorities and understanding of just a few stakeholders (the policymakers, researchers, and payers who typically design, develop, and implement performance measurement) ignores the experience and insight of the people most impacted by measurement: front-line service providers and the people they serve.
Rather than offer recommendations for measuring different processes or outcomes, we instead advocate for measuring differently. We call on leaders across the social sector to first recognize the harm caused by current approaches to performance measurement, and then share power with frontline service providers, and service recipients, to collectively redesign performance measurement.
Redesigned social sector performance measurement must make benefitting service recipients its first objective. Achieving this objective requires more than stated goals and good intentions. To benefit the people at the heart of their missions, schools, healthcare organizations, and human service agencies must design measurement in partnership with service providers and the patients, students, and families they serve. Together with system administrators, these partners must:
Collaborate in measurement design from the beginning, including the process of deciding what to measure, with clarity around how those metrics ultimately benefit service recipients.
Co-design how to collect data and then make data transparent and accessible to all impacted community members.
Make sense of metrics through an iterative and collaborative process that interprets data in light of real-world experiences and social, cultural, and historical context.
Decide together how to respond when performance metrics show room for improvement by ensuring service providers and recipients share power in designing and implementing solutions.
This kind of power sharing with front-line service providers and recipients is a radical departure from current performance measurement practice. But radical does not mean impossible.
Efforts to co-design measurement are already happening in healthcare, child welfare, and community development. Anti-racist, community-centered program and policy design approaches, like those proposed by Sonya Soni and colleagues,6 push us to uphold a standard beyond co-design to “community ownership” where those in power hold “equitable, as opposed to equal” power with community members. As part of the Well-being In the Nation (WIN) initiative, more than 100 communities, organizations, and community members collaborated to identify measures of well-being that span economic, health, food, transportation, and public safety sectors.
Co-author Ellen Schultz oversaw a series of pilot projects, funded with support from the Robert Wood Johnson Foundation, that developed and implemented measures of healthcare quality in partnership with patients and family caregivers. These projects demonstrated that partnerships among patients, family caregivers, clinicians, and researchers yielded innovative measures that all stakeholders found more meaningful. More recently, the Centers for Medicare and Medicaid Services has encouraged more widespread patient engagement in measure development efforts, though the extent of engagement in practice is unclear and decision making about performance measurement remains firmly in the hands of federal administrators. In contrast, development and implementation of a set of Indigenous Health Indicators, developed by and for the Swinomish Indian Tribal Community, demonstrates the potential for community-driven measurement efforts.7
Like other measure reform efforts, many of these examples focus primarily on questions of what to measure, with less attention on how policymakers and systems administrators use performance measurement or its impact on those delivering and receiving social services. Yet these examples also demonstrate some of the transformation possible when those most impacted by performance measurement begin to shape how we define and measure desired outcomes.
Avoiding perverse effects of performance measurement does not require throwing out performance measurement as an accountability tool. Tailoring performance measurement to the social sector does require acknowledging the harm caused by status quo practices and partnering with frontline service providers and those they serve to co-design a human-centered measurement system. We do not have to incentivize humanity; we are innately human. We simply have to stop creating systems that keep us from being who we are.
Notes
See Dennis McIntyre, Lisa Rogers, and Ellen Jo Heier, “Overview, History, and Objectives of Performance Measurement,” Health Care Finance Review, vol. 22, no. 3, p. 7-21, 2001 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4194707/
See Aaron Mendelson, et al., “The Effects of Pay-for-Performance Programs on Health, Health Care Use, and Processes of Care,” Annals of Internal Medicine vol. 166, no. 5, p. 341-353, 2017. https://doi.org/10.7326/M16-1881
See Noel Elderidge, et al., “Trends in Adverse Event Rates in Hospitalized Patients, 2010 - 2019”, Journal of the American Medical Association, vol. 328, no. 2, p. 173-183, 2022. doi: 10.1001/jama.2022.9600
See Lawrence P. Casalino, et al., “US Physician Practices Spend More Than $15.4 Billion Annually To Report Quality Measures”, Health Affairs, vol. 35, no. 3, 2016 https://doi.org/10.1377/hlthaff.2015.1258
See David J. Deming and David Figlio, “Accountability in U.S. Education: Applying Lessons from K-12 Experience to Higher Education”, Journal of Economic Perspectives, vol. 30, no. 3, p. 33-56, 2016 https://pubs.aeaweb.org/doi/pdfplus/10.1257/jep.30.3.33
See Sonya Soni, Jessica Mason, and Jermeen Sherman, "Beyond Human-centered Design: The Promise of Anti-racist Community-centered Approaches in Child Welfare Program and Policy Design." Child Welfare, vol. 100, no. 1, p. 81-109, 2022.
See Jamie Donatuto, Larry Campbell, Robin Gregory, “Developing Responsive Indicators of Indigenous Community Health”, International Journal of Environmental Research and Public Health, vol. 13, no. 9, p. 899, 2016.https://www.mdpi.com/1660-4601/13/9/899
Acknowledgements
We are grateful to Rachel Davis, Rebecca Sax, and Gwynn Sullivan, and for making time to review and critique an earlier version of this article. We also thank Jason Turi for introducing us to one another - your spark lit a fire for us both.
*Shout out Wylly Suhendra for taking the the beautiful cover photo found on Unsplash here. Attribution is important.
