Geographic variation in health care spending and the ecological fallacy

by Academyhealth on January 31, 2013 · 6 comments

I don’t spend much time in D.C. so I rarely know what, if any, research papers are making waves, however small. But I got a tip that Louise Sheiner’s “Why the Geographic Variation in Health Care Spending Can’t Tell Us Much about the Effciency or Quality of our Health Care System” (PDF) was getting some attention. From the title alone, I can see why.

It’s an attempted takedown of much of the Dartmouth work on regional spending variations. There’s a lot in it, a great deal of which I don’t agree with, and more than I could cover in one blog post. Fortunately, the key results hinge on one methodological choice, offering an opportunity for a parsimonious response. That methodological choice will be my focus.

Backing up a bit, the key question is this: to what extent is geographic variation in health care spending a signal of variation in health versus variation in health systems and practice patterns? Or, more simply, is it the patients or the providers that drive spending differences? Finding out is important because it suggests different policy interventions. If spending variation is largely due to regional differences in underlying health, then we need to focus on causes of poor health, including lifestyle factors, but also access to health care, public health investment, and educational and work opportunities.

If the variation is largely due to health systems and practice pattern differences, then we need to focus more on things like putting evidence into practice, payment incentives to motivate change, and fostering organizational structures that promote high quality, yet efficient, care.

For a long time, and over many publications, Dartmouth researchers, and others, have suggested organizational inefficiencies are the chief culprit, though variations in health play a role. Even controlling for underlying health, a lot of variation in health spending remained unexplained in their work. Meanwhile, large variations – correlated with those in spending – in how systems operated and how doctors practiced were observed. Though not everyone agrees, based on this body of work, many health systems experts believe that a lot of wasteful, overuse of care is driven by patterns of practice.

Enter Sheiner who says this is all wrong. In a cross-sectional, state-level analysis, she finds that almost all the variation in Medicare spending can be explained by just a few aggregate measures of health and well being.

[T]he combination of per capita income and age distributions explains only about 30 percent of the variation in acute Medicare spending across states. However, including measures of health—in particular, the obesity rate in the state and the percent sedentary, increases the explained share of spending to 48 percent. Adding in the percent uninsured raises the explained share of spending to 75 percent, and adding in the percent black raises it to 81 percent.

With so much of spending variation – 81 percent of it – explained by age distribution, per capita income, percent uninsured, rates of obesity and sedentary lifestyles, there’s not much left to be explained by organizational factors and practice pattern variation. Did Dartmouth get it all wrong?

No.

Sheiner’s findings are due to her selection of unit of analysis, the state. Much of the Dartmouth work over the last decade is at the individual level. This makes all the difference. That one can obtain disparate results due to level of aggregation alone has a name: the ecological fallacy. Responding to Sheiner, Dartmouth researchers Jonathan Skinner and Elliot Fisher wrote,

Why should individual-level risk-adjustment data be preferred to state-level data? The answer is: the ecological fallacy problem. It occurs when a researcher makes an inference about individual behavior based on group-level averages. One example is the following: In 2004, the Republican candidate, George W. Bush, won the 10 states with the lowest income, while the Democratic candidate, John Kerry won 9 of the 10 highest-income states. Thus one might conclude from the state-level data that low income voters tend to vote Republican and high-income voters vote Democrat (with a stunningly high R2). Yet as Gelman and colleagues have noted, high income voters are more than 10% more likely to vote Republican than are low-income voters – precisely the opposite pattern. Only by use of individual level data is the extent of the ecological fallacy revealed. [Hyperlink added.]

For this reason, we should question Sheiner’s conclusions. Indeed, Richard Kronick and Todd Gilmer have documented the role that level of aggregation plays in analyses of health spending variation. Interestingly, in an individual-level analysis of correlates of Medicare spending variation, Stephen Zuckerman and colleagues (PDF) found that some of the key factors that explained variation in Sheiner’s state-level analysis – income, most measures of race, and BMI – were not statistically significant. More generally, there’s just no reason to believe aggregate measures that explain a lot of variation at the state level are the same factors that drive variation at the level at which health care and any intervention thereof ultimately takes place: the individual. Consequently, Sheiner’s state-level findings in no way invalidate the individual-level analyses by Dartmouth scholars and others.

I wish Sheiner’s conclusions could be believed. It would really simplify things if the vast majority of health spending variation was due to obesity, sedentary lifestyles, lack of insurance, and a few other things. Were that true, we’d know better how to shape policy. However, based on the body of evidence, it’s far more likely the story is more complex and also includes the efficiency with which care is delivered.

Delivery system variation still matters, despite what they read in the papers in D.C.

Austin

Share

{ 3 comments }

Buz Cooper January 31, 2013 at 3:43 pm

It is true that the Federal Reserve paper focused on states and it would have been better to also disaggregate them into smaller units. But one of the strengths of the Federal Reserve paper is that it distinguished between Medicare and state-based resources. This serves to highlight a major deficiency in all of the Dartmouth work, which uses Medicare data alone, a point that I made in a paper in Health Affairs 2009;28(1):w103-15. For reasons mentioned below, the distribution of Medicare expenditures is anomalous. The Federal Reserve paper also clarified the conceptual error that results from the use of residuals as a statistical tool, as was the case in state studies by the Dartmouth group.

Unfortunately, the smaller unit of analysis deployed in most of the Dartmouth research (HRRs), is artificial and inconsistent, neither large enough (like states) to reveal characteristics of a coherent economic unit nor small enough (like Zip codes) to reveal something about diverse populations. As a result, their HRRs are unable to explain the variation that they see, so by default they attribute it to waste. However, when HRRs are disaggregated into ZIP codes, the “unexplained variation” (aka waste) is explained. It is a manifestation of gradient poverty, as discussed in a paper that my colleagues and I published in the J Urban Health 2012;(89(5)848-47.

Nonetheless, state level data are important, as I discussed in a Commentary in JAMA. 302:1113-13, 2009. States are important because it is their wealth that sets the parameters, just as the wealth of countries in the OECD sets their parameters. The difference is that the parameters among states are narrowed by federal transfers between them, mainly through Medicare and Medicaid, whereas countries are on their own.

Once the parameters are set, the spending in each state or country is disproportionately on the poor who live there. Since Mississippi doesn’t have much wealth even after transfers, spending isn’t great, except in the population that is principally affected by the transfers (Medicare). In New York and Massachusetts, there are substantial resources, and the poor use them disproportionately.

Communal (state) wealth and individual (Zip code) income explain the vast bulk of geographic variation in health care.

Jay Brazier February 5, 2013 at 1:36 pm

“Interestingly, in an individual-level analysis of correlates of Medicare spending variation, Stephen Zuckerman and colleagues (PDF) found that some of the key factors that explained variation in Sheiner’s state-level analysis – income, most measures of race, and BMI – were not statistically significant.”

-It looks like Zuckerman et al were looking at the statistical attributes of the over-65 population and correlating them with practice/spending variations in a given geographic area (Please correct me if I’m mistaken).

While an analysis that excludes the working-age cohort from the analysis can still be quite informative, it seems plausible that variations in the working age cohort could influence spending, practice, and outcome variations in ways that aren’t fully captured by analyses that are confined to the Medicare population.

The primary limitation of the geographic variation literature as a whole is the limited coherence across data sets. Spending and outcomes variations in Medicaid patients don’t accurately predict variation amongst privately insured patients. Data at the HRR level don’t necessarily predict variation at the state or zip-code level.

The absence of coherence across statistical boundaries suggests that studies that arrive at different conclusions regarding the influence that a given demographic variable has on spending can be seen as informing one another rather than invalidating one another – at least until someone has developed a model that generates consistent predictions across all data sets based.

Zhongwenz February 22, 2013 at 10:43 am

On the contrary,

It would really simplify things if doctors prescribing too many tests, making too many referrals, etc, explained why U.S. health care is so expensive. Why? Because it implores Democrats to let go of political hot potatoes–Medicare-for-all, public option, etc and join Republicans on certain health policies. Of course, while regional variation in Medicare may be of interest, it is far from the reason why U.S. health care is so expensive, as one could observe from a cursory glance around the developed world (where one would see plenty of Medicare-for-all systems accounting for 9-12% of GDP). By the way, per capita growth in Medicare expenditures is markedly lower than in the private insurance market, regional variation or not.

Politicos have latched on to the Dartmouth group precisely because it makes their life easier. Remember the public option’s journey through the U.S. Congress? Something tells me Dartmouth inspired policies would have an easier time getting through.

Comments on this entry are closed.

{ 3 trackbacks }