Whenever international comparisons are made on quality metrics, inevitably someone complains that the results are invalid because it’s impossible to standardize quality across different systems. Those people are not entirely wrong. It is hard to make sure that you are measuring the exact same thing in different countries. That’s why it’s important to use a variety of metrics, and to acknowledge the limitations in any study and any comparison.

That said, it’s sometimes easier to standardize metrics of access. Asking patients the same questions in different countries can lead to useful and valid information about how easy it is to obtain care when you need it. This topic was the focus of a recent Perspectives piece in the New England Journal of Medicine, entitled “Equitable Access to Care — How the United States Ranks Internationally“.

It should come as no surprise to anyone who hears the rhetoric surrounding health care reform that more people in Canada wait a week or more to see a doctor than do so here in the United States. More people also wait that long in Norway than here. But more than one quarter of Americans need to wait that long to see a doctor, which is more than in Australia, France, Germany, the Netherlands, New Zealand, Sweden, and the United Kingdom.

Fewer Americans report that it is easy to obtain after-hours care than all of those countries except Canada, France, and Sweden. Physicians are less likely to report that they have an arrangement for after hours care in the United States than any of those countries, period.

But one of the most stunning metrics, and one that I use in almost all talks I give on the subject is this: It’s not just the poorer half of Americans who have problems obtaining care. Even the wealthier do.

In Table 1 of this perspectives piece, the authors note the percentage of people with below average income who experienced barriers to care. For instance, 39% of people with below average incomes report that they did not visit a doctor because of cost in the last year. The next two lowest countries were New Zealand, with 23%, and the Netherlands, with 16%. About 31% of below-average income Americans didn’t get a recommended test, treatment, or follow-up because of cost, and 30% didn’t fill a prescription or skipped doses because of cost. In all of these, the United States was clearly worse than all the other countries.

But in Table 2, the authors look at people with above average income. In the United States, 17% of people with above average incomes report that they did not visit a doctor because of cost in the last year, compared with an average 5% i the other ten countries. In the United States, 11% of above-average income Americans didn’t get a recommended test, treatment, or follow-up because of cost (average 3.4% for the other countries), and 12% didn’t fill a prescription or skipped doses because of cost (5.7% for the other countries).

In the following chart, I’ve used slightly older data to give you a better picture of this. I’ve graphed the percentage of people in each country who report having encountered at least one of these barriers in accessing care. The blue bars are the wealthier half of each country, and the green bars are the poorer half of each country.


What’s striking is that a larger percentage of the richer half of America is more likely to avoid care because of cost than the poorer half of almost every other country. Cost is an issue not just for people with below average incomes – it’s an issue in the United States for people with above average incomes.

It’s still too early to tell whether the ACA will change any of this. Surely insurance coverage will make it easier for some people to acces the health care system who could not before. But many barriers, including increased cost sharing and high prices, still exist. Only time will tell whether our efforts at health care reform will change some of these metrics.



{ 1 comment }

Are we part of a global community?

The United States policy community is well known for its almost allergic reaction to considering any lessons from other countries to improve health policy or care delivery. American exceptionalism is alive and well: just ask any policy advisor or Washington insider and they will roll their eyes and caution against any mention of “those” countries where they have “socialized” medicine. Clearly these well-tread tropes are present in our public discourse. Yet, the United States could stand a little humility and openness – after all, we live shorter lives in poorer health compared to other OECD countries. And, as Betsy Bradley has pointed out in her book, The American Health Care Paradox: Why Spending More Is Getting Us Less, when you add up what we and other countries spend on health and social services, it is fairly comparable—we just spend more of it on health care and they spend it on social services.

In my opinion, this insular approach has always been short sighted, but I think it is particularly wrong-headed now, as countries across the globe are grappling with similar issues: how to provide insurance coverage at a sustainable cost, address the burden of chronic disease, build systems of care that can not only deliver reliable, high quality care but also respond to crises such as the Ebola epidemic, and test new workforce models-such as community health workers and task shifting-to deliver more responsive, efficient care. These were all topics at the recent 3rd Global Symposium on Health Systems Research, held in Cape Town, South Africa and sponsored by the now two year old organization Health Systems Global, the first international membership organization fully dedicated to promoting health systems research and knowledge translation. The key themes of the conference are summarized here.

[click to continue…]



Before I knew too much for my own good, I used to think that there ought to be a very simple solution to understanding health spending growth. Surely, I thought, one could decompose it into its constituent parts, analyze those parts, and “figure it all out.”

Sounds naive, right? It is! But why?coordinates

There is no shortage of reasons. Here’s one: There isn’t a unique basis of decomposition. Rather, there are lots of equally valid ways to express health spending growth as a combination of other factors. If you’ll excuse the math analogy, it’s a lot like locating a point in space with respect to a coordinate system. You could do so in Cartesian coordinates (good old, familiar x, y in two dimensions) or polar coordinates (r, theta). They’re both useful, providing a different way of characterizing the same thing and lending themselves to addressing different problems. Neither is “correct.”

Similarly, one can decompose health spending growth into different sets of factors. One set, studied by many researchers, is changes in demographics, insurance, income, prices, administrative costs, and technology (the residual). I provided a summary of work that decomposes health spending growth in this way in a post in 2012. The chart below provides a some of the results at a glance.

The consensus finding is that technology growth is a (if not the) leading driver of health spending growth, though other factors play large rolls too.

Another way to decompose health spending growth is explored in a recent study in Health Affairs by Martha Starr, Laura Dominiak, and Ana Aizcorbe. The authors examine the effects on health spending growth of disease prevalence, cost of care per case, and other factors that play smaller roles. They expand the findings of earlier work by Charles Roehrig and David Rousseau who showed that health spending growth between 1996 and 2006 was driven predominantly by the cost of care, not increasing disease prevalence, as shown in the following chart.

Martha Starr, Laura Dominiak, and Ana Aizcorbe extend the analysis back to 1980, with similar findings, as shown below. (In the chart, “other changes” include small ones due to changes in population composition and changes due to interactions between the various factors in the decomposition.) In this decomposition, cost per case is the leading and consistent driver of health spending growth.

growth decomp

Conceptually, we may be able to reconcile the result that much of health spending growth is due in large part to technology, in one decomposition, and spending (or “cost”) per case, in another. Though other factors no doubt drive spending per case (including income growth, third-party payment, and so forth), it’s a fair bet that technology plays a large roll, with the typical case for many diseases treated more intensively and with more use of ever-newer technology that costs more.

Indeed, Starr and colleagues suggest that medical innovation, coupled with insurance coverage, is a potential driver, and that more analysis is needed to more fully explain the linkages.

[T]he vast majority of growth in annual costs of treatment per disease during the period studied can be attributed to people under age sixty-five who were covered by private insurance and people ages sixty-five and older (who are almost all covered by Medicare). This is generally consistent with the idea that insurance coverage has been centrally involved in increases in costs per case, perhaps by raising returns to medical innovation, lowering its risks, or both. [...]

Careful analyses of what explains variations in the pace of cost growth across more narrowly defined disease categories, including information about the timing and spread of innovations in diagnostic and therapeutic technologies, could be valuable for improving our understanding of what accounts for the longer-term pattern of robust growth in average cost per case.

Coordinate system image credit: Aerodynamics for Students.

Austin B. Frakt, PhD, is a health economist with the Department of Veterans Affairs and an associate professor at Boston University’s School of Medicine and School of Public Health. He blogs about health economics and policy at The Incidental Economist and tweets at @afrakt. The views expressed in this post are that of the author and do not necessarily reflect the position of the Department of Veterans Affairs or Boston University.



Largely lost in the recent coverage of the general moderation of growth of health care spending is the fact that health care spending growth continues to strain state and local budgets. That’s the focus of a recent paper presented at Brookings by Donald Boyd report by the Pew Charitable Trusts.

The Pew report notes that states and local governments were partly spared the increasing costs of health care between 2008 and 2010, due to a temporary increase in federal spending on Medicaid. That increase was included in the American Recovery and Reinvestment Act of 2009 and extended by subsequent legislation. Despite an increase in Medicaid spending over that period, states’ share of it actually declined.

Going forward, however, Medicaid and other state and local health care spending are projected to impose significant budgetary burdens.

In the years ahead, the Centers for Medicare and Medicaid Services projects that state and local government spending will rise by 49 percent in inflation-adjusted dollars from 2012 to 2022.

Looking further ahead, the Government Accountability Office—the nonpartisan investigative arm of Congress —warns that health care spending is the primary driver of the long-term fiscal challenges that it expects state and local governments to face. According to the GAO’s simulation, state and local health-related expenditures will nearly double as a percentage of gross domestic product from 2014 to 2060.

Importantly, Donald Boyd’s analysis reveals that the majority of projected state health care spending will not be driven by Medicaid or its expansion.

[State and local health care] cost increases are driven more by non-Medicaid costs than by Medicaid costs. The non-Medicaid increases are driven first by retiree health care costs [] and second by costs of health care for the existing workforce. The Medicaid cost increases are driven primarily by costs for the elderly and disabled. Children, adults, and Medicaid expansion enrollment play a much smaller role in the increases.

All told, future health care spending will take a bigger bite out of state and local government budgets. Consider, for example, the following chart included in Pew’s report. It provides historic and projected health and non-health state and local government spending as a percentage of GDP.

local budgets

Now, it’s a bit of hubris to predict trends out 45 years or so. Nevertheless, even from the historic and near-term predictions, the above chart suggests that health care spending is crowding out other state and local priorities. (See also Reid Wilson’s summary and comments on the Pew report.)

A similar note was struck in the Massachusetts Health Policy Commission’s 2013 Cost Trends Report, in which you’ll find the following chart. It shows growth between 2001 and 2014 in state spending on health care coverage on the left and, on the right, spending growth (reduction, really) in other areas: mental health, public health, education, human services, infrastructure, housing and economic development, law and public safety, and local aid. (The health care category includes spending for the Group Insurance Commission (GIC), which provides coverage to state and municipality employees, MassHealth, the state’s Medicaid program, and other coverage, like subsidies for private insurance under the state’s health reform.)

mass budget

In Massachusetts, as state spending on health care coverage grew 37% (after adjusting for overall economic growth) over the 14 year period as spending for other areas declined 17% in aggregate. More troubling still, some of those other areas of spending also contribute to health and wellbeing, and perhaps more so than the marginal dollar spent on health care coverage.

Both the Pew report and the evidence from Massachusetts echo the work of Elizabeth Bradley and Lauren Taylor, which suggests U.S. health outcomes lag those of peer nations because we spend too little on services that relate to social determinants of health, in favor of far too much spending on health care directly. (Allan Joseph summarized and commented on this work for The Incidental Economist.) As the health sector gobbles up more of state, local, and national, budgets, this imbalance only grows.

Additional spending on health care to expand coverage may be well worth the state and local spending it will require (see my paper with Aaron Carroll relevant to this), but we would be wise to also consider the merits of that which additional health care spending crowds out.

Austin B. Frakt, PhD, is a health economist with the Department of Veterans Affairs and an associate professor at Boston University’s School of Medicine and School of Public Health. He blogs about health economics and policy at The Incidental Economist and tweets at @afrakt. The views expressed in this post are that of the author and do not necessarily reflect the position of the Department of Veterans Affairs or Boston University.


{ 1 comment }

Surveys of the health services research (HSR) and health policy fields have shown that the racial and ethnic makeup of the field of HSR does not reflect the populations for whom we hope to improve health and health care.

As the professional society for the field, AcademyHealth believes that the inclusion of diverse individuals, with equally diverse perspectives and training, leads to a richer and more nuanced understanding of health and health care issues. Diversity among our members and the field at large, in terms of race, ethnicity, disability, sexual orientation, gender identity, and other historically underrepresented backgrounds, is therefore a priority for AcademyHealth.

It is because of this commitment that AcademyHealth, in partnership with the Aetna Foundation, has established the AcademyHealth/Aetna Scholars in Residence Fellowship Program to promote diversity and inclusion in science. Housed within the AcademyHealth Center for Diversity, Inclusion and Minority Engagement in HSR Research, this fellowship is designed to retain underrepresented racial/ethnic minorities* in health services research by providing professional, training, and networking opportunities for junior to mid-level researchers and clinical practitioners, who are conducting disparities research with a focus on population health, to present their research.

Read more about this fellowship and the application process on the AcademyHealth website.

*AcademyHealth defines underrepresented racial/ethnic minorities in the field of HSR as the following groups: Black/African American, Hispanic/Latino, American Indian/Alaska Native, Native Hawaiians and Other Pacific Islanders. This definition is supported by findings of a recent study by Johnson and Holve



I have long argued against the argument that tort reform is a viable means to cost control. One of my main reasons for doing so is that there is very little evidence that passing tort reform results in significant changes in physician practice.

One of the reasons for this is that there are many reasons for ordering tests, procedures, and hospitalizations outside of a fear of litigation. An obvious one, sometimes overlooked, is that there is a financial benefit to many physicians, offices, and hospitals for ordering that stuff as well. It’s sometimes easier to say that you’re doing so for “defensive” reasons because it gets you off the hook for overusing services and care.

It’s for this reason that many blunt assessments of the cost of defensive medicine may be overblown. Lots of docs may say that they’re doing something defensively when they are actually doing it for many reasons, even if malpractice litigation fears are legitimately part of the cause.

That’s why I was so pleased to see a new study published in the journal JAMA Internal Medicine, “The Cost of Defensive Medicine on 3 Hospital Medicine Services“. The authors rounded up 36 physicians who rated more than 4200 orders for 769 patients. Orders included tests, procedures, and hospitalizations. They were all ranked on a 5 point scale from 0 (not at all defensive) to 4 (completely defensive). Costs were analyzed using both cutoffs of 1 (defensive at all) and 4 (completely and only defensive).

The number of orders places was similar between physicians with fewer and more defensive order. The costs per patient were also similar between these two types of doctors ($1679 versus $1700). The mean cost per patient overall was $1695.

Of this, 13%, or $226, was found to be at least partially defensive in nature. But completely defensive orders, or those that had no justification outside of a fear of litigation, comprised only 2.9% of costs. Most of this was due to additional hospital days.

This is an important point. As the researchers note, past studies have found that “27% of computed tomographic scans, 16% of laboratory tests, and 14% of hospital admissions were ordered owing to concerns about liability.” But such studies would include any level of defensiveness in the orders at all. We can realistically expect, however, that only completely defensive orders would be eliminated by tort reform. After all, if there are other reasons to order tests above our fears of being sued if we don’t, those reasons will still exist even after comprehensive malpractice reform became law.

If we assume that overall health care spending is about $2.7 trillion, then 2.9% of that would be about $78 billion. That’s not chump change, mind you, but it’s still a very small component of overall health care spending. Given that there’s little evidence that tort reform would lead to a significant reduction in this already small percentage of spending, there seems little reason to pursue it as a means to dramatically reduce health care spending in the United States.

None of this should be taken as a message that the malpractice system is not dysfunctional in this country, and could not benefit from significant reform. But evidence seems to indicate that in doing so, a reduction in the numbers of nuisance cases and an increase in the number of legitimate claims might be a more achievable outcome than a significant reduction in spending.




On 11 September, 2014 Louise Sheiner presented a paper at Brookings that challenges some of the the interpretations of Dartmouth research on geographic variation in health care. Her work suggests that patient, not provider, factors explain most of geographic (in her case, state) variation in spending. Coincidentally, Austin Frakt had already prepared a post reviewing work that comes to the opposite conclusion. That post appears below, and is not intended as a rebuttal to Sheiner’s work.  

One of the most important and vexing strands of research on the American health care system began with a 1973 study of tonsillectomy in Vermont. That year, John Wennberg and Alan Gittlesohn published “Small Area Variations in Health Care Delivery,” in which they showed that in one Vermont town, 66% of children had their tonsils removed, but in neighboring towns 16-22% did.

As Aaron wrote at The New York Times,

There were no differences in the children [...]. Tonsillitis was not more common in one town, and the children were no more or less healthy before or after the procedure. The doctors just seemed to have very different attitudes toward how to take care of tonsils.

Since the Wennberg/Gittlesohn study, many investigators have examined geographic variation in health care utilization and spending and their relationships to health outcomes. A great deal of this literature is summarized in a Handbook of Health Economics chapter by Amitabh Chandra, David Cutler, and Zirui Song, addressing they key question, “What drives who gets what care?” These scholars categorize the possible factors as either demand side (e.g., income and preferences for care), supply side (e.g., quality), or situational (e.g., idiosyncratic contextual or behavioral influences).

There is no debate that large regional differences in health care provision, costs, and outcomes exist. For example, as Atul Gawande famously documented, in 2006 Medicare spent $15,000 per beneficiary in McAllen County, Texas and half that in the very similar county of El Paso, Texas. But there is considerable debate as to what drives such differences, in general, as Chandra et al. explain.

Some researchers argue that variation is accounted for by population disease burden (Zuckerman et al., 2010), but other authors argue that prevalence of diagnoses itself is endogenous across areas (Song et al., 2010Welch et al., 2011). Most of the literature agrees that patient characteristics and preferences do not explain much of the differences across areas, and that substantial variations in treatment practices remain after controlling for patient characteristics (Anthony et al., 2009O’Hare et al., 2010Baicker et al., 2004).

The authors are also brutally honest, writing, “None of the theories for which there is a lot of evidence can be shown to explain a major part of cross-individual or cross-area variation in treatments.” However, their tentative conclusion is that supply side factors are more important than demand side factors in driving treatment choice.

An August 2013 paper by David Cutler, Jonathan Skinner, Ariel Dora Stern, David Wennberg is consistent with this conclusion. The authors examined the extent to which patient and physician preferences (collected by surveys) explained regional variation in health care spending.

Ultimately, the largest degree of regional variation appears to be due to differences in physician beliefs about the efficacy of particular therapies. Physicians in our data have starkly different views about how to treat the same patients, and these views are not highly correlated with demographics, background, and practice characteristics, and are often not consistent with professional guidelines for appropriate care. As much as 36 percent of end-of-life Medicare expenditures, and 17 percent of overall Medicare expenditures, are explained by physician beliefs that cannot be justified either by patient preferences or by clinical effectiveness.

In other words, many physicians believe in the value of certain types of care even when the evidence does not support that belief, and this explains a substantial fraction of spending.

Theirs is not the first work to highlight the importance of supply side (physician) factors. In Gawande’s examination of  McAllen and El Paso, he found that McAllen physicians seemed to be more entrepreneurial than their counterparts in El Paso, for instance owning “imaging centers, surgery centers, or another part of the hospital they directed patients to.” Francis Lucas and colleagues found significant peer effects: “27% of respondents reported ordering a cardiac catheterization if a colleague would in the same situation frequently or sometimes.”

Clinicians’ beliefs may play an important role in geographic variation in health care and the inefficiencies such variation suggests. (This is not to the exclusion of patient factors, of course.) As such, an interesting line of research would focus on the extent to which those beliefs are malleable. Overpowering them with payment incentives could work, if those incentives are large enough, but I wonder if beliefs can be addressed head on.

Austin B. Frakt, PhD, is a health economist with the Department of Veterans Affairs and an associate professor at Boston University’s School of Medicine and School of Public Health. He blogs about health economics and policy at The Incidental Economist and tweets at @afrakt. The views expressed in this post are that of the author and do not necessarily reflect the position of the Department of Veterans Affairs or Boston University.



One of the most popular provisions of the Affordable Care Act is the one that allows young adults up to age 26 to stay on their families’ plans. Since this age group is pretty healthy in general, the cost of allowing them to remain on others’ plans is relatively low. It’s also resulted in what appear to be significant drops in the uninsurance rate for those in this age group.

A recent study in JAMA Pediatrics explores this phenomenon further. “Limited Impact on Health and Access to Care for 19- to 25-Year-Olds Following the Patient Protection and Affordable Care Act“:

IMPORTANCE The Patient Protection and Affordable Care Act (PPACA) allowed young adults to remain on their parents’ insurance until 26 years of age. Reports indicate that this has expanded health coverage.

OBJECTIVE To evaluate coverage, access to care, and health care use among 19- to 25-year-olds compared with 26- to 34-year-olds following PPACA implementation.

DESIGN, SETTING, AND PARTICIPANTS Data from the Behavior Risk Factor Surveillance System and the National Health Interview Survey, which provide nationally representative measures of coverage, access to care, and health care use, were used to conduct the study among participants aged 19 to 25 years (young adults) and 26 to 34 years (adults) in 2009 and 2012.

EXPOSURE Self-reported health insurance coverage.

MAIN OUTCOMES AND MEASURES Health status, presence of a usual source of care, and ability to afford medications, dental care, or physician visits.

Researchers in this study used data from the BRFSS and the NHIS to look at both access and utilization of health care for young adults age 19 to 25 years. They also looked at adults 26 to 34 years as a comparator. They did this in both 2009 (before the new provision) and in 2012 (after it). The main exposure of interest was, of course, whether they were insured. The main outcome measures were their health, whether they had a usual source of care, and how they rated their ability to afford care.

Confirming previous reports, the percentage of young adults who were insured increased from 2009 to 2012, from 68% to 72%. The percentage of adults who were insured, however, dropped dramatically over this time, from 78% to 70%. In this respect, it appears that the new provision was associated with significant improvements in insurance compared to adults 26+. Both groups reported a decrease in having a usual source of care, but the decrease was less for young adults, likely because adults saw a much higher drop in their chance of being insured.

However, there was no improvement in the chance that young adults went for a routine checkup. Young adults saw no greater improvement in their health status compared to adults over this time. They were not more likely to be able to afford prescription drugs, dental care, or visits to the doctor. They weren’t even more likely to get a flu shot.

As with any study, this one has limitations. All of these data are self-reported, so it’s possible that improvements occurred but aren’t being seen. It may take a longer time for the effects of coverage to have an impact on utilization or health. It may be that the accompanying downturn in the economy had a large effect that washed out that of insurance.

But the bottom line is that this confirms something we need to acknowledge. Access is necessary, but not sufficient, for good health. Many other things matter, too. In terms of improving the health care system, improving access and reducing uninsurance is almost the easiest step. Improving quality is much harder. But if we want better outcomes, we need to do it.



{ 1 comment }

In recent years, there has been a steady movement away from small, private practices of physicians into larger group practices. The Affordable Care Act likely is hastening this transition with its encouragement of Accountable Care Organizations. It’s impossible for small or individual practices to meet the requirements of being an ACO, and as more and more reimbursement is tied to metrics associated with them, it’s becoming harder to avoid joining them.

Many are comfortable with this development because they believe in the idea of a “medical home”. Some research has shown that when a personal physician who knows a patient well and can coordinate all aspects of their care, both primary and specialty, then patients do better. It’s much easier to run medical homes out of larger practices that have more resources and practitioners of all types.

A recent study in HSR showed that we might save money through medical homes as well. It examined Medicare Part A and B claims over three years, and looked at how practices designated as NCQA-recognized patient centered medical homes compared to practices which weren’t. They found:

Relative to the comparison group, total Medicare payments, acute care payments, and the number of emergency room visits declined after practices received NCQA PCMH recognition. The decline was larger for practices with sicker than average patients, primary care practices, and solo practices.

According to this study, medical home patients were seeing improved quality. Medical home practices had reduced numbers of emergency department visits, with the largest improvements seen where patients were sicker. Not only that, but they saw reductions in total Medicare payments. Medical homes were associated with improved quality and reduced spending.

A more recent study in Health Affairs, however, countered this claim:

Nearly two-thirds of US office-based physicians work in practices of fewer than seven physicians. It is often assumed that larger practices provide better care, although there is little evidence for or against this assumption. What is the relationship between practice size—and other practice characteristics, such as ownership or use of medical home processes—and the quality of care? We conducted a national survey of 1,045 primary care–based practices with nineteen or fewer physicians to determine practice characteristics. We used Medicare data to calculate practices’ rate of potentially preventable hospital admissions (ambulatory care–sensitive admissions).

This study actively surveyed practices to determine their practice characteristics. Then they used Medicare data to see what their preventable hospital admission rates were. They found that compared to practices with 10-19 physicians, practices with 3-9 physicians had 27% fewer preventable admissions. But practices with only 1-2 physicians had ever fewer, one third fewer preventable admissions. Practices owned by doctors had fewer preventable admissions than those owned by hospitals.

This, of course, contradicts the other study. It would seem to show that moving into larger group practices, and away from single or small practices, might reduce patient quality.

But there are some nuances to these studies worth considering. The first study did not look at practice size, per se. It looked at medical home designation. It’s totally possible for solo practices to become medical homes, and those that were performed better. In fact, if you delve into the tables, it seems that the largest improvements for hospitalizations were seen in solo practices with medical home designation. It also seems that the largest improvements in medical specialist visits were in 2-person practices.

In fact, there seems to be some general agreement that there’s a lot of good outcomes associated with small and solo-practices. Medical homes may lead to better outcomes, but that doesn’t necessarily mean that large ACOs will do just as well.




The White House Blog has a post up on “How Low-cost Randomized Controlled Trials Can Drive Effective Social Spending“:

The Office of Science and Technology Policy and the Coalition for Evidence-Based Policy convened leaders from the White House, Federal agencies, Congress, philanthropic foundations, and academia this week to explore an important development in the effort to build credible evidence about “what works” in social spending: low-cost randomized controlled trials (RCTs). The goal of the conference was to help advance a broader Administration effort to promote evidence-based policy, described in the evaluation chapter of the 2014 Economic Report of the President, and the Performance and Management section of the President’s budget.

Large and rigorous RCTs are widely regarded as the most valid method of evaluating program effectiveness, but they are often perceived as too costly and burdensome for practical use in most contexts. The conference showcased a new paradigm: by measuring key outcomes using large administrative data sets already collected for other purposes – whether it be student test scores, hospitalization records, or employment and earnings data – sizeable RCTs can be conducted at low cost and low burden.

The conference showcased a number of RCTs that were conducted for between $50,000 and $350,000 (a fraction of the usual multimillion dollar cost of such studies), yet produced valid evidence that informed important policy decisions.

First of all, I’m thrilled at the idea, in general, of using data, especially those derived from randomized controlled trials, to make policy decisions. Using evidence to drive policy has always been one of the goals of this blog, as well as others I write for.

But this post focused on how it’s possible to do randomized controlled trials at an amazingly low cost. It describes, for instance, a study of Recovery Coach services to substance-abusing parents who had temporarily lost custody of their children. The cost of the nine-year trial was about $100,000, which is stunning.

But these types of opportunities are the exception, not the rule. I think it highlights a misunderstanding of where grant money goes much of the time.

If you looked at the budgets of any R01 funded randomized controlled trial I’ve had funded, the number one cost, by far, is the salaries of those who are conducting the study. Even if it’s just 10-20% of someone’s time, and it’s just a few people, the salaries add up. When you add in benefits and overhead, you can get into the hundreds of thousands of dollars pretty quickly. Also, research comes with indirect costs. The rent for people’s offices, the electricity, their phone lines, support staff, taxes, etc. There are a lot of costs.

You often need to pay for a number of people because you’re setting up some new intervention! That takes time, it takes effort, and it takes money to pay for it.

The opportunities like those described in the White House blog are relatively rare. All of the interventions discussed were already being run by personnel who were likely employed through other public lines of support. That’s great, but that doesn’t happen often. If the intervention is already paid for, if those running it are already salaried, then most of the costs of a grant are wiped clean. If the data are already collected as part of administrative data, that’s awesome, but that wasn’t free. It was paid for by someone else. Yes, that makes the “grant” cost less, but not the research itself.

It appears that all that had to be paid for in the studies presented was the cost of the analysis. That may easily be $100,000. But that’s only the last part of an RCT, usually.

It’s totally worth it to try and do research on the cheap when you can. It’s incredibly efficient. But we shouldn’t be under the illusion that research is cheap. Lots of great things can’t be done this way, and it’s important to do those things, too.