In four separate posts, Austin Frakt and Aaron Carroll from The Incidental Economist will describe their translation and dissemination process, specifically how they turn academic papers into interesting blog posts. Each of their posts will cover one of these questions:
- How do we read research papers?
- How do we decide what to write about?
- How do we make our posts interesting?
- How do we decide where to publish them?
This is the first post:
How do we read research papers?
Austin: My first point of (potential) entry is a paper’s title and authors. By email, I receive tables of contents of every journal I’m aware of that focuses on U.S. health care policy, health services research, or health economics, as well as those from some of the top medical journals. I lack the time to read even just the abstract of every paper in those dozens of tables of contents. But, I can, and do, look at every title and list of authors.
If a title looks interesting to me or the author list includes one of my favorite authors, I’ll look at the abstract online. My rough guess is that I look at the abstract of 10% of the papers that cross my inbox. Of those, perhaps one-third or so seem interesting or sound enough for me to want to read more. So, perhaps I read some of the text of about 3% of papers in the tables of contents I receive. That’s a rough estimate, but it’s probably not far off.
When I read a paper, it’s always in PDF Expert on my iPad. That way I can read it anywhere and mark it up.
If I’m considering showcasing a paper in a post, I’ll read the PDF of whole thing. I do this in the most boring way possible — top to bottom/front to back, linearly, annotating important passages, questions, and references to other papers I want to read. (If I’m not showcasing paper, but need to find just some bit of info, I read more selectively, just to find what I need.)
If the methods of a paper are very complex (e.g., some health economics papers), often I only get the gist of them. When I do that, I find a colleague with greater methodological expertise to vet the methods. I do not rely on the journal review process to have served this role. Nor do I trust that a paper’s discussion of limitations to reveal all the important ones. I’m a very skeptical guy.
With high frequency, but not always, I send authors questions. Sometimes I wonder if they’ve done the analysis another way. Often they have, and they share the results. Sometimes I just don’t understand something, and they clear it up. Sometimes I find errors, which they confirm.
I guess, to me, a paper is often just the starting point of a dialog with authors and colleagues.
Aaron: I’m so glad we’re doing this. I’m fascinated by your response. I never, ever look at the authors. I look at tons of tables of contents as well, but only for the titles. If something piques my interest, I’ll go further.
My next step is to look at the abstract, trying to answer some specific questions at each point. I hit the objective — is this a hypothesis that interests me? I look at the methods — are they appropriate to test the hypothesis? I look at the results — what did the methods reveal? I tend not to spend too much time on the background or conclusions.
If I’m still interested, I will now pull the paper and read the methods critically. I’m already trying to poke holes in the work. Where are its flaws? What are the limitations?
Next, I jump to tables and figures. Table 1 usually tells me if this was an appropriate population for study. It also tells me how generalizable the findings may be. Then, I look at the other tables and figures, which should tell me about the key findings of the paper. Finally, I will read the results to make sure I didn’t miss anything.
I do read conclusions, but less to see what the authors felt about the results than to see what they thought the limitations were. Do they agree with me? Did they see things I didn’t? Did they miss something I picked up?
I rarely contact the authors of a paper, unless I have a specific question. I like to think that studies should stand on their own. If I can’t figure something out, though, I do reach out. I set a deadline on that, though, and will update after posting if I hear back.
One thing I never do, though, is read press releases. I make it a point not to discuss papers I haven’t read myself.
But we’ve skipped a step: There are more good papers out there than we have time to write about. How do we decide which to give attention? We’ll address that in our next post in this series.
Austin B. Frakt, PhD (@afrakt), is a health economist with the Department of Veterans Affairs, an Associate Professor at Boston University’s School of Medicine and School of Public Health, and a Visiting Associate Professor with the Department of Health Policy and Management at the Harvard T.H. Chan School of Public Health. Aaron E. Carroll, MD (@aaronecarroll), is a professor of pediatrics at Indiana University School of Medicine. Both blog about health economics and policy at The Incidental Economist. The views expressed in this post are those of the authors and do not necessarily reflect the position of the Department of Veterans Affairs, Boston University, Harvard University, or Indiana University.