October 22, 2024

Podcast

Uncovering Hidden Bias in Clinical Research

By Lisa Bero, Monya Baker

Check the end of any recent study, and there will be a list of study funders and disclosures about competing interests. It’s important to know about potential biases in research, but this kind of transparency was not always the norm. Understanding bias in research and helping policymakers use the most reliable evidence to guide their decisions is a science in itself.

Lisa Bero, a professor at the University of Colorado Anschutz Medical Campus, has been at the forefront of understanding how corporate funding biases research and how to assess what scientific evidence is reliable. She talks to host Monya Baker about her investigations into the tobacco and pharmaceutical industries, techniques industries use to shape evidence to favor their products, and the importance of independent research to inform policy.

SpotifyApple PodcastsStitcherGoogle PodcastsOvercast

Resources

Transcript

Monya Baker: Welcome to The Ongoing Transformation, a podcast from Issues in Science and Technology. Issues is a quarterly journal published by the National Academy of Sciences and Arizona State University.

Check the end of any recent study and you’ll see a list of funders and disclosures about competing interests. It’s important to know about potential biases in research but this kind of transparency was not always the norm. In fact, helping policymakers use the most reliable evidence to guide their decisions is a science in itself.

My name is Monya Baker. My guest today, Lisa Bero, has been at the forefront of understanding how corporate funds bias research and how to assess what scientific evidence is reliable. Her investigations of the tobacco industry reveal techniques that industries use to shape evidence that favors their products and her analyses of statistical indicators of bias and drug-sponsored research has caused journals to change publication practices. She’s a professor at the University of Colorado Anschutz Medical Campus.

Lisa, welcome!

Lisa Bero: Happy to be here.

Baker: Tell me about how you started studying bias in clinical research.

I realized that a lot of policy solutions didn’t seem to align with what we were studying in the lab and we had the potential for all these great breakthroughs and they weren’t getting translated into a policy.

Bero: So, my PhD is in pharmacology and I was working in a lab doing studies of drugs, trying to figure out the mechanism of action for addiction. And I was also, at the same time, volunteering at places that serve people suffering from addiction. And I really got a bit disillusioned because I realized that a lot of policy solutions didn’t seem to align with what we were studying in the lab and we had the potential for all these great breakthroughs and they weren’t getting translated into a policy. So, I did start a postdoc in my basic science area but then I switched and did a postdoc in health policy and epidemiology. And frankly, that really changed my life because then I really wanted to shift to how we use evidence better to make informed health policy decisions.

But unfortunately, I got even a little bit more disillusioned because I realized that the evidence itself could be manipulated and that, if we wanted to have good evidence-based decisions, the first thing we need to do was have good evidence and we needed to make it understandable to policymakers. So, that set me off on my entire career looking at bias in evidence and how we better package that so people can understand it and use it.

Baker: Some of your early work was in tobacco and drug sponsored symposia. Tell me a little bit about what those are.

Bero: My first paper really looking at bias in a body of evidence was looking at drug studies. There were all of these sort of quasi-clinical trials being published in what were called symposia. So, they were usually attached to peer-reviewed journals; they often said they were peer-reviewed articles; they looked like peer-reviewed journal articles. But when we started to investigate them, we realized that often they weren’t peer-reviewed, at least to the same standard, and there was very much a lack of balance in these articles. So, they might say something like new breakthroughs in anxiety but only one drug for anxiety was ever examined and drugs weren’t compared to each other. There were all sorts of problems with how the studies were done that made them look very much in favor of whatever drug the symposium was about.

Drug companies would pay to get the symposium published and then use it for marketing.

So, we published that saying that these symposium were basically biased and, a few years later, actually, many journals stopped publishing these symposia because they realized that they were actually being used for marketing purposes and they were just… Drug companies would pay to get the symposium published and then use it for marketing. And then I was literally at some reception and somebody came up to me and said, “Oh, you think drug studies can be biased, you really ought to look at tobacco.” So, I did. I started looking into particularly studies on the harms of tobacco and second-hand smoke.

Baker: Tell me a little bit about the tobacco papers.

Bero: So, when I was researching tobacco, I was doing what we call these meta-research studies. So, we look across a body of evidence on tobacco and then we can detect these biases. What that tells you is that there is a bias towards a finding in favor of the industry but it doesn’t actually tell you how that bias comes about. And it was in the early 1990s that a whistleblower, actually, at tobacco companies started to release some documents that actually revealed how the companies were manipulating research. And those were such a breakthrough because, like I said, we could detect the bias but we couldn’t actually see how it was done.

And then the master settlement agreement came along among the attorneys generals in the US with the tobacco companies in 1998 I think it was. And basically, part of that settlement agreement said that all of the documents that were reviewed in lawsuits against the tobacco companies had to be made public. And that was an avalanche of information and it was amazing. They would say things like, “Oh, if such and such a study that we’re sponsoring does not find a statistically significant result, we’re not going to publish it.” So, basically the decision to publish a study was predetermined by the finding. There are also lots of documents on how they courted researchers by paying them lots and lots of money but they would cut them off and not pay them anymore if they started publishing findings that were not favorable to the industry.

The tobacco industry documents also showed how the companies worked together to do this. So they weren’t just working alone, they formed whole research organizations.

And the tobacco industry documents also showed how the companies worked together to do this. So they weren’t just working alone, they formed whole research organizations like the Center for Indoor Air Research to drive research that found results favorable to tobacco, in this case, second-hand smoke. So the Center for Indoor Air Research, they published a lot of research on basically whether green plants in your office were good for you or carpet off-gassing was bad for you. But interestingly, they didn’t publish research on the harms of tobacco but we found out they had a separate program that was an internal program that was focused on studying the harms of tobacco but those weren’t published in the peer-reviewed literature.

Baker: It just sounds antithetical to your goal of making sure that there was good evidence to be used to guide policy. Were you surprised?

Bero: Well, we were surprised and I did a lot of studies looking at, basically, public commentary that companies would submit on proposed regulation. And in this commentary, you would find all this science, they were quoting tons and tons of science and scientific articles, refuting independent studies, refuting Surgeon General’s reports and this and that and you’re like, “Where do these copious numbers of studies come from?” And then you dig around and you find out, “Oh, well, they come, actually, from the industry themselves.” And they may even be in the peer-reviewed literature or they may be a study that wasn’t published in the peer-reviewed literature and they say is a consensus of scientists.

And so that’s why, actually, all of this evidence was being produced: it was so industry could use it to protect their product from regulation. And, really, the public health community, they weren’t submitting a lot of counter evidence to the companies. The companies had a lot more money to produce and submit that evidence but, still, the public health community needs to respond and I think we’ve learned that. We’ve learned that we need to counter all these industry-sponsored studies.

Baker: So, speaking of just copious studies, what is the work to look at all of these together? Tell us about meta research, systematic reviews, how that plays into bias.

Bero: So, a systematic review is when you try to identify all of the studies that have been done on a particular topic. And then once those are identified, you extract data from those studies and you do what’s called a risk of bias assessment, which is basically to assess each individual study for methodological biases, and then these studies are summarized some way. And if it’s quantitative, it’s called a meta-analysis but they can also be qualitatively summarized. So, the key to systematic review is it doesn’t just look at one study or two in isolation, it looks at all the relevant studies on a topic and it includes this risk of bias assessment. So, that’s why that method can not only give you an answer to whatever the question is, but it can also identify bias across a body of evidence.

The outcomes of these studies only differ by who the sponsor is; they’re not different in the way they were conducted or in their methods. And so, there’s some other bias going on.

So, we’ve done a lot of studies in nutrition, for example, looking at studies that have been sponsored by sugar companies or cereal companies or dairy companies, comparing them to other studies and we see that the industry studies have outcomes that favor the sponsor’s product even when you control for other risk of bias in the study. The outcomes of these studies only differ by who the sponsor is; they’re not different in the way they were conducted or in their methods. And so, there’s some other bias going on.

Baker: I’m going to go a little bit on a tangent here because it’s not just corporate interests that can bias work. There’s also some academic research that is problematic that’s been published and that’s also something that you have done a lot of work to uncover.

Bero: Yes. And if you look at any individual study, you can usually detect biases. And so, that’s why this meta research is very important because you look across a body of evidence. That’s when you can detect that there’s an industry bias because, as I mentioned, the only thing that seems related to the outcome is who is sponsoring it. And so, if you look at other biases that academics might have, for example, a strong intellectual view on something or a track record or personal experience in a particular area, those aren’t in a single direction like the industry bias is which is in favor of their product. And so, you may have researchers who have different views on weight loss, for example, but they’ll all be different and so it doesn’t have this driving force like the industry research.

The other thing—and we’ve studied this—is academic researchers, if they’re good academic researchers and a lot of them are, they may change their mind over the course of their career. They may find that their hypothesis isn’t panning out and so they shift and do something else and that’s quite common as well. And you don’t see industry sponsors doing that, they stick with their hypothesis.

Baker: Tell me a little bit about your work at Cochrane and maybe start by explaining what that is.

Bero: Yeah. So, Cochrane is an international organization that publishes systematic reviews and other types of evidence synthesis and has really set the standards, over the last 20 years, for how you do systematic reviews. One of the things that Cochrane has been very concerned about is we apply these tools to assess risk of bias to the studies that we include in the review but, if the study isn’t real or if there are problems with the data that can’t be assessed by reading the study or looking at the data, then we wind up synthesizing evidence that’s not true. And so, what’s happened over the last few years is, in certain fields, people have started to do studies to show that a certain proportion of clinical trials probably have faked data or faked analysis. It’s not always just the data, it can be part of the clinical trial.

And I can’t give you an exact number on what that is because it seems to really vary by field but the fact that any of these fake studies exist and could be included in a systematic review is really a problem because we’ve assessed them for bias, we’ll put them in the review. And so, some of my research with Cochrane now is to develop a tool to try to detect problematic studies. So, studies that, not only may be fake, but may have major data integrity problems. So, that’s what I’m doing. There are quite a few tools out there and we hope to have one that will be agreed upon and recommended by Cochrane through professional consensus. And I can imagine, in a few years, this will become part of a standard systematic review method to assess studies for their data integrity before including them in a systematic review.

Baker: I wonder if maybe you could talk a little bit about how the status quo around transparency has changed.

Bero: When I talk about this, sometimes I feel like, “Oh, I’m Miss Doom and Gloom and everything’s so terrible.” But actually, we have made huge advances that have improved the scientific literature and most of those have been around transparency. And so by that I mean we know a lot more about how studies are conducted, who’s funding them, the conflicts of interest of the authors. So, it’ll be hard for some people imagine but, many years ago, there was no disclosure of who was funding it or what the conflicts of the interest of the authors are and that is much, much more common now. The International Committee of Medical Journal editors took this up, many individual journals took it up, so it’s really common practice now to have these disclosures.

If there’s one message I want to get across here is that transparency is not the same as independence.

We also have transparency databases in certain areas. So, for example, that’ll tell you payments that drug companies make to doctors. It’s also become much more standard practice to publish protocols of studies before you start conducting the study and then to also make the data from the study openly available. And that’s really important because that means someone can go and look and say was this study conducted according to the protocol. And if you have access to the raw data, you can look and see if there’s any manipulation of that data during the analysis. So, this is great, I am all for transparency but, if there’s one message I want to get across here is that transparency is not the same as independence. And we know, for example, that all this disclosure does not prevent the industry bias that we observe in these meta-research studies because they’re based on detecting the industry sponsorship through disclosure.

And so, we’ve got the disclosures on the studies and that’s how we detect that there’s an industry bias because we can classify them as industry or not. And there’s also been some really elaborate experiments that have been done in psychology that show that, when people disclose they have a financial tie, they actually give more biased advice. So, disclosure is great and it’s absolutely necessary but it’s not sufficient to ensure independence from these industry sponsors.

Baker: How do you think independence can move forward?

Bero: So, I define independence as researchers being able to do their research free of any interference with how the research agenda is set, how the question is framed, how the study is conducted and whether it gets published in full or not because those are all the areas where we’ve seen that industry can introduce bias. But on the other hand, we know that corporate interests are big funders of research. And so, one way to make this funding independent of all these decisions about the research is really to change the funding model. So, one, we need more funding for research that’s independent of industry. And some people think, “Well, that’s really pie in the sky, it’s a high bar.” It is a high bar but, on the other hand, if you think about it, why should companies be evaluating their own products for harm? That’s like having an arsonist do fire inspections.

And so, some ways to separate that funding that will be separate from having more public funding would be to have companies put their funding into a pool and then no one company can have any say about the research. And in Italy, actually, there was a model for this for a while that was quite successful where drug companies in Italy were charged a fee for how much they spent on drug advertising in Italy. And then all those fees were collected and used to fund independent studies reviewed by the government and independent peer reviewers on harms of those drugs. So, it is possible to have these consortia funding.

So, I think changing the funding model will need more than just editors behind it, it’ll really need regulatory agencies. This has also been proposed for the FDA, for example, that there ought to be a separate government agency funded by pooled money from pharmaceutical companies that would focus on assessing harm of drugs, for example. Some journals have actually acted on this so there are bans on funding industry funded research in some journals, a lot of journals, British Medical Journal led on this and there’s some journals will not publish research funded by industry. But again, that’s after the fact, the research has already been done so much better if we can prevent it up front.

Baker: I think you were also telling me that there had been some student activism about making sure that, if the industry did give funds for research, there were no strings attached?

Bero: Yeah. So, this is great. The American Medical Student Association, years ago, I can’t remember exactly when it started, but it was when there was a lot of information coming about undisclosed financial ties. They had this idea—and I worked with some of these students—that they ought to know if their professors are getting paid by the pharmaceutical industry if they’re lecturing them on teaching them about drugs made by that company, for example. And so, they created something called the AMSA Scorecard and so they would score all the medical schools on how strong their conflict of interest policies were. And this may be another approach to get people to change their behavior because medical schools actually became quite competitive, they wanted to have a good AMSA score.

And so, some of them were actually creating conflict of interest policies, making their professors disclose and also putting even restrictions on what the professors could receive because they wanted to increase their score and look good to these medical students. So, that was a very powerful, I think, grassroots effort to push on transparency.

Baker: You’ve looked at various different topics. Do you see much difference in different industries?

It shows a timeline of when harmful effects of PFAS were detected by the company versus when they were published in the scientific literature and there was a big time lag. So, the company knew things years before what was getting published in the scientific literature.

Bero: Nope. Actually, we’ve deliberately compared industry tactics, actually, to influence research. And the more we look, the more similarities we see. Many years ago, we did a study looking at tactics used by pharmaceutical, tobacco, vinyl chloride, lead and asbestos industries and, again, we had some internal documents that were available related to those different companies and they were all using the same tactics. Nowadays, UCSF, the University of California San Francisco, has an archive of internal documents that now includes many, many types of documents, including documents from chemical companies. In one of the more recent studies I did, we looked at documents from 3M and other companies that make these forever chemicals called PFAS. We actually deliberately compared the strategies revealed in those documents to what we studied with other industries and they were the same. And I think one of the most compelling figures in that paper, it shows a timeline of when harmful effects of PFAS were detected by the company versus when they were published in the scientific literature and there was a big time lag. So, the company knew things years before what was getting published in the scientific literature.

Baker: Is there anything else that can be done to decrease corporate bias at research besides independence?

Bero: Well, I think there’s a lot being done now and my main heroes in this whole type of work I do are all the people who get attacked by industry because they are doing independent research. And I think that that’s something that… It’s been written about quite a bit but I don’t think people realize the scale of it. And so, one of the main things we can do to have independent research is to support independent researchers more. And I don’t mean financially, I just also mean through their university structures and through some of these attacks that they get from industry. So, that’s really, really important. And I think that some of the stuff I’m doing in the environmental area is to say that, when people are doing risk assessment of a chemical, they need to recognize that the industry-sponsored studies have a bias in favor of their product and they need to take this into account when they’re doing evidence synthesis.

I’m not saying that they should never look at industry-sponsored studies but I am saying that they should take that into account as a risk of bias and do a sensitivity analysis or some analysis that would control for that.

So, I’m not saying that they should never look at industry-sponsored studies but I am saying that they should take that into account as a risk of bias and do a sensitivity analysis or some analysis that would control for that and see if there really is a bias just related to the sponsor. So, I think one way to improve the independence of research is to just support it more and really take care of the researchers who are doing that independent work.

Baker: So, in preparation for this interview, I was reading one of your articles from last year and you were looking at, I think, a decade of work and you found that 80% of articles that were criticizing systematic reviews in policy had an industry chai compared to 35% of positive responses. And then you also found that, if somebody was responding to a systematic review, that they often didn’t respond in the same journal, they would respond in a different journal and that would be paywalled.

Bero: That’s a great example of an orchestrated industry response to an independent study. So, yeah, Adrian Traeger was doing Cochrane reviews looking at the effects of spinal cord stimulation on pain and found that, basically, when you assessed the risk of bias of the studies and if you looked at industry sponsorship as a potential risk of bias, then you found that the independent studies say there’s no effect of this intervention on pain and, in fact, there is some harm of the intervention. And when he published this Cochrane review, they were just… This orchestrated attack began and I’ve seen this over and over and over which is why I wrote that article with Adrian. It happens in all sorts of fields.

Baker: I know that, when you uncovered all this evidence that there were academics who were submitting, essentially, fake clinical research that one response that really concerned you was that some people conducting systematic reviews would say, “Okay, well, I’m just not going to include any research if it comes from a country that seems to have a high rate of faked clinical research.” And I wonder if you think that it would make sense to leave out research by industry?

Bero: Yeah, that’s a very contentious topic, actually, and there certainly are folks… I was just, yeah, giving a talk where this came up earlier this week at an environmental health meeting and there are certainly people who do research and commercial determinants of health who think that industry-sponsored research shouldn’t be considered at all. I’m actually not like that, I think that industry research should be assessed for risk of bias using these meta research methods. So, not just focusing on individual studies, but seeing if they really are driving the research agenda or not publishing all the data, that sort of thing. And, if so, then they need to be discounted because of that bias.

And there are tools for doing this, there’s a tool called the Navigation Guide that’s used in environmental health that Tracey Woodruff’s group has come out with so there are ways to do this. And the reason I say that is because, like anything, when you do a clinical trial—and I’m sure people get this—you get a statistical estimate of the effect. And so, you give a drug and 30% more of the patients who got the drug don’t have a heart attack which is the desired outcome and that’s considered a huge effect in a clinical trial. 30% of the treated population had the desired outcome. And when we do these meta research studies, it’s the same thing. We can say that drug industry sponsored studies overestimate effects of their drugs 30% of the time. So, that means that there’s a big bias there but it also doesn’t mean that every single study has an invalid result. So, that’s why I think we need to consider these industry sponsorship as a risk of bias and take it into account by discounting those studies. Depending on the area, it would be by a different amount.

Baker: Okay. So, be wary but don’t apply too broad of a brush.

Bero: Yeah. I’d say be very worried and don’t apply too broad of a brush. And I do think, in some areas, and this is interesting, I mentioned before how, in some areas, almost every single study will be funded by a sponsor. And in that case, I think we need to be very, very worried and say we don’t have the answer. Because if every single study that looked at effects of a particular drug are funded by the maker of the drug, then we don’t have anything to compare it to, we don’t have any independent research.

And interestingly, I was working with some patient groups with Cochrane several years ago where we were trying to make our abstracts into more plain language and one of the things we did not put in our abstracts were how many of the studies included in the reviews were industry sponsored. And the patients really wanted to know and I remember one of them saying to me, “Well, I would like to know if there’s four drug trials in this review and they were all funded by the company that makes the drug because I’m just not going to believe it then.”

Baker: I want to get back to the question that has motivated your career. What do you think can be done so that systematic reviews are better serving the needs of policymakers?

Bero: Well, I think policymakers need to have skills or helpers who can help them figure out what reviews come from a trusted source. As systematic reviews have become much more prevalent and, actually, much more useful and respected by policymakers, industry just cranks out systematic reviews like crazy. Just like original research studies, they may not agree with the results of independent reviews. One of the first studies I did was looking at reviews on studies of second-hand smoke and I looked at tobacco industry sponsored versus not. And this was the biggest effect size I’ve ever found but the tobacco industry funded reviews were about 80 times more likely to say that second-hand smoke was not harmful compared to reviews with other sponsors and that control for the methods of the reviews and all that.

Sometimes we don’t have studies for important policy questions and when a review is done, the answer is, “Oh, we’re uncertain, we don’t know because there’s not enough evidence,” and that’s a very, very disappointing outcome for a policymaker.

But the tobacco industry had been producing all of these reviews and getting them directly in front of policymakers saying, “See, we have a review of second-hand smoke, it shows it’s not harmful,” so they were directly competing with the independent reviews. And so, that’s the first thing. I think policymakers need to look for reviews from trusted sources and that have actually tried to … That have independence from industry sponsors. So, for example, Cochrane reviews cannot be sponsored by a company whose product is being assessed in the review. We don’t allow any corporate sponsorship of Cochrane reviews so we’ve just eliminated that as a risk factor for the review.

So, I think that’s one thing that policymakers need and I think another is, and this has been an eternal battle, is to get systematic reviews in line with the important policy questions because systematic reviewers review studies that exist, usually. And sometimes we don’t have studies for important policy questions and when a review is done, the answer is, “Oh, we’re uncertain, we don’t know because there’s not enough evidence,” and that’s a very, very disappointing outcome for a policymaker. That’s a little more complicated because that means we also have to get the original research in line with the policy questions and we need to do some good forecasting.

Baker: So the researchers know what’s needed and the systematic reviewers know what’s needed and that it’s there?

Bero: Yes, exactly, exactly.

Baker: What other things do you think are important in bringing good evidence to policy through evidence synthesis?

Bero: Well, we need to make them understandable too and, again, that’s also been a long battle because, just like a clinical trial, when you do a systematic review, you get a numerical result. So, I can say pharmaceutical industry sponsored studies are about 30% more likely to have a favorable result than those with other sponsors and then we always put an uncertainty level around that like plus or minus or 95% confidence interval. And it’s very hard to make a decision when there’s a lot of uncertainty around the effect. But I’m not a science communication expert but I know there’s a lot that can be done to help people understand uncertainty in a finding and how to deal with that.

There’s a lot that can be done to help people understand uncertainty in a finding and how to deal with that.

So, I think that’s another thing. Systematic reviews are complicated and there’s a lot that goes into them. There’s usually multiple analyses in one review and they may look at effects, they may look at harm, they may look at both and so then you have to weigh all that when making a decision about a product, for example. So, they’re pretty complicated and figuring out the best ways to summarize and convey those results is really important. But I would still say that there are much better assessment of the evidence than just looking at one or two studies out of context.

Baker: Lisa, this has been a fascinating conversation. To learn more about how bias impacts scientific research and what can be done about it, visit our show notes to find links to more of Lisa Bero’s work. Please subscribe to The Ongoing Transformation wherever you get your podcasts and write to us at [email protected]. Thanks to our podcast producer Kimberly Quach and our audio engineer Shannon Lynch. My name is Monya Baker, thank you for joining us.

Your participation enriches the conversation

Respond to the ideas raised in this essay by writing to [email protected]. And read what others are saying in our lively Forum section.

Cite this Article

Bero, Lisa and Monya Baker. “Uncovering Hidden Bias in Clinical Research.” Issues in Science and Technology (October 22, 2024).