How to make the scientific method more…scientific
Lesson summary
Hi there everyone. I’m Jeff and this is Plain English, where we help you upgrade your English with stories about current events and trending topics. In each episode, I share a story about what’s going on in the world. And I also show you how to use a common English expression.
On today’s story: fraud in science. When I was a kid, I learned the “scientific method.” The scientific method has been in use since the 1600s, when science was first formalized.
The scientific method is seven steps. I’ll summarize them. You start with a question, develop a hypothesis, perform an experiment, analyze the results, and report your findings. Scientists most often report their findings in a paper published in an academic journal.
But all this relies on the ethics of the scientist. And as we’ll learn today, that’s not something that we can always take for granted . This is the first part of a two-part story. The next part will come out on Monday.
In the second half of today’s story, today’s episode, I’ll show you what it means to “cross a line.” Let’s get started.
Incentives and academic fraud
Academic journals are like the gatekeepers of science. For anyone in academia—anyone who’s a professor or wants to be one—publishing new, scientific findings in an academic journal is critical. And it’s not just hard sciences like chemistry, medicine, and biology; the same is true for history, psychology, economics, and others.
In fact, there’s a saying in English: “publish or perish.” That means, if you’re in academia, you need to continue to publish a lot of findings…or you’ll perish. Your career won’t go anywhere. So there’s a lot of pressure for academics to publish a lot of findings.
That would be great if all the professors, PhD students, and aspiring academics were producing useful findings, things that would add to humanity’s body of knowledge.
But that’s not always what happens. Often, authors submit papers that are irrelevant, badly interpreted or outright falsified.
There are some protections to prevent this. The most prestigious journals are peer-reviewed. That means that a group of experts in the field reviews all papers accepted for publication. The reviewers might look at the data. They might make methodological critiques. They might request that the authors think about something in a new way. They might raise questions about the conclusions. The goal is to make the paper as valid and useful as possible.
But the peer review process is not perfect. It’s not a quality control process. The reviewers don’t go into the lab; they don’t investigate the data and findings to make sure everything was done correctly. And there’s no way for them to know if the researchers acted ethically during the research process.
Here’s hypothetical example of a researcher acting unethically. Let’s say a researcher runs an experiment and his hypothesis does not appear to be supported by the evidence. Ordinarily , this would be the end of the line . Journals don’t like to publish papers where the hypothesis was rejected, where something was not found. So when that happens, it’s time to start over with a new hypothesis and a new experiment.
But that same researcher might be tempted to cross a line . That researcher might look at the data from the experiment and say, well, let’s just see what we can find here. He might then run analysis after analysis in the data, looking for any two things that appear connected, even if there’s no logical or theoretical basis for it.
If he finds anything interesting in the data, he can then create a new hypothesis that fits the data and—voila—write a paper and submit it to a journal. This is called p-hacking, named after a statistical measure, and it’s a shortcut around the scientific method. This is considered unethical, but the peer review process wouldn’t necessarily catch this behavior.
Or consider a different temptation. Imagine a researcher does an analysis and, once again, the hypothesis is rejected. That researcher might be tempted to change the data so that it does confirm the original hypothesis. It’s very possible to make small changes to a dataset to make a hypothesis work and, again, the peer review process might not catch it.
Why would a scientist do this? Remember: publish or perish. Academia is competitive. Top researchers need to continue publishing to stay at the top of their fields; aspiring academics need to publish to prove their worth and get good jobs. The vast majority of journal authors are honest. But too many are not: they’re tempted to act unethically to advance their careers.
Here’s another thing. Most professors are paid a middle-class salary, but they don’t make a lot of money—from their universities. But if a researcher makes a high-profile finding, that professor could be hired by private companies for consulting and speaking projects. So the temptation is there to exaggerate or even falsify academic findings.
A blog called DataColada has set out to investigate the quality of scientific papers. The professors who run that blog say that as many as two percent of all academic papers should be retracted. And they have looked closely at some famous papers and, they say, they have found evidence of fraud.
Ironically, several examples of academic fraud come from professors studying and writing about dishonesty. One influential research paper claimed to find that people were more honest when filling out a form if they signed that form at the top instead of the bottom. If it were true, this finding could be very useful in the real world. In fact, insurance companies hired the authors of that paper as consultants.
But the researchers at DataColada raised serious doubts about the quality of the data in the study. They found that two different data sets had been tampered with, all in a paper about dishonesty.
The lead author of that paper, Francesca Gino, said she never falsified data. She says she takes the allegations seriously and will address them in the future. She has been put on leave while her employer, Harvard University, investigates. Other researchers have tried to reproduce the paper’s findings, with no success.
But so what if some professor publishes a paper with falsified data? Who cares?
This is important because people often change their behavior based on what’s reported in academic journals. Tax agencies and insurance companies, for example, changed their forms to have customers sign on the top—all based on what might be fraudulent research. Other researchers might try to build on this knowledge by designing an even more complex experiment—but they’d be wasting their time if the first paper was fraudulent.
Wasting time and money is bad. But what about studies of prescription drugs or medical procedures? The journal “Nature” published an article that suggests that up to a quarter of clinical trials are problematic or entirely made up. That can have real-world consequences to patients’ health.
Just a few weeks ago, Harvard Medical School launched an investigation into several papers produced by its research hospital. Independent analysts found what they said were Photoshop copy and paste falsifications in the images used in medical research papers.
Now, though, there’s a new idea that can help correct the incentives and improve the quality of research. We’ll talk about that in our next story, number 650, which comes out on Monday, February 19.
Jeff’s take
I am not in academics; I don’t have a graduate degree. But I always thought that these academic journals would have the highest standards for quality. And I always thought that they would have controls to prevent fraud.
But it’s really not true. A lot of “data” in these studies is just Excel worksheets—anyone could change anything in an Excel worksheet! And how can Photoshop copy and paste errors make it into prestigious medical journals? When the authors are at a hospital associated with Harvard University? I just always thought there were higher standards and more controls for things like this.
Anyway—remember to come back on Monday because there’s a good idea to improve the quality of scientific papers.
Great stories make learning English fun
