It's graduate school application season. Pay attention to research integrity!

When choosing a research group, ask about their attitudes towards research transparency and reproducibility

I am writing this for folks applying to graduate school (and, to a much lesser extent, the schools hoping to recruit them).In brief, I describe here how to judge a school not only by conventional criteria but by other things that matter even more. Chief among these is the research integrity track record and promise of any place where you make the effort to apply, and certainly before accepting an offer. My discussion will be focused on the US, where application deadlines are coming up in the next few weeks, but I suspect it will also be helpful for those elsewhere, and even scientists in, or transitioning into, industry positions. 

Devil in a research lab, by Bing AI

For those in the US and Canada, where you apply to a university and not necessarily to a particular research group, the first piece of advice, which is not unique or new, is this. Figure out which groups you are interested in being part of. There should be at least one that works for you, or do not apply to that school. Don’t try to reverse engineer from “this place is famous, so there must be something I can do there”. Read up on the groups that interest you, and understand what they do. At the very minimum, if you describe your thoughts about them, it will make your application more compelling. But broadly speaking, you will be spending multiple years at the place and not at the university-at-large but in one single lab, so you can shape your future by studying up.

The low hanging fruits are to pick up some key words from the general snippets available on their websites. You may also google them or even check their social media, though the representation of senior scientists there is fairly patchy - they are old! It is common to check the list of publications. Many have a profile on Google scholar, where the first thing you get to see is their h-index and the number of citations, and because of how Google sorts the list you see at the top all the papers in prestigious journals, like N****e and S*****e. I understand and sympathize with the impulse to analyze this, at least as a starting point. You may feel that publications reflect the productivity of the group, because you don’t want to be stuck in a dead-end project, or because you want the best chance of landing a top position after graduate school. You may feel that this depends on what kind of papers you will be able to write. I cannot say this is not true today, though I hope this changes over time.

I encourage you to study deeper, however. I recognize that this takes extra time. And if you are applying to 10+ schools the effort can add up. Still, it is worth it, at least for those places that communicate with you, invite you for a meeting online, or a visit to campus. You can open some of the papers that sound interesting to you and read them. Try to understand what was done. Try to assess the evidence for yourself and glean the research style from how the paper is written. Does it sound confident and make strong claims? Is it nuanced with a lot of technical details? Some of this can be evaluated without having the background knowledge, especially if you look at papers from different groups side-by-side.

The reason I am suggesting this is because there is something important about the scientific process that is often misunderstood. People primarily experiencing science through their classroom education, such as undergraduate students, are used to working with textbooks. Books contain facts that are vetted and verified, and you are not exposed to the possibility of questioning them. Active research is different in that it is fluid - conclusions of any paper can be overturned, or evolve when new findings are made. This happens fairly often. But the full story is more complicated.

In fact, large swaths of science are either never attempted to be reproduced (for example because they are too specialized), or their reproduction fails. This is known as the ‘reproducibility crisis’. The awareness of this crisis by practicing scientists is not great, and many of them think that it does not concern their discipline or them personally. But in reality, challenges in reproducing results are related to how we do science, and not to which science we do. In many cases, it is human nature that pushes a scientist to make an irreproducible claim in their paper. Everyone wants to report a big discovery, and the person easiest to convince is often yourself!

Why is this not caught? Aren’t all published papers reviewed by peers? There are many reasons. Some of the key ones are these. Reviewers often do not have the time to check the entire work, they skip through the paper until they get an overall impression and compare this to their expectations. On top of this, most papers do not provide enough data to really scrutinize the claims - just 3-5 figures out of thousands of measurements! Finally, there are so many journals out there: you can shop your paper around until it gets accepted somewhere - even if the referees at the first journal found mistakes in your work. Remember, most referee reports are confidential, so editors at other journals don’t get to see them. Journal prestige does not make it immune to errors, in fact more famous journals attract more unreliable claims because publication in them is highly coveted. Many retractions happen at top journals.

Back to your review of the groups that have caught your interest and where you might want to work. When you are reading their papers, I recommend installing the Pubpeer plugin in your browser. Pubpeer is a platform used for collecting comments on papers post-publication. Many of the comments are critical. If a paper you are reading is mentioned on Pubpeer, the plugin will let you know. You will be able to read what other experts are saying about the paper. Pubpeer is not universally used by all, but this is one straightforward way to find out whose work is coming under scrutiny by peers. Another resource you can check is Retraction Watch, or the Retraction Watch Database - does the group you are interested in have papers there? Why were they retracted? Remember, a retraction does not automatically mean a research integrity violation. Sometimes honest mistakes are found, and that is okay. At other times, research integrity violations are mischaracterized as honest mistakes, so look for patterns, like more than one retraction or many criticisms of the same group.

If you get to visit your prospective graduate school, you may be able to ask about the climate in the group. People usually ask about working hours, social atmosphere, i.e. friendliness, treatment of women and minorities, expectations from a group member. These are all vitally important questions, and some research integrity issues arise  from the work culture. For example, pressure to deliver results can trigger data fabrication.

I suggest that you also ask if there were any research integrity type situations. If anything significant happened in the past, somebody might tell you. But some of the incidents may be confidential. You can also ask related questions. Does the group share their data? If the answer is ‘no’, that is potentially a red flag, because this can create a situation where true findings are hidden. Where is the data stored? If there is not a system of preserving data, then there is no way of checking the findings, and thus no idea that research should be accountable. Many funding agencies require a data management plan, but is it actually followed in the lab? Do researchers share their code? Do they post their papers on a preprint server such as arXiv? Why do they make their choices?

You could also check out the university policies for evidence of transparency. But I cannot recommend evaluating a place based on their institutional stance towards research integrity, open science, reproducibility etc. Experience shows that if you go high enough in any organization, you will likely be disappointed. For instance, my own University just issued a policy towards data that states that “…Research Records shall be available only to those who need such access and to the minimum amount necessary”. Fortunately many faculty are significantly more transparent, releasing and sharing data for all our recent publications and many older ones.

On the flip side, even if an institution has a declared progressive pro-transparency policy, for instance in some European countries, they may not be enforcing it. Institutions tend to cover up misdeeds committed within their walls, because they think it helps them preserve their reputations, keep or win funding, or avoid penalties. I think this is misguided and by acting this way they are only hurting themselves. And of course they are hurting science, and the junior scientists involved. 

If you end up in an actual research integrity situation at one point in your career, you may unfortunately find that you are on your own. The people in charge of investigating this at your school will act like they don’t believe you, they will take forever to reach conclusions, and you may experience retaliation. In this case I recommend reaching out to somebody outside with similar experience who may be able to help. I hope this changes, but it appears to be this way at the moment. Examples of universities not acting on clear allegations of misconduct are plenty. And in some cases, universities have to pay for their actions. But in others, nothing happens to the people involved.

So your best bet is to try to avoid a group where the research culture may create a situation like this. The person who can make the most difference is the principal investigator. And this is why I recommend trying to figure out the attitudes towards transparent, reproducible research in the laboratories you are interested in and letting that inform where you apply. Good luck!