Academic fraud may be the symptom of a more systemic problem

‘Academic fraud may be the symptom of a much more systemic problem’

OPINION - It is not surprising that a case of scientific fraud occasionally comes to light, according to associate professor Empirical Political Science Alex Lehr. ‘Many of us are doing our stinking best to maintain scientific integrity and produce rigorous research. But we do that mostly despite the incentives created by the academic system, not because of them.’

A recent case of scientific fraud by a – now dismissed – colleague at Radboud University prompted me to go off on a long rant on social media. Vox asked me to share it here in slightly edited form. This is not something I would normally be inclined to do, but in light of current trends in science and society, I guess it is good to speak openly and clearly about these things. So, here it goes, with the provision that I am not a methodologist and not a philosopher of science, just a practicing researcher.

Don’t hate the player, hate the game

A very understandable reaction to academic integrity violations is to view them as idiosyncratic, malicious actions of individuals. We shrug our shoulders and continue business as usual. I think it’s the wrong reaction.

The real issue here is the misalignment of incentives with the desired outcomes. We get rewarded for telling simple and clean stories – preferably the kind of stories others want to hear. For applying a credible looking veneer of rigorous science to those stories. For theoretical novelty. For demonstrating productivity with lots of publications, preferable “top-publications”. That’s what’s getting us the research grants, the career promotions, the media attention and the policy influence.

What we are not getting rewarded for, is being open about how complex, messy and uncertain research results can be. For articulating all those pesky assumptions our inferences are based on – or even for worrying about them too much ourselves. For showing how variable our results are, depending on the models we choose apply to our data. For being transparent about all the decisions we make as part of our scientific workflow, at best based on reasoned and reasonable trade-offs between practical feasibility and maximum rigor rather then self-serving opportunism.

For spending the time, effort and money needed for collecting high-quality data when there are cheaper short-cuts available. For showing the results of our analyses that don’t fit so nicely with the story. For admitting that the data are simply too noisy to provide a clear signal. For taking the time that is necessary to complete research projects that adhere to FAIR principles, rather than just pumping out another publication. For doing the kind of theoretical and empirical grunt work that is needed to contribute to the accumulation of knowledge instead of just telling interesting “new” stories.

By no means universally, with large variations between different areas of study and between academic communities, and with incremental improvements over time. But still way too pervasively.

I do believe that many of us are doing our stinking best to maintain scientific integrity and produce rigorous research. But let’s not kid ourselves, we do that mostly despite the incentives created by the academic system, not because of them.

So, by all means, hold individual scientists accountable. But also recognize that they might merely be responding to the incentives that we, as a scientific community, are creating. Be willing to consider that the known cases of academic fraud may well be the symptom of a much more systemic problem.

A small victory for open science

As much as we would like to take it for granted that science is always self-correcting, in many areas of study where repeated, independent replication is uncommon or difficult, corrections can only happen when errors in existing research are identified. This requires transparency. And while transparency alone is not sufficient for producing scientific integrity, it sure does create some helpful guardrails.

The fraud case at Radboud University came to light in part because at least some information was openly archived in the Open Science Framework (OSF) repository. Not so long ago, it likely would never have come to light at all. So, as painful as what happened may be, there is a real silver lining here. I hope that the lesson we take from this is to push for more open science. Because in many areas of study, it is still not the norm.

Light-switch

‘Any sufficiently crappy research is indistinguishable from fraud’. This quote is not mine, it’s Andrew Gelman’s variant of Clarke’s third law. But it highlights an important insight: scientific rigor is not a light-switch that’s either on or off. Some research is not outright fraudulent with malicious intent, it’s simply so shoddy that the difference becomes meaningless.

Yet, standards are so variable between researchers, and subject to so much change over time, that it’s unavoidable that what one considers perfectly valid research practices, others may view as hanging offences. This appears to have happened here, going by the reaction of the researcher involved. Science is hard, science is messy, and science is always changing.

How come someone felt compelled and justified to do science in this way?

So, someone violated scientific integrity and got dismissed (though for other reasons). The discussion we should be having is what this researchers’ scientific community, its culture, and the standing research and career policies, did to prevent this situation from even occurring in the first place. If what happened in this case is not in line with desired practices, how come someone felt compelled and justified to do science in this way? Are we sure it’s an extreme and idiosyncratic case, or are there now quite a few others out there thinking ‘ouch, this could have been me?’.

And to avoid misunderstanding, I’m certainly not singling out a specific research group, department, research institute or even our university. These issues are pervasive and systemic.

I don’t think firing someone without also having some critical reflection on the role of the community and the institute in the wider sense, is the best course of action. In the worst case, it might create an atmosphere of fear. Fear of being found out, fear of falling short of scientific standards, fear of being punished for honest mistakes. That kind of fear might just create more incentives for not providing full transparency on one’s research, and for creating plausible deniability and outsourcing risk (“it wasn’t me, it was my research assistant…”).

Deliberate and outright fraud is a no-go, but admitting to honest mistakes, being open to learn and revise, and being willing to retract publications when necessary, should be stimulated, and never be career-ending offences (just as “prestigious” research grants or “top-publications” based on shoddy research should never be career-making acts). That message should be a lot clearer.

Growth will be painful

I think one of the most frustrating issues for those of us that view these things as being problematic can be the inertia of academia. At the same time, it’s perfectly understandable. To a large extent, science is governed by scientists. Scientists that gained their positions by being academically successful. Leading change when the implication of that change is that practices that brought you success may no longer be acceptable can create quite a bit of cognitive dissonance. Good academic leaders are able to overcome themselves, but it takes some guts, humility and self-reflection.

Science is also a social endeavor, and that, too, makes change difficult. You will not always make friends by being the critical voice in the room, especially not if the academic culture is not one that takes kindly to criticism. Power dynamics make this even more difficult.