Scientific datasets are riddled with copy-paste errors

The above data comes from a landmark paper in Parkinson's Disease research, which provided the first-ever evidence that the disease originates in the gut rather than the brain. The paper received media coverage from major outlets and has amassed over 3000 citations from other scientific papers. But the underlying data contains sequences of duplicated values that should belong to completely different individual mice. The dataset has been publicly available on Dryad - an open-access repository where scientists upload their raw data - for more than 8 years. Why didn't anyone notice the blatant copy-paste errors until now?

Before going into more detail about this case, let me give some background on how we detected the issue: It was flagged by a piece of software I started building last year, which was inspired by two cases of data fabrication that made the news in recent years. One by Nobel laureate Thomas Südhof's lab and one by spider ecologist Jonathan Pruitt. Both cases had publicly available datasets with entire blocks of copy-pasted data that seemed quite trivial to detect. I was curious what I could dig up by creating a program that would correctly flag those cases, and then unleash it on all datasets available in open-access repositories.

Together with a few volunteer contributors, we've finished reporting all cases from the first 600 datasets we've scanned. There were 18 cases we felt were serious enough to raise concerns. Here are 3 of the most exciting ones:

Case 1: A seminal Parkinson's paper

• Article: Gut Microbiota Regulate Motor Deficits and Neuroinflammation in a Model of Parkinson’s Disease Cell(2016) - Dryad dataset | Pubpeer comment (by contributor Ben Woden)

We took mice that were genetically predisposed to developing symptoms of Parkinson's and we just cleared out their microbiome - all their symptoms went away.

That's how the paper's senior author Sarkis Mazmanian summarized the findings of the study when he went on the Rich Roll Youtube channel:

These are the results he is referring to:

Graphs B, C, and D contain measurements of the motor function of different groups of mice. They tell a clean story where the ASO mice (genetically predisposed to developing the mouse model Parkinson's disease) take longer to complete the tasks than regular wild-type (WT) mice only when they have retained a normal gut microbiome (SPF, Ex-GF) - not in mice stripped of their microbiome (GF, Abx).

There are two issues with the data:

• There are two sets of 5 identical sequential numbers in the "Adhesive removal times" column, shared between the SPF mice (brought up with a normal, healthy microbiome) and the ExGF mice (raised with a stripped microbiome, but then had their gut re-colonized).
• There's a pair of sequences of 3 identical numbers within "Pole-descent time" data for the germ-free wild-type mice.

Verdict

It could be either a fat-finger mistake when editing the Excel file or deliberate tampering to cover up real data that didn't tell the right story.

The study has low sample size, so the impact on the conclusions of the paper is serious, even if we trust the rest of the data. The duplicated rows make up 50% of the SPF samples and 42% of the ExGF samples. While the motor function values are not the only data the authors rely on for their conclusions, they're necessary for showing that the gut bacteria actually caused Parkinson's-like symptoms.

The issue was reported in January, and the authors have so far not responded.

Case 2: Ostrich-snake mixup

• Article: Constraints on the evolution of toxin-resistant Na,K-ATPases have limited dependence on sequence divergence PLOS Genetics(2022) - Dryad dataset | Pubpeer comment

In this paper, the authors investigated how different animals have evolved resistance to a family of toxins called cardiotonic steroids (CTS). There's an arms race in nature where prey species produce CTS to protect themselves against animals such as birds and snakes, who in turn have developed mutations to help protect themselves against the toxin.

The authors created a blend of Ouabain (a CTS) and Na,K-ATPase (the protein that the poison targets) to get a dose-response curve showing how well different versions of the protein survived the poison at various doses.

Issues

The yellow cells are precise duplicates between the Ostrich/Sandgrouse data and the Xenodon data (a species of snake). More concerning are the orange cells, which are near-duplicates that differ by one or two digits, always ending with the same digit. For example, 0.538 on row 39 becomes 0.518 on row 71. There are six such pairs out of eight total non-duplicate pairs.

In her response, the lead author Shabnam Mohammadi concedes that this probably is a copy-paste error. For the suspicious orange cells she gives the following explanation:

We suspect that they [the cells with one-digit tweaks marked in orange] stem from measurement variation among multiple reads of the same plate, and that the Ostrich and Xenodon results were copied from different readings of the same plate. However, since the original reader outputs are not available, we cannot confirm that this is the source of the variation.

Verdict

Could the one-digit tweaks be caused by multiple readings of the same plate as the author theorizes? My best guess is no, but there is enough uncertainty to give me pause.

When doing multiple readings of the same plate we would generally expect many of the values to jump around a tiny bit. Instead we see almost all of the values staying the exact same except 8 that change by a lot. In email correspondence, the author provides a potential explanation for this: before taking the readings they add a chemical to stop the reaction, which should stabilize the values. But some wells will stabilize faster than others, which could explain why those 8 values still varied by so much.

That still leaves the issue of the 6 pairs of values that each happen to end with the same digit. It would be supremely unlikely for this to happen purely by chance.

The BioRad model 680 is a basic machine that records raw light measurements as it sees them without any post-processing, but perhaps it's still possible that it introduces some bias that makes it more likely for readings to end on the same digit.

The alternative explanation is that the authors either accidentally or deliberately copy-pasted the Ostrich/Sandgrouse data on top of the Xenodon data. Then they might have manually tampered with the data to make it better suit their hypothesis or make it look less suspicious, not realizing that the copy-and-paste would leave behind this signature. However, the fraud theory suffers from there not being any obvious reason for why the authors would tamper with these specific values.

The authors state that they will try to replicate the process and report what they find on Pubpeer.

Case 3: Scrambled fish sizes

• Article: Behavioural individuality in clonal fish arises despite near-identical rearing conditions Nature communications(2017) - Dryad dataset | Pubpeer comment (by contributor Ben Woden)

Background

This is a paper from 2017 about fish personalities. They took a bunch of genetically identical fish, then checked how much their swimming behavior varied in terms of distance traveled. The idea was to see how much their behavior differed while keeping genetics and environment identical. Besides movement, they also took the size of each fish into account to check that any difference in movement was not just a result of bigger fish moving differently from smaller fish.

Issues

The SL column is the length of the individual fish measured in millimeters. Can you spot anything that looks out of place?

It looks quite strange that 3 different individual fish (ID 5, 8 and 10) all have the exact same sequential size measurements. But wait, can an individual fish really get >10% shorter over the