The real lesson from Untermassfeld, Pedra Furada, and the Cerutti Mastodon is that the peer reviewed journal has become counter-productive.

My recent discussion of Utermassfeld and Pedra Furada focused on criteria for accepting assemblages as archaeological, but both episodes, as well as that of the Cerutti Mastodon claim, have implications for the traditional peer reviewed journal system that are worth discussing.

All three present very surprising and ultimately poorly supported claims. All three claims were published in top journals: Pedra Furada in Antiquity (Boëda et al 2014), Untermassfeld in Quaternary International (Garcia et al 2013, Landeck and Garcia Garriga 2017) and The Journal of Human Evolution (Landeck and Garcia Garriga 2016) and the Cerutti Mastodon in Nature (Holen et al 2016). Two of the three claims (Pedra Furada and Cerutti Mastodon) were accompanied by supportive comments and discussion in their respective journal issues. All three support their claims mainly with arguments known to be problematic since the early phases of the eolith controversy a century ago, namely the presence of claimed lithic artefacts co-mingled with a natural geological matrix, and claimed human-processed bones co-mingled with background environmental samples. All three claims received serious examination and critique only after publication.

Pre-publication peer review as filter

The core benefit of the traditional review journal is allegedly that it acts as a filter on quality of claims in order to focus the attention of the community on the most productive work. The cases discussed here, published in leading journals are clear evidence that pre-publication peer-review does not act as an effective filter on quality of claims. This is not surprising. Given the heavily inter-disciplinary nature of the work being published in top journals these days, single editors appointing two or three reviewers is a strange way to run a quality filter.

The journals did succeed in getting the claims in front of many people who could properly evaluate them. That second function of dissemination to foster evaluation, however, does not require a peer reviewed journal. It does not require a journal at all.

The narrative recently floated in Nature (Callaway 2017) that the problem with Untermassfeld is one of fraud and data provenancing is a red herring. The claim that Untermassfeld is an archaeological site can easily be shown to be unsupported, regardless of stolen bones and collection intrigues. The idea that if only the journals impose even more filters on the publication of claims makes sense only if you accept the basic assumptions of the peer-reviewed journal system. The other two cases considered here show that the problem is broader and more systemic.

The fact that journal slots are limited (and the higher ranked the journal, the more limited the slots) and that publication times are long, slows down dissemination and in many cases makes it impossible (i.e. if the paper is rejected). If the traditional pre-publication review process does not act as a good filter on quality of claims, then its only remaining side effect is gate keeping, and it is essentially indefensible. Its only real function is to sustain the prestige economy of academia.

Costs and benefits of the peer reviewed journal

The cost of the journal system is that it limits the number of claims available for evaluation and that it slows down their dissemination. If its main claimed benefit is imaginary, the traditional journal is worse than useless in the current context.

All three claims would have been disseminated and evaluated much faster without the review process. All three claims would have been more widely available and potentially even better and more rapidly evaluated by a much wider audience if they had not been published in a traditional journal but if they had simply been posted to publically accessible repositories or blogs.

Some say that removing the filter of pre-publication review and the supply limit of journal slots will expose us to an unmanageable flood of claims that we won’t be able to properly evaluate. What many forget is that it will also increase the number of people who can examine those claims by making them more widely available instead of locking them behind journal walls.

The emerging post-publication review system

The least well-supported claims are easy to dismiss. Their evaluation requires only a small investment of time and energy.  Most people will not be motivated to examine most claims. Even today on social media, where claims are very rapidly disseminated, a division of labour emerges for their evaluation. Evaluators self-select instead of being appointed by editors.

The crucial feature of this emerging post-publication review system is that the evaluations are disseminated just as rapidly and as widely as the claims themselves. Even a tweet or a Facebook post can frame an effective critique of the central claim of an article.

The production of claims is the easy part of scholarship. Analyzing data and marshalling argument in support of claims is more difficult and time consuming. Evaluating the claims is the most difficult but also the most critical part of the process. It follows that claims must be disseminated as widely as possible, that the work of evaluation must be distributed as widely as possible, that evaluations themselves must be widely and rapidly available, and that limited journal slots and pre-publication gate keeping seriously hamper scholarship.

To put it in cultural evolutionary terms, the production of claims produces diversity, their easy and barrier-less dissemination makes them available for selection, which is much better performed by the broadest possible set of minds rather than by a few arbitrarily chosen ones.

The implications are clear. Forget the journals. Open the gates. Let people make claims and disseminate them. Let them support their claims. Let everyone examine the claims if they want to. Let everyone disseminate their evaluations of the claims.

References

Boëda E, I Clemente-Conte, M Fontugne, C Lahaye, M Pino, G Daltrini Felice, N Guidon, S Hoeltz, A Lourdeau, M Pagli, AM Pessis, S Viana, A Da Costa, E Douville 2014a. A new late Pleistocene archaeological sequence in South America: The Vale da Pedra Furada (Piaui, Brazil), Antiquity 88:927-955.

Callaway E 2017. Archaeologists say human-evolution study used stolen bones, Nature News. https://www.nature.com/news/archaeologists-say-human-evolution-study-used-stolen-bone-1.22984?WT.mc_id=TWT_NatureNews&sf166021520=1

Garcia J, G Landeck, K Martinez, E Carbonell 2013. Hominin Dispersals from the Jaramillo subchron in central and western mediterranean Europe : Untermassfeld (Germany) and Vallparadis (Spain), Quaternary Interational 316 :73-93.

Holen SR, TA Deméré, DC Fisher, R Fullagar, JB Paces, GT Jefferson, JM Beeton, RA Cerutti, AN Rountrey, L Vescera, KA Holen 2016. A 130 000 year old archaeological site in sourthern California, USA, Nature 544:479-483.

Landeck G and J Garcia Garriga 2016. The oldest hominin butchery in European mid-latitudes at the Jaramillo site of Untermassfeld (Thuringia, Germany), Journal of Human Evolution 94: 53-71.

Landeck G and J Garcia Garriga 2017. New taphonomic data for the 1 Myr hominin butchery at Untermassfeld (Thuringia, Germany), Quarternary International 436:138-161.