Journalology #94: Peer Review Week 2024



Hello fellow journalologists,

Many of us took part in Peer Review Week, which ends today. The sheer scale of the number of events was overwhelming (see here). You can watch my own contribution, alongside Danielle Padula from Scholastica, on YouTube by clicking the play button below:

video preview

We talked about how the role of a journal editor may be affected by advances in technology.

(My 8-year-old son was very impressed that I’m a ’YouTuber’, but he complained: “I didn’t understand anything you said”. Hopefully you will fare better.)

As an industry we’re more than a little obsessed with peer review and it’s worth stopping to consider the implications of that and what the opportunity costs might be. A huge amount of effort goes into PRW each year, but are we making progress on the topics that matter? Is there too much discussion and not enough action? It feels as though we’ve been talking about the same topics for the past 25 years (and probably longer).

For me, the number one priority is to increase the diversity of the reviewer pool. We need to share the peer review load more widely and encourage more diverse viewpoints to be aired.

We use the term “peer-reviewed journal” a lot, because we think that provides a veneer of respectability. But of course a journal that’s willing to rely on a two sentence peer-review report is very different from one that has created a culture whereby the journal’s advisors regularly provide in-depth, considered reports. More journals should publish the peer reviewers’ reports alongside the original research papers to increase transparency and trust.

I’d also like to see more journals publish the name of the handling editor alongside the paper. We expect corresponding authors to take responsibility for the paper; should the handling editor be named too? This is especially important on high volume journals where the editor-in-chief is unable to read every paper in depth. PLOS has been doing this since its inception.

In this week’s newsletter I’ve created a sub-section under the Opinion heading for articles related to peer review. This is by no means exhaustive. I struggled to keep up.

If you’re a journal editor or work with journal editors, please consider joining the Journalology coaching programme; you can read more n the box below. I can help you to make your journal more impactful (in every sense of that word).

Become a better editor: Journalology coaching programme

Creating an impactful journal is hard, especially for busy academics trying to fit in editorial work alongside research, teaching and other commitments.

Unfortunately, despite the size of the market there are relatively few resources available to help editors hone their skills and get better at their craft.

Most coaches work across multiple industries and are unable to provide useful insight into scholarly publishing. The Journalology coaching programme is different. I’ve got a proven track record, as both an editor and as a publisher, and can help you to create more influential and impactful journals.

News

‘We are embarrassed’: Scientific rigor proponents retract paper on benefits of scientific rigor

A high-profile paper about ways to improve the rigor of research papers has been retracted after critics attacked its own rigor. The study, published on 9 November 2023 in Nature Human Behaviour, purported to show the benefits of rigor-boosting measures including so-called preregistration—announcing the goals, methods, and other planned features of a study ahead of time—large sample sizes, and methodological transparency. It reported that these measures boosted the “replicability” of 16 findings in social-behavioral science to 86%, far more than the 30% to 70% reported in some analyses.
“Editors no longer have confidence in the reliability of the findings and conclusions reported in this article,” the journal said in a retraction note published yesterday.

Science (Martin Enserink)

JB: This story originally broke in The Chronicle of Higher Education, which you can read here: This Study Was Hailed as a Win for Science Reform. Now It’s Being Retracted. The Nature Human Behaviour Matters Arising comment (in which independent researchers criticise the original study) can be read here.

Embarrassing indeed...


Publisher adds temporary online notifications to articles “under investigation”

Some journal articles on the Taylor & Francis website now bear a pop-up notification stating the papers are “currently under investigation.”
The publisher began adding the notices to articles such as this one in June, according to a spokesperson, as a way to inform readers about an ongoing investigation “so that they can exercise appropriate caution when considering the research presented.”
Like the “editor’s notes” posted on Springer Nature articles under investigation, Taylor & Francis’ pop-ups only appear on the publisher’s website, not in databases where researchers might be searching for papers.

Retraction Watch​ (Ellie Kincaid)

JB: There’s some logic to this approach. Expressions of concern stay with a paper forever, whereas this is a temporary measure that acts as a yellow flag for readers. The big downside is that the warning message does not appear on indexers’ websites. Also, the messages may not appear because of ad blockers.

According to the T&F website:

If deemed necessary by the Publishing Ethics & Integrity team, a pop-up notification may be temporarily added to the online version of an article to inform readers an article is under investigation. This is not a permanent note (unlike an Expression of Concern, Correction, or Retraction notice), but is to indicate an investigation is in progress. Please note, these are not added to every article under investigation.

IOP Publishing shines a light on the power of constructive feedback in peer review

In celebration of Peer Review Week, IOP Publishing (IOPP) is raising awareness of the importance of constructive, respectful peer review feedback and how innovations can help to create a positive peer review culture. The campaign shares examples of unhelpful reviews received by researchers early in their careers through a short video and highlights the emotional toll of such feedback. It then contrasts these hurtful comments with their impressive academic achievements.

IOP Publishing (Faye Holst)

JB: See also this article in Nature: ‘Substandard and unworthy’: why it’s time to banish bad-mannered reviews


A responsible framework for evaluating the societal impact of research

Funders, governments, and institutions across the globe are increasingly expected to demonstrate the societal impact of research investments, particularly in areas such as health, climate change, and technological progress. However, there is currently no established framework for such an evaluation.
To meet this challenge, the Institute for Scientific Information (ISI)TM has developed a framework to evaluate and visualize the societal impact of research in a responsible manner. Read the report to learn more about this innovative framework in more detail.

Clarivate​ (announcement)

JB: You can download the report here. The authors are: Dmytro Filchenko, David Pendlebury, Nandita Quaderi and Jonathan Adams.


1 in 7 scientific papers is fake, suggests study that author calls ‘wildly nonsystematic’

Heathers’ study pulls data from 12 different analyses from the social sciences, medicine, biology, and other fields of research. All those studies have one thing in common: The authors of each used various online tools to estimate the amount of fakery taking place in a set of papers.
“There’s a really persistent commonality to them,” Heathers said. “The rough approximation for where we end up is that one in seven research papers are fake.”
Heathers said he decided to conduct his study as a meta-analysis because his figures are “far flung.”
“They are a little bit from everywhere; it’s wildly nonsystematic as a piece of work,” he said.

Retraction Watch​ (Dalmeet Singh Chawla)


EDP Sciences Releases 2024 Transparency Report for Mathematics Journals

EDP Sciences and SMAI (Société de Mathématiques Appliquées et Industrielles) have just published their latest transparency report for 2024, marking the fourth consecutive year of open access (OA) publication under the Subscribe to Open (S2O) model – and the fourth transparency report. This initiative continues to enhance accessibility while maintaining a strong commitment to transparency and financial accountability for the six mathematics journals involved, promoting an equitable OA model that benefits both readers and authors.

EDP Sciences (announcement)

JB: This extract is also worth noting:

Financial sustainability: The number of subscriptions continued to increase. Revenue from traditional subscriptions covered 51% of the publication costs in 2023, with additional funding from institutions and supporters. The average publication cost across the journals is €950 per article, and a concerted effort to reduce costs is ongoing. In 2023, 72% of costs were covered by combined revenue sources, marking an improvement in the financial outlook of the journals, though some deficit remains.

You can read the report in full here.


MDPI New Journal Proposal—Invitation to Shape the Future of Open Science Together

In order to enhance exchange and cooperation with scientific researchers and share the results of open science, MDPI invites experts and scholars from various fields to submit proposals for new journal collaboration opportunities. If your proposal is approved, you could take on the role of Editor-in-Chief, Deputy Editor-in-Chief, or Editorial Board Member of the journal. This position allows you to manage the manuscript review process alongside our editorial department, participate in calls for papers, strengthen your network with scholars in the field, expand scientific research cooperation, enhance your personal and academic influence internationally, and play a leading role in academia.

MDPI (announcement)

JB: Is this a new take on the guest-edited model, but without the ’guest’ part?


Meet the candidates and vote in our 2024 Board elections

Each year we do an open call for board interest. This year, the Nominating Committee received 53 submissions from members worldwide to fill four open board seats.
We maintain a balanced board of 8 large member seats and 8 small member seats. Size is determined based on the organization’s membership tier (small members fall in the $0-$1,650 tiers and large members in the $3,900 - $50,000 tiers). We have two large member seats and two small member seats open for election in 2024.

Crossref​ (announcement)

JB: You can read the candidates’ statements here. I don’t have a vote, but if I did I would vote for Amanda Ward and Dan Shanahan in the Tier 2 category. Both are excellent. Amanda, in particular, has deep knowledge and experience of research infrastructure. She also has a strong track record of getting stuff done. (Aaron Wood may well be excellent, too, but our paths haven’t crossed.)


Gender inequity persists among journal chief editors

We determined the gender of chief editors at the 200 top-ranked scientific journals in the Scimago database (see go.nature.com/3xwuies), on the basis of their names and publicly available images. At 174 (87%) of these journals, a single person held the position; of them, 62 (36%) were women. There was no statistically significant difference in this figure between medical journals (32%) and non-medical journals (38%) in the sample.

Nature (M. Hossein Nowroozzadeh & Somaye Tafraghe)

JB: This is a correspondence letter in Nature. This is clearly an important topic, but assigning gender based on names and mug shots doesn’t seem like the most robust methodology to use. Regardless, the trend is clear and the message is obvious: journal owners need to appoint more women to Editor-in-Chief roles. We have been discussing this for decades. Yes, editors have contracts and change takes time. But it has taken far, far too long to reach parity.

The authors go on to say:

The percentage of female chief editors was 31% in the United States, 49% in the United Kingdom and 9.5% elsewhere. Notably, among the 43 Nature-branded journals in the sample, 51% had female chief editors, compared with 31% for other journals.

The best journals show leadership and challenge the status quo, as the Nature journals have done here.

The 9.5% figure for editors based outside of the UK and USA is a shocking figure. Collectively we have to do better.


Data integrity concerns flagged in 130 women’s health papers — all by one co-author

A team of scientist–sleuths has flagged data-integrity concerns in 130 studies authored by the same biomedical researcher, a specialist in women’s health and gynaecology, and his colleagues. The sleuths published their findings in a peer-reviewed paper earlier this year.
Some of the studies that were identified as potentially problematic have been cited by other researchers or included in analyses that could inform clinical practice. The number of papers being questioned is among the highest by a still-active life-scientist, say some specialists.
The 130 studies were published between 2014 and 2023 and report the results of clinical trials and other research on maternal and women’s health. The highlighted problems include oddities in reported statistics, unfeasible results and text that is identical to other papers.

Nature (Mariana Lenharo)

JB: The scale of this is gobsmacking. This news story is well reported and is a must read.


“The Conversations” by Wiley Ushers in New Era of Dialogue and Innovation in Scholarly Publishing

The premiere episode explores the current state of publishing technology, how it hasn’t kept up with consumer technology, why that might be, and how it needs to change. The experts discuss the need for trust, quality, and robust peer review to remain the focus for publishers as AI adoption grows.
The episode also explores a necessary paradigm shift towards more dynamic, "format-less" publishing models that enhance accessibility, searchability, and usability of data for both humans and machines—a two-pronged approach to sharing information.

Wiley (press release)

JB: A high-production talk show for scholarly publishing? Yes please! You can watch the first episode here.


Announcing the Launch of the Peer Review Terminology Website | A Collaboration Between STM and NISO

As Peer Review Week kicks off, STM is excited to announce the launch of the Peer Review Terminology website—a new resource designed to bring greater clarity, consistency, and transparency to the peer review process. Created through a partnership between STM and NISO, this initiative reflects our shared mission to enhance scholarly communication and streamline research practices.

STM (announcement)

JB: This is helpful. You can read the terminology document here.


Other news stories

‘A Massive Scandal': Does a Landmark Lawsuit Against Academic Publishers Have Legs?

Code Ocean and Springer Nature expand partnership across the Nature Portfolio

AIP Publishing’s Newest Open Access Journal, APL Electronic Devices, Now Open for Submissions

Introducing a Prototype Directory of Organizations and Initiatives in Open Scholarly Communication

Becaris Publishing Enhances Real-World Evidence Dissemination with Cadmore Media Support

IEEE: latest publisher to join GetFTR

MD Anderson launches first scholarly journal: Advances in Cancer Education & Quality Improvement (ACE-QI)

ESE expands journal portfolio with the launch of Environmental Endocrinology and Obesity and Endocrinology

ReadCube expands literature management with new AI Assistant and comprehensive search

World’s Largest Computing Society (ACM) Signs Major Open Access Agreement With Consortium of 30 French Universities and Research Institutions


Opinion

Recommendations on the Use of AI in Scholarly Communication

We strongly recommend that editors, journals, and publishers develop policies related to the use of AI in their publishing practices and that they publish those policies on the journal’s website. For instance, policies for authors should be listed in the journal’s ‘Instructions to Authors’ or ‘Submission Guidelines’, while policies for reviewers should be in the journals’ ‘Reviewer Guidelines’. The policies should be clearly communicated to authors and reviewers through email communication and in the online submission system. The policies should also include information on how parties should raise concerns about possible policy infringement, consequences that any party might face in case of infringement, and possible means to appeal to journal decisions regarding these policies. Additionally, the policies should be supplemented with educational resources or links to information on the responsible use of AI (see for example Guidelines on the responsible use of generative AI in research developed by the European Research Area Forum). Editors should also consider announcing the release or update of AI policies through published editorials.

EASE (guidelines)


Using PubPeer to screen editors

As I have emphasised before, the gatekeepers to journals are editors. Therefore it is crucial that they are people of the utmost integrity and competence. The growth of mega-journals with hundreds of editors has diluted scrutiny of who gets to be an editor. This has been made worse by the bloating of journals with hundreds of special issues, each handled by "guest editors". We know that paper millers will try to bribe existing editors, and to place their own operatives as editors or guest editors, use fake reviewers, and stuff articles with irrelevant citations. Stemming this tide of corruption would be one effective way to reduce the contamination of the research literature. Here are two measures I suggest that publishers should take if they seriously want to clean up their journals.

​BishopBlog (Dorothy Bishop)


Advancing Research Integrity with SuSy Innovations

SuSy is MDPI’s in-house online submission system. It was launched in 2011 as a submission-only platform. The tool was designed with flexibility in mind and has since been updated to include additional functions based on feedback from academic editors, authors, and reviewers.
...
By having an in-house tool, MDPI can aspire towards its aim of simplicity. The whole process from submission through to peer review and editing is carried out in one place. This enables the quick turnaround of scholarly papers without sacrificing quality, with research typically published within 5‒7 weeks of submission.
Furthermore, to support the sustainability of scientific publishing, MDPI has a service in which publishers can use a modular and flexible version of SuSy for their own journals, Journal Management System (JAMS). JAMS is a full suite of journal management tools to support you throughout the publishing process.

MDPI Blog (Jack McKenna)

JB: The two publishers that have grown the fastest over the past 5 years (Frontiers and MDPI) both developed their own peer review system. Does correlation equal causation? Probably.

MDPI also published blog posts about Eureka “an Artificial Intelligence tool designed to assist in the reviewer selection process”, the MDPI Online Proofreading tool, and the society publishing team.

I’ve previously noted that MDPI has been a ’black box’ in the past, so it’s good to see the company being more open about their tools and processes.


CMS Strategies for Scholarly Publishing: A Hybrid Approach

The scholarly publishing sector has historically been a bit “snowflakey” when it comes to publishing solutions, preferring to use 'publishing specific' tools. While bespoke publishing solutions are certainly necessary for managing scholarly workflows, the same is not true for website content management—despite what many might think.
This hybrid strategy demonstrates that leveraging existing open-source infrastructure like WordPress in innovative ways can provide immense value. By combining the malleable, template-driven capabilities of FLAX with the user-friendly and extensible nature of WordPress, scholarly publishers can achieve the best of both worlds: a powerful, cost-effective solution that meets the unique needs of scholarly communication while also benefiting from the scale and polish of one of the most widely used CMS platforms in the world.

Robots Cooking (Adam Hyde)

JB: This week Adam also published another blog post — PubPub's Strategic Pivot: Insights from a Similar Journey — which technology teams may benefit from reading.


Peer Review Week

Some Thoughts on the Promise and Pitfalls of Innovation and Technology in Peer Review

Is AI the Answer to Peer Review Problems, or the Problem Itself?

Innovation and Technology in Peer Review: Some User Perspectives

From Bottleneck to Breakthrough: AI’s Role in the Future of Peer Review

Randomisation can resolve the uncertainty at the heart of peer review

If generative AI accelerates science, peer review needs to catch up

EDP Sciences Celebrates Peer Review Week 2024

Ensuring scientific integrity through rigorous peer review

Harnessing technology to boost diversity, equity and inclusion in peer review

Privatizing Review: IRB Lessons

What role can technology play in the peer review process?


Other opinion articles

Detecting genAI: beside the point?

From siloed data to shared knowledge: How WorldFAIR is shaping the future of research - International Science Council

Publish OA, support the SDGs: Make your research impactful

Cooperative open access can right many of publishing’s wrongs

Publishers must not let AIs train on academic output


Three ways I can support you


And finally...

Newsletters can be a helpful way to keep up to date with what's going on. If you’re interested in how AI could affect scholarly publishing then you may want to sign up to: Augmenting Scholarly Publishing: Intelligent Emerging Tools by Chirag Jay Patel and Chhavi Chauhan.

I was also interested to see that the Nature Briefing brand has expanded once again, with Nature Briefing Careers launched recently. You can sign up to that newsletter here.

Until next time,

James

P.S. This made me laugh. When they say ’early’ how early do they mean? Can toddlers apply?!?


113 Cherry St #92768, Seattle, WA 98104-2205
Unsubscribe · Preferences

Journalology

The Journalology newsletter helps editors and publishing professionals keep up to date with scholarly publishing, and guides them on how to build influential scholarly journals.

Read more from Journalology

Subscribe to newsletter Hello fellow journalologists, A few hours after hitting the ’send’ button on last week’s newsletter I saw a tweet about Heliyon — a broad-scope journal published by Elsevier and hosted on the Cell press website — being put ’on hold’ for indexing by Web of Science. That piqued my interest because I had just started drafting an article about Heliyon and Cureus, the two journals that grew the fastest in 2023, for the Digital Science Dimensions blog. I naturally wanted to...

Subscribe to newsletter Hello fellow journalologists, Are scholarly publishers a force for good? Many academics certainly don’t perceive commercial publishers that way. This week we consider whether (with apologies to Stephen Sondheim): We ain’t no delinquents, We’re misunderstood. Deep down inside us there is good! The same topics come up again and again in this newsletter: open access equity, research integrity, peer review, reproducibility and so on. This week is no different. There’s a...

Subscribe to newsletter Hello fellow journalologists, This week I attended the ALPSP annual conference and met some Journalology readers at the event, which was lots of fun. The quality of the presentations and panel discussions was excellent; the organising committee and wider ALPSP team did a fantastic job. I’m still reflecting on what I learned, in particular what the future might hold for small society publishers in an open access world where scale wins. The star of the show was...