Breaking: Peer Review Is Broken!

The subhead of The Pandemic Claims New Victims: Prestigious Medical Journals by Roni Caryn Rabin reads:

Two major study retractions in one month have left researchers wondering if the peer review process is broken.

Below the fold I explain that the researchers who are only now "wondering if the peer review process is broken" must have been asleep for more than the last decade.

"More Is Not Better" Revisited

Source

I have written many times on the topic of scholarly communication since the very first post to this blog thirteen years ago. The Economist's "Graphic Detail" column this week is entitled How to spot dodgy academic journals. It is about the continuing corruption of the system of academic communication, and features this scary graph. It shows:

• Rapid but roughly linear growth in the number of "reliable" journals launched each year. About three times as many were launched in 2018 as in 1978.
• Explosive growth since 2010 in the number of "predatory" journals launched each year. In 2018 almost half of all journals launched were predatory.

Below the fold, some commentary.

Pre-publication Peer Review Subtracts Value

Pre-publication peer review is intended to perform two functions; to prevent bad science being published (gatekeeping), and to improve the science that is published (enhancement). Over the years I've written quite often about how the system is no longer "fit for purpose". Its time for another episode draw attention to two not-so recent contributions:

• Nearly two years ago Martin Klein and co-authors posted Comparing Published Scientific Journal Articles to Their Pre-print Versions to arXiv.org. They had presented the work at the Fall 2015 CNI. I really thought I had written about it at the time, but apparently it slipped through the cracks. I was reminded about it by Glyn Moody, who wrote on the occasion of its "official publication" two weeks ago in the International Journal of Digital Libraries. They show that pre-publication review's effect on article texts is barely detectable.
• Last October Sir Timothy Gowers, Rouse Ball Professor of Mathematics at Cambridge, had an important article in the Times Literary Supplement entitled The end of an error?, describing how the switch from pre- to post-publication review has improved mathematics research.

Below the fold, the details.

Crowd-sourced Peer Review

At Ars Technica, Chris Lee's Journal tries crowdsourcing peer reviews, sees excellent results takes off from a column at Nature by a journal editor, Benjamin List, entitled Crowd-based peer review can be good and fast. List and his assistant Denis Höfler have come up with a pre-publication peer-review process that, while retaining what they see as its advantages, has some of the attributes of post-publication review as practiced, for example, by Faculty of 1000. See also here. Below the fold, some commentary.

What is wrong with science?

This is a quick post to flag two articles well worth reading.

Glyn Moody on Open Access

At Ars Technica, Glyn Moody writes Open access: All human knowledge is there—so why can’t everybody access it? , a long (9 "page") piece examining this question:

What's stopping us? That's the central question that the "open access" movement has been asking, and trying to answer, for the last two decades. Although tremendous progress has been made, with more knowledge freely available now than ever before, there are signs that open access is at a critical point in its development, which could determine whether it will ever succeed

It is a really impressive, accurate, detailed and well-linked history of how we got into the mess we're in, and a must-read despite the length. Below the fold, a couple of comments.

Signal or Noise?

I've been blogging critically about the state of scientific publishing since my very first post 9 years ago. In particular, I've been pointing out that the several billion dollars a year that go to the publisher's bottom lines, plus the several billion dollars a year in unpaid work by the reviewers, is extremely poor value for money. The claim is that the peer-review process guarantees the quality of published science. But the reality is that it doesn't; it cannot even detect most fraud or major errors.

The fundamental problem is that all participants have bad incentives. Follow me below the fold for some recent examples that illustrate their corrupting effects.

More Is Not Better

Hugh Pickens at /. points me to Attention decay in science, providing yet more evidence that the way the journal publishers have abdicated their role as gatekeepers is causing problems for science. The abstract claims:

The exponential growth in the number of scientific papers makes it increasingly difficult for researchers to keep track of all the publications relevant to their work. Consequently, the attention that can be devoted to individual papers, measured by their citation counts, is bound to decay rapidly. ... The decay is ... becoming faster over the years, signaling that nowadays papers are forgotten more quickly. However, when time is counted in terms of the number of published papers, the rate of decay of citations is fairly independent of the period considered. This indicates that the attention of scholars depends on the number of published items, and not on real time.

Below the fold, some thoughts.

Special Issue of Science

I'd like to draw attention to the special issue of Science on scientific communication, which the AAAS has made freely available. It is all worth reading but these pieces caught my eye:

• Tania Rabesandratana's profile of Vitek Tracz, highlighting his roles in open access publishing via BioMed Central and Faculty of 1000, and now in post-publication peer review via F1000 Research. Kudos to Science for this article on a competitor!
• Randall Munroe's infographic.
• Diane Harley's corrective to the Web 2.0 hype, stressing that change will come only as institutions change their tenure and promotion practices to reduce their emphasis on flawed measures such as impact factor.

The Big Deal

Andrew Odlyzko has a fascinating paper with a rather long title, Open Access, library and publisher competition, and the evolution of general commerce (PDF). He describes how the relationship between the libraries and the publishers in the market for academic journals has evolved to transfer resources from libraries to the publishers, and how a similar strategy might play out in many, more general markets. Below the fold I discuss some of the details, but you should read the whole thing.

The value that publishers add

Here is Paul Krugman pointing out how much better econoblogs are doing at connecting economics and policy than traditional publishing. He brings out several of the points I've been making since the start of this blog six years ago.

First, speed: 

The overall effect is that we’re having a conversation in which issues get hashed over with a cycle time of months or even weeks, not the years characteristic of conventional academic discourse.

Second, the corruption of the reviewing process:

In reality, while many referees do their best, many others have pet peeves and ideological biases that at best greatly delay the publication of important work and at worst make it almost impossible to publish in a refereed journal. ... anything bearing on the business cycle that has even a vaguely Keynesian feel can be counted on to encounter a very hostile reception; this creates some big problems of relevance for proper journal publication under current circumstances.

Third, reproducibility:

Look at one important recent case ... Alesina/Ardagna on expansionary austerity. Now, as it happens the original A/A paper was circulated through relatively “proper” channels: released as an NBER working paper, then published in a conference volume, which means that it was at least lightly refereed. ... And how did we find out that it was all wrong? First through critiques posted at the Roosevelt Institute, then through detailed analysis of cases by the IMF. The wonkosphere was a much better, much more reliable source of knowledge than the proper academic literature.

And here's yet another otherwise good review of the problems of scientific publishing that accepts Elsevier's claims as to the value they add, failing to point out the peer reviewed research into peer review that conclusively refutes these claims. It does, however include a rather nice piece of analysis from Deutsche Bank:

We believe the [Elsevier] adds relatively little value to the publishing process.  We are not attempting to dismiss what 7,000 people at [Elsevier] do for a living.  We are simply observing that if the process really were as complex, costly and value-added as the publishers protest that it is, 40% margins wouldn’t be available.

As I pointed out using 2010 numbers:

The world's research and education budgets pay [Elsevier, Springer & Wiley] about $3.2B/yr for management, editorial and distribution services. Over and above that, the worlds research and education budgets pay the shareholders of these three companies almost $1.5B for the privilege of reading the results of research (and writing and reviewing) that these budgets already paid for.

What this $4.7B/yr pays for is a system which encourages, and is riddled with, error and malfeasance. If these value-subtracted aspects were taken into account, it would be obvious that the self-interested claims of the publishers as to the value that they add were spurious.

Journals Considered Harmful

Via Yves Smith and mathbabe I found Deep Impact: Unintended consequences of journal rank by Björn Brembs and Marcus Munaf, which is a detailed analysis of the arguments I put forward at the Dagstuhl workshop on the Future of Research Communication and elsewhere.

The authors draw the following conclusions:

The current empirical literature on the effects of journal rank provides evidence supporting the following four conclusions: 1) Journal rank is a weak to moderate predictor of scientific impact; 2) Journal rank is a moderate to strong predictor of both intentional and unintentional scientific unreliability; 3) Journal rank is expensive, delays science and frustrates researchers; and, 4) Journal rank as established by [Impact Factor] violates even the most basic scientific standards, but predicts subjective judgments of journal quality.

Even if you disagree with their conclusions, their extensive bibliography is a valuable resource. Below the fold I discuss selected quotes from the paper.

DNA as a storage medium

I blogged last October about a paper from Harvard in Science describing using DNA as a digital storage medium. In a fascinating keynote at IDCC2013 Ewan Birney of EMBL discussed a paper in Nature with a much more comprehensive look at this technology. It has been getting a lot of press, much of it as usual somewhat misleading. Below the fold I delve into the details.

The 5 Stars of Online Journal Articles

David Shotton, another participant in last summer's Dagstuhl workshop on Future of Research Communications, has an important article in D-Lib entitledThe Five Stars of Online Journal Articles — a Framework for Article Evaluation. By analogy with Tim Berners-Lee's Five Stars of Linked Open Data, David suggests assessing online articles against five criteria:

• peer review
• open access
• enriched content
• available datasets
• machine-readable metadata

For each criterion, he provides a five-point scale. For example, the open access scale goes from 0 for no open access to 4 for Creative Commons licensing. The full article is well worth a read, especially for David's careful explanation of the impacts of each point on the scale of each criterion on the usefulness of the content.

The article concludes by applying the evaluation to a number of articles (including itself). In this spirit, here is my evaluation of our SOSP '03 paper:

• peer review: 2 - Responsive peer review
• open access: 1 - Self-archiving green/gratis open access
• enriched content: 1 - Active Web links
• available datasets: 1 - Supplementary information files available
• machine-readable metadata: 1- Structural markup available

What Problems Does Open Access Solve?

The library at the University of British Columbia invited me to speak during their Open Access Week event. Thinking about the discussions at the recent Dagstuhl workshop I thought it would be appropriate to review the problems with research communication and ask to what extent open access can help solve them. Below the fold is an edited text of the talk with links to the sources.

A Level Playing-Field For Publishers

Stuart Shieber has an interesting paper in PLoS Biology on the economics of open-access publishing. He observes the moral hazard implicit in the separation between the readers of peer-reviewed science and the libraries that pay the subscriptions to the publishers that make the peer review possible. His proposal to deal with this is that grant funders and institutions should make dedicated funds available to authors that can be used only for paying processing fees for open access journals. After all, he observes, these funders already support the subscriptions that allow subscription journals not to charge processing fees (although some still do charge such fees). His proposal would provide a more level playing field between subscription and open access publishing channels. Below the fold is my take on how we can measure the level-ness of this field.

JCDL 2010 Keynote

On June 23 I gave a keynote address entitled to the joint JCDL/IACDL 2010 conference at Surfer's Paradise in Queensland, Australia. Below the fold is an edited text of the talk, with links to the resources.

Persistence of Poor Peer Reviewing

Another thing I've been doing during the hiatus is serving as a judge for Elsevier's Grand Challenge. Anita de Waard and her colleagues at Elsevier's research labs set up this competition with a substantial prize for the best demonstration of what could be done to improve science and scientific communication given unfettered access to Elsevier's vast database of publications. I think the reason I'm on the panel of judges is that after Anita's talk at the Spring CNI she and I had an interesting discussion. The talk described her team's work to extract information from full-text articles to help authors and readers. I asked who was building tools to help reviewers and make their reviews better. This is a bete noir of mine, both because I find doing reviews really hard work, and because I think the quality of reviews (including mine) is really poor. For an ironic example of the problem, follow me below the fold.

Mass-market scholarly communication

I attended the Workshop on Repositories sponsored by the NSF (US) and the JISC (UK). I apologize in advance for the length of this post, which is a follow-up. As I wrote it new aspects kept emerging and more memories of the discussion came back.

In his perceptive position paper for the workshop, Don Waters cites a fascinating paper by Harley et al. entitled "The Influence of Academic Values on Scholarly Publication and Communication Practices". I'd like to focus on two aspects of the Harley et al paper:

• They describe a split between "in-process" communication which is rapid, flexible, innovative and informal, and "archival" communication. The former is more important in establishing standing in a field, where the latter is more important in establishing standing in an institution.
• They suggest that "the quality of peer review may be declining" with "a growing tendency to rely on secondary measures", "difficult[y] for reviewers in standard fields to judge submissions from compound disciplines", "difficulty in finding reviewers who are qualified, neutral and objective in a fairly closed acacdemic community", "increasing reliance ... placed on the prestige of publication rather than ... actual content", and that "the proliferation of journals has resulted in the possibility of getting almost anything published somewhere" thus diluting "peer-reviewed" as a brand.
  

In retrospect, I believe Malcolm Read made the most important observation of the workshop when he warned about the coming generational change in the scholarly community, to a generation which has never known a world without Web-based research and collaboration tools. These warnings are particularly important because of the inevitable time lags in developing and deploying any results from the policy changes that the workshop's report might advocate.

Late in the workshop I channeled my step-daughter, who is now a Ph.D. student. Although I was trying to use her attitudes to illuminate the coming changes, in fact she is already too old to be greatly impacted by any results from the workshop. She was in high school as the Web was exploding. The target generation is now in high school, and their equivalent experience includes blogs and MySpace.

I'd like to try to connect these aspects to Malcolm's warnings and to the points I was trying to communicate by channeling my step-daughter. In my presentation I used as an example of "Web 2.0 scholarship" a post by Stuart Staniford, a computer scientist, to The Oil Drum blog, a forum for discussion of "peak oil" among a diverse group of industry professionals and interested outsiders, like Stuart. See comments and a follow-on post for involvement of industry insiders.

I now realize that I missed my own basic point, which is:

Blogs are bringing the tools of scholarly communication to the mass market, and with the leverage the mass market gives the technology, may well overwhelm the traditional forms.

Why is it that Stuart feels 2-3 times as productive doing "blog-science"? Based on my blog experience of reading (a lot) and writing (a little) I conjecture as follows:

• The process is much faster. A few hours to a few days to create a post, then a few hours of intensive review, then a day or two in which the importance of the reviewed work becomes evident as other blogs link to it. Stuart's comment came 9 hours into a process that accumulated 217 comments in 30 hours. Contrast this with the ponderous pace of traditional academic communication.
• The process is much more transparent. The entire history of the review is visible to everyone, in a citable and searchable form. Contrast this with the confidentiality-laden process of traditional scholarship.
• Priority is obvious. All contributions are time-stamped, so disputes can be resolved objectively and quickly. They're less likely to fester and give rise to suspicions that confidentiality has been violated.
• The process is meritocratic. Participation is open to all, not restricted to those chosen by mysterious processes that hide agendas. Participants may or may not be pseudonymous but their credibility is based on the visible record. Participants put their reputation on the line every time they post. The credibility of the whole blog depends on the credibility and frequency of other blogs linking to it - in other words the same measures applied to traditional journals, but in real time with transparency.
• Equally, the process is error-tolerant. Staniford says "recognition on all our parts that this kind of work will have more errors in any given piece of writing, and its the collaborative debate process that converges towards the truth." This tolerance is possible because the investment in each step is small, and corrections can be made quickly. Because the penalty for error is lower, participants can afford to take more creative risk.
• The process is both cooperative and competitive. Everyone is striving to improve their reputation by contributing. Of course, some contributions are negative, but the blog platforms and norms are evolving to cope with this inevitable downside of openness.
• Review can be both broad and deep. Staniford says "The ability for anyone in the world, with who knows what skill set and knowledge base, to suddenly show up ... is just an amazing thing". And the review is about the written text, not about the formal credentials of the reviewers.
  
• Good reviewing is visibly rewarded. Participants make their reputations not just by posting, but by commenting on posts. Its as easy to assess the quality of a participant reviews as to assess their authorship; both are visible in the public record.

Returning to the Harley et al. paper's observations, it is a commonplace that loyalty to employers is decreasing, with people expecting to move jobs frequently and often involuntarily. Investing in your own skills and success makes more sense than investing in the success of your (temporary) employer. Why would we be surprised that junior faculty and researchers are reluctant to put effort into institutional repositories for no visible benefit except to the institution? More generally, it is likely that as the mechanisms for establishing standing in the field diverge from those for establishing standing in the institution, investment will focus on standing in the field as being more portable, and more likely to be convertible into standing in their next host institution.

It is also very striking how many of the problems of scholarly communication are addressed by Staniford's blog-science:

• "the proliferation of journals has resulted in the possibility of getting almost anything published somewhere" - If scholarship is effectively self-published then attention focusses on tools for rating the quality of scholarship, which can be done transparently, rather than tools for preventing low-rated scholarship being published under the "peer-reviewed" brand. As the dam holding back the flood of junk leaks, the brand looses value, so investing in protecting it becomes less rewarding. Tools for rating scholarship, on the other hand, reward investment. They will be applied to both branded and non-branded material (cf. Google), and will thus expose the decreased value of the brand, leading to a virtuous circle.
• "increasing reliance ... placed on the prestige of publication rather than ... actual content" - Blog-style self-publishing redirects prestige from the channel to the author. Clearly, a post to a high-traffic blog such as Daily Kos (500,000 visits/day) can attract more attention, but this effect is lessened by the fact that it will compete with all the other posts to the site. In the end the citation index effect works, and quickly.
  
• "a growing tendency to rely on secondary measures" - If the primary measures of quality were credible, this wouldn't happen. The lack of transparency in the traditional process makes it difficult to regain credibility. The quality rating system for blogs is far from perfect, but it is transparent, it is amenable to automation, and there is an effective incentive system driving innovation and improvement for the mass market.
  
• "difficult[y] for reviewers in standard fields to judge submissions from compound disciplines" - This is only a problem because the average number of reviewers per item is small, so each needs to span most of the fields. If, as with blogs, there are many reviewers with transparent reputations, the need for an individual reviewer to span fields is much reduced.
  
• "difficulty in finding reviewers who are qualified, neutral and objective in a fairly closed acacdemic community" - This is only a problem because the process is opaque. Outsiders have to trust the reviewers; they cannot monitor their reviews. With a completely transparent, blog-like process it is taken for granted that many reviewers will have axes to grind, the process exists to mediate these conflicting interests in public.
  

Of the advantages I list above, I believe the most important is sheer speed. John Boyd, the influential military strategist, stressed the importance of accelerating the OODA (Observation, Orientation, Decision, Action) loop. Taking small, measurable steps quickly is vastly more productive than taking large steps slowly, especially when the value of the large step takes even longer to become evident.

Why did arXiv arise? It was a reaction to a process so slow as to make work inefficient. Successive young generations lack patience with slow processes; they will work around processes they see as too slow just as the arXiv pioneers did. Note that once arXiv became institutionalized, it ceased to evolve and is now in danger of loosing relevance as newer techologies with the leverage of the mass market overtake it. Scientists no longer really need arXiv; they can post on their personal web sites and Google does everything else (see Peter Suber), which reinforces my case that mass-market tools will predominate. The only mass-market tool missing is preservation of personal websites, which blog platforms increasingly provide. Almost nothing in the workshop was about speeding up the scholarly process, so almost everything we propose will probably get worked around and become irrelevant.

The second most important factor is error tolerance. The key to Silicon Valley's success is the willingness to fail fast, often and in public; the idea that learning from failure is more important than avoiding failure. Comments in the workshop about the need for every report to a funding agency to present a success illustrate the problem. If the funding agencies are incapable of hearing about failures they can't learn much.

What does all this mean for the workshop's influence on the future?

• Unless the institutions' and agencies' efforts are focussed on accelerating the OODA loop in scholarship, they will be ignored and worked-around by a coming generation notorious for its short attention span. No-one would claim that institutional repositories are a tool for accelerating scholarship; thus those workshop participants describing their success as at best "mixed" are on the right track. Clearly, making content at all scales more accessible to scholars and their automated tools is a way to accelerate the process. In this respect Peter Murray-Rust's difficulties in working around restrictions on automated access to content that is nominally on-line are worthy of particular attention.
• Academic institutions and funding agencies lack the resources, expertise and mission to compete head-on with mass market tools. Once the market niche has been captured, academics will use the mass market tools unless the productivity gains from specialized tools are substantial. Until recently, there were no mass-market tools for scholarly communication, but that's no longer true. In this case the mass-market tools are more productive that the specialized ones, not less. Institutions and agencies need to focus on ways to leverage these tools, not to deprecate their use and arm-twist scholars into specialized tools under institutional control.
• Insititutions and agencies need to learn from John Boyd and Silicon Valley themselves. Big changes which will deliver huge value but only in the long term are unlikely to be effective. Small steps that may deliver a small increment in value but will either succeed or fail quickly are the way to go.
• Key to effective change are the incentive and reward systems, since they close the OODA loop. The problem for institutions and agencies in this area is that the mass-market tools have very effective incentive and reward systems, based on measuring and monetizing usage. Pay attention to the way Google runs vast numbers of experiments every day, tweaking their systems slightly and observing the results on user's behavior. Their infrastructure for conducting these experiments is very sophisticated, because the rewards for success flows straight to the bottom line. The most important change institutions and agencies can make is to find ways to leverage the Web's existing reward systems by measuring and rewarding use of scholarly assets. Why does the academic structure regard the vast majority of accesses to the Sloan Digital Sky Survey as being an unintended, uninteresting by-product? Why don't we even know what's motivating these accesses? Why aren't we investing in increasing these accesses?

I tend to be right about the direction things are heading and very wrong about how fast they will get there. With that in mind, here's my prediction for the way future scholars will communicate. The entire process, from lab notebook to final publication, will use the same mass-market blog-like tools that everyone uses for everyday cooperation. Everything will be public, citable, searchable, accessible by automated scholarly tools, time-stamped and immutable. The big problem will not be preservation, because the mass-market blog-like platforms will treat the scholarly information as among the most valuable of their business assets. It will be more credible, and thus more used, and thus generate more income, than less refined content. The big problem will be a more advanced version of the problems currently plaguing blogs, such as spam, abusive behavior, and deliberate subversion. But, again, since the mass-market systems have these problems too, scholars will simply use the mass-market solutions.