The Future (?) of Peer Review
by Thomas von Foerster
From the beginnings of modern science or "natural philosophy," in
the last part of the 17th century, and through the early part of the 19th
century, the give and take of science, the discussion of experiments and
results, the construction and critique of theories, took place at meetings of
learned societies at which the members heard reports of each others’ work and
read (aloud) letters from their corresponding members. The scientific
literature of the period then consisted of the books or pamphlets published by
individuals concerning their investigations together with the printed reports
of the meetings of the learned societies. These published minutes generally
contained not only the report of a paper or letter read at the meeting, but
also a summary of the discussion that followed it. These societies (such as the
Royal Societies of London, and Edinburgh, and the scientific Academies of
Paris, Berlin, and St. Petersburg) restricted their membership to Fellows
(local and corresponding) of known reputation and stature. Indeed, being
elected to one of these bodies came to be a mark of high scientific
distinction; for a few academies the anointing of members, deriving honor from
honoring the excellent, has become their major function.
However, with increased specialization—the division of natural
philosophy into chemistry, physics, mathematics, biology, and the rest—open
discussions with local savants interested in other matters became less and less
useful, while interchanges with other specialists, some of them remote, became
more important. Meetings thus became less useful than correspondence and
publication. As a result, around the beginning of the 19th century, individual
scientists began publishing scientific journals of their own, independent of
the old learned societies. Thus, for example, L. von Crell began a Chemisches
Journal in 1781, A.L. Crelle began his journal of mathematical
researches in 1826, and J. Liebig started the Annalen der Chemie (originally
as Annalen der Pharmacie) in 1836. In these journals the editors
selected or commissioned the articles to be printed, only sometimes turning to
colleagues or associate editors for advice. The journals thus reflected their
editors’ interests and preferences and often carried the editor’s name as part
of their title, as indeed some, such as Crelles Journal, still do.
It was only with the great increase in specialization of the
sciences around the end of the 19th century and the rise of publications of
specialized societies (such as the American Physical Society or American
Chemical Society) that journal editors began to rely almost entirely on reviews
by others knowledgeable in the particular area of a submitted paper. Even so it
is only in the last few decades that such "anonymous peer review" has come to
dominate scientific publication. The technique is so dominant, in fact, that it
has come to be used almost universally in science whenever assessments of
quality bare to be made (evaluating grant proposals, promotions to tenure, and
so forth).
I suspect that it is the bureaucratization of science, particularly
the indirect effects of the funding of scientific research by governments, that
has provided much of the impetus. The bureaucratic imperative is always to
diffuse responsibility and to make decisions appear "objective" and based on
consensus. Thus, program officers relying on their own judgment of what might
yield interesting science (which some agencies maintained until the 1960s) have
been replaced with panels that rate proposals for funds according to complex
multidimensional scales that are then reduced with specious precision to five-
or six-digit ratings that determine the success of an application. Editors of
important scientific journals, not dealing with public funds, need not go to
such extremes, but they too are under the same pressures (especially if the
journal is published by a society for its members) to be "objective" and
"democratic."
There are some exceptions; a recent example is a new journal, Materials
Research Innovations, started by Rustum Roy, who already has a
reputation as a maverick, and whose negative opinion of anonymous peer review
is well known. The new journal calls itself "The first Journal using super peer
review ...A radically new process of peer-reviewing based on each author’s past
record, ... avoiding the long delays and bias against the new inherent in
traditional peer review."
In the traditional scheme of peer review, editors receive articles,
send them out to one or several technical referees, and base their decision to
publish or not on the reviewers’ reports. While the reports are generally sent
on to the authors, the names of the referees are not. (A few journals have
tried to hide the names of authors from the reviewers, but this has rarely been
successful—the subject and style often give good clues to the author’s
identity.) Anonymity is supposed to maintain the objectivity of the referees
and the integrity of the process, but it can also lead to abuses (and has
indeed led to some recent lawsuits). There are a few journals (more in the
social sciences) that do not use anonymous referees, and in one or two of
these, the referees’ reports are printed along with the article if there are
strong differences of opinion on the merit or contents of the article. In the
physical sciences, however, anonymous peer review is the standard procedure
used to ensure the quality and integrity of the research literature.
The primary function of these reviews is quality control: to
determine what is included in the table of contents of a particular journal.
But they do serve a secondary purpose as well, namely, to provide constructive
criticism for the author. In the best cases, the comments lead to improvements
not only in the presentation of the ideas, but also in the arguments or data
themselves. To name one recent example, the discovery of lacunae in Wiles’s
proof of Fermat’s last theorem required another massive investigation to
correct. That such reviews are not common can be concluded, for example, from
the small number of papers in which the author thanks a reviewer for
substantive comments.
Electronic "Enhancements"
Electronics—chiefly the Internet, but also the enormous reduction
in the costs of telephone links and the enormous increase of computer power—has
is affecting how scientists communicate their results to their colleagues.
Important information is communicated by telephone, e-mail, and other means
that involve no paper, let alone scientific journals. Beginning with
high-energy physics, but spreading rapidly to other disciplines, preprint
servers are more and more becoming the primary means of distributing recent
results to one’s colleagues.
The Internet has of course also changed the paper flow associated
with peer review drastically, in that much of the correspondence about articles
to be published takes place via e-mail. In many cases the articles themselves
are also sent to reviewers in electronic form. Although the reviewing process
is less demanding on the presentation (format, language) of an article than
publication, there are still some problems in using only electronic manuscripts
for review because of difficulties in reading on one machine the files prepared
on another. As "universal" readers (such as Techexplorer or Adobe Acrobat)
improve, these difficulties will certainly decrease, and the purely electronic
procedures will become more and more widely adopted. This is, however, a only a
change in the procedures, and it is unlikely, in the absence of other changes,
to change the relationship of editor, author, referee, and reader. On the other
hand, these procedural changes are easy to implement and to improve
incrementally (that is, they are adiabatic changes—in Eherenfest’s, not the
thermodynamic, sense of adiabatic), so most journals in the physical sciences
will probably be developing them in the near future. The journal editors with
whom I have discussed this concur in that view.
Because e-mail is not as secure as paper (it is easy to alienate an
address, for example, to make a message seem to come from someone other than
the author), editors who use electronic means to send out papers and obtain
reviews must take some precautions. An editor can, for example, give reviewers
passwords by telephone or fax both to access a read-only version of a submitted
article and to authenticate the reviewers’ comments. As papers submitted to a
journal become more technologically demanding, some reviewers may find they do
not have the means to review all aspects of a paper, and editors will have to
take such limitations into account. Of course, as scientific literature becomes
more dependent on computer power, not only reviewers and editors but also
scientists will find that technological developments limit access to parts of
the scientific literature to those with the funds to acquire the necessary
technology.
One aspect of the standard procedure of peer review, however, will
not be altered by the Internet: The number of publications in scientific
journals has been growing at a much faster rate than the number of scientists
and, consequently, orders of magnitude faster than even the most active
editor’s Rolodex. Each reviewer thus is being asked to referee an
ever-increasing number of papers, to the frustration of many reviewers and
occasionally to the detriment of the quality of the reviews. Electronic
publishing by itself does nothing to alter that relationship; it is embedded
into the sociology of publishing and reviewing.
The coexistence of electronic and traditional means of
communicating scientific information can lead to occasional anomalies. Thus,
for example, one can easily imagine a scenario in which a referee receives from
a journal editor an article that he has already seen on a preprint server, and
to which he has already posted a comment on the article; it would be most
natural to use the same comment in the review for the journal, thereby not only
voiding the anonymity of the review but also raising the question, what has the
journal added to the article that the open forum has not already provided?
For some disciplines, such as high-energy physics, much of the
discussion of scientific results already takes place via the preprint servers.
In these fields, computers have taken over one of the functions of the
traditional journal: to serve as a forum for discussion of new ideas.
Interesting preprints spawn comments from workers in the area, the original
papers are improved and new versions posted, other workers are stimulated to do
experiments or suggest new ideas, and the field progresses without a single
tree being felled.
It seems highly likely that such electronic forums will become the
norm of the future. Once a critical fraction of the leading researchers in a
field make use of a preprint server, no other worker in the field can afford to
ignore it, and the Matthew principle will take over. The way in which such a
forum works seems more akin to the 17th and early 18th century discussions of
the learned societies than to those in the learned journals of the 19th and
20th centuries. Within the forum, linked papers and commentary replace the
meetings and reports of the society, and the journals’ peer review is replaced
by dialogue. The discussion itself becomes the review.
Such a system works well only if the group that interacts in such a
forum is sufficiently cohesive. The members must know that for the most part
they can trust each other’s judgment—otherwise the posted material is not worth
reading. This may be part of the reason why preprint servers find ready
acceptance in some fields such as astronomy or high-energy physics but not in
other, larger, fields, such as condensed-matter physics. To preserve the needed
social cohesion, the forums of the future may find they have to limit access to
their discussions, just as the old learned societies restricted their
membership to an elite (usually) of learned members. There are of course many
ways to restrict access to such a forum; one can, for example limit it to
people who:
-
have learned an arcane jargon (K-theory, geologic stratigraphy, Latin);
-
have or do not have specified domain names in their e-mail addresses;
-
have been elected by the existing participants;
-
have paid dues (money or scientific contributions) to join; or
-
have access to a restricted domain on the Internet (such as the Internet 2).
Limitations on access to the papers and discussions in an
electronic forum, however useful they may be in keeping out crackpots and
others who have no business in the field, do come up against a fundamental
value in science: as John Ziman pointed out, science is public knowledge. It is
precisely the openness of scientific discourse that has allowed modern science
to progress. Errors are corrected (some sooner than others) and new ideas are
propagated by open discussion. It is only when discussion is restricted that
wrong ideas such as Lysenkoism can thrive. While workers in fields such as
reaction kinetics or graph theory find it easy to separate useful papers from
useless or crackpot ideas simply from the structure of the argument and the use
of jargon, the separation of wheat from chaff is more difficult in other
fields. Alan Sokal’s hoax in Social Text — whatever else one may think
of it — is a recent example of an interloper penetrating a field’s barriers.
The conflict between openness and restriction is of course not new to the
electronic medium (as Sokal’s example shows), but the explosion of the Internet
into the general public makes the conflict more prominent and more difficult to
resolve.
Because different fields have different social structures and
different ways of validating ideas, the electronic forums will also have
different ways of operating, but the payoff in terms of speed of communication
and ease of interaction seems so large that most disciplines in the natural
sciences are likely at some point to start using a preprint server.
What of the Future?
Would there be a role for something analogous to a journal (and
thus for a publisher) in such a future? I think there is. A truly open,
entirely electronic literature is too unwieldy and too ephemeral. One wants
material that has been graded for quality (even TV guides rate films, giving
stars for those worth watching), and one wants the really good stuff readily
accessible for a long time. Those interested in the development of an idea
(historians of science, prize committees) also want a trail of who said what
when.
There is therefore, even in the electronic future, a place for a
repository for scientific papers of certified quality that has a long lifetime,
stable contents, and a known means of access. That is, there is a place for,
more or less, what one now calls a journal.
While a preprint server provides ready access, and possibly also a
long life, for a paper, it will, by its nature, not provide the certification
of quality. A journal, even an electronic journal, must thus be a separate
entity, though it could of course refer to the papers on the preprint server
for its contents, assuming that the preprint server can guarantee long-term
accessibility for the contents. (Print on paper remains readable and
understandable for centuries, and in favorable cases for millennia, so a purely
electronic archive has to meet a high standard of longevity and accessibility).
Quality control in a journal based on electronic papers may be
provided by some form of review and certification, for which some person or
group (an "editor" or "editorial board") of known stature and discernment would
take responsibility. The certification may be provided directly by one of the
editors (as in Liebig’s or Roy’s journals) or by anonymous referees to whom the
editor sends the paper (as in most current scientific journals).
The long life and ready access would be provided by some entity
that one hopes has a long life, or at least a well-thought out plan of
succession. This entity (more or less what one could call a "publisher") could
be a university, a scientific society, a traditional publisher (profit-making
or non-profit), or a library. The publisher would make the article available
electronically and would maintain it on their servers (or some practical
equivalent thereof) forever. Libraries could maintain mirrors of the
publishers’ material for a licensing fee. This would add some redundancy to the
archive, as well as providing readier access for some users; it could also lead
to problems in verifying and authenticating what is the "true" article.
While publishers at present do not directly guarantee the quality
of the traditionally printed literature — the editorial boards do that — they
do provide the administrative apparatus that makes the control possible. The
electronic model does not change that. Publishers also have not been guarantors
of the longevity of the literature, but for the electronic media there may be
incentives for publishers to assume that function. Providing the electronic
versions of articles on their servers and restricting access to paying
customers may be a good way to preserve their investments in the production of
the literature. In addition, by providing efficient search engines for the data
stored in their own archives and tailored to the research communities they
serve, publishers can add considerable value to the literature they control.
The reviewing and certification of articles for such an electronic
journal could take any of several forms that can be grouped into two
categories: transmission-based forms, such as the electronic version of the
traditional peer review, with editors accepting or rejecting manuscripts
submitted to them, based on their own ideas or on comments from referees they
select; or collection-based forms, in which editors or other readers
periodically browse through articles and associated comments posted in any
appropriate preprint archive, certifying specific articles for their "journal"
or personal collection as they see fit. In either case, the editor adds value
to the accepted article by certifying its interest and merit. The latter form
is already to some extent in use: most WWW home pages for individual
researchers or research groups have a list of relevant publications, web sites
and the like. In a sense, even the simplest of these home pages already
provides a primitive form of electronic journal.
Which form most of the reviewing and certification will take will
probably depend on social factors more than technology, on what researchers,
including journal editors, like to use: this will vary from discipline to
discipline, since, as we know, each has its own sociology. I suspect, however,
that the "collection" forms will seem more "natural" to electronic
communication, and that we may see more of these arising n the future.
More than this I would hesitate to predict.
Back to the Table of Contents
|