Guidelines for Journal Article Reviewing

The peer review process is an invaluable tool for quality control in our research literature. Without it our journals would contain articles of much lower quality and our science would be progressing at a much slower rate. We need the insightful comments of colleagues on our work, and we need a system that allows only quality work to be published. These guidelines are intended to provide assistance to reviewers in helping them provide useful feedback to authors and editors.

There are two interrelated purposes of peer review. The first is to help the editor make a decision about a manuscript, and the second is to provide feedback to authors for improving the current and possibly future work. Editorial decisions are based on the contribution of the paper to the literature, and the quality of the work. A good review should address both issues. Often a flawed paper can report results of research that is an important addition, and should be published after the problems are fixed. Other times papers make little contribution, but have little or nothing wrong with them. There is an important distinction between contribution or "value added" and presentation. Too often authors fix all the presentation issues only to have a paper rejected because the contribution hasn’t changed.

Most journals ask reviewers to provide a review that will be shared with the author/s, and to provide private comments to the editor. Included in private comments is often a discussion of the contribution or potential contribution of a paper. Many journals ask for specific recommendations, e.g., "accept", accept with minor revisions", "reject but invite resubmission", and "reject outright". The recommendation, supported with the comments in the review, help the editor reach a decision.

The purpose of the review is to provide feedback about the strengths and weaknesses of a manuscript. The reviewer should focus on major issues that are important in evaluating the manuscript, as well as things that are done incorrectly. The review is not a statement of how the reviewer would have done the study, or every possible issue including those that are true of every article in the literature. Authors should not have to report results of every possible analysis, cite every tangentially relevant study, control for every variable that could have been measured, and interpret results from every existing theoretical perspective. The goal should be clear scientific communication not literary perfection. A paper should be free of flaws in analysis, method, reasoning, and theory, and it should communicate all of these things clearly.

Remember reviewers are constructive critics, not co-authors. Often well meaning reviewers insist authors spend considerable effort working and reworking their papers in an effort to perfect them—according to the reviewer’s subjective perspective. By the time this process is finished, the reviewer is really a hidden co-author or what Arthur Bedeian calls a "ghostwriter". Keep in mind that study after study of the review process has shown that reviewers don’t agree with one another, and what one insists is a better way of doing something another will insist is not.

1. Don’t confuse quantity with quality. A good review outlines the major strengths and weaknesses and explains flaws. It may give some advice for improvements, perhaps by clarifying things that need clarification. In most cases a page or two is sufficient, although some reviewers can be longwinded, and might spend several pages just covering a few modest points.

2. Avoid attributional remarks about authors. Comments should be about the paper and not the author. Don’t assume that a failure to state something clearly reflects on the author’s abilities. There is no reason to insult an author.

NO: "The author doesn’t understand the basic idea of this theory."

"The author is not thinking clearly on this issue."

YES: "The basic idea of this theory is not clearly described."

"The logic in the explanation on page__ is flawed."

3. Avoid rhetorical questions that involve things that can’t be changed and may not be critical. Only raise issues that are important and not as questions.

NO: "Why did you collect all data from a single organization rather than several?"

"Why didn’t you use the ABCD rather than the EFGH to assess the constructs?"

YES: "The nature of the question asked cannot be addressed with data from a single

organization because..."

"The EFGH is not a valid measure of the constructs of interest because..."

In both cases the issue should only be raised if there is a question about the appropriateness of what was done. Rhetorical questions don’t make clear whether or not what was done was appropriate and they fail to provide clear guidance about what does or does not need to be done.

4. Don’t insist that authors should have used a methodology that they could not have known about when the data were collected. I’ve seen reviewers complain that a new scale should have been used, but that scale wasn’t published until after data collection was done, e.g., the paper says the data were collected in 1995 and the new scale wasn’t developed until 1996.

NO: "The new ABCD scale should have been used."

YES: "The authors might consider using the new ABCD scale in their future work in this

area"

5. Authors should be free to choose their own methodological and theoretical approaches as long as they are appropriate. Too many reviewers have a linear view of scientific progress and assume that the latest article on a topic is the latest word and represents the next step in a progression toward ultimate truth. In reality science doesn’t progress linearly, but advances through a multitude of approaches. A healthy science allows researchers to approach problems from a variety of positions and use diverse methods. Reviewers should resist the temptation to force other’s research into their own frameworks. Not every personality paper must (or should) use the five factor (Big Five) model, and not all factor analysis must be confirmatory using structural equation modeling software. Both of these are controversial, but extremely popular approaches, which I've seen crammed down authors’ throats.

6. Be clear in distinguishing what is a friendly suggestion from what is necessary (see also point 4).

NO: "Smith (1995) was not cited."

YES: "The authors might find Smith (1995) interesting."

7. Be a little tolerant about less than perfect writing. The goal should be clear communication. With a growing number of international scholars, this is likely to become more common place. The major role of the reviewer is to evaluate scientific merits rather than grammar. Specific copy editing comments are far more helpful than complaints that a paper is poorly written.

8. Provide a dispassionate and objective review. If you find the paper annoys or angers you (perhaps by criticizing your own work), you should probably decline doing the review, and at very least, you should let the editor know that you might not be completely objective.