Paying for Peer Review?

by Nicole Eva
University of Lethbridge
Alberta, Canada

I got to talking with a newer professor in our Faculty of Management about issues around open access and the huge profits of the big publishers, and found that she was surprisingly passionate about the subject. I say surprisingly, because often I find faculty members, especially in that particular Faculty, somewhat resistant / unconcerned about the issue. But this particular professor is from a younger generation, having recently completed her PhD, and is not yet totally convinced about the traditional scholarly communications models. As we drove home from a trip our book club had taken, we had hours to muse about the state of academic publishing and the unfairness of it all.

Then she said something shocking, at least to me: “Those publishers should be paying us for peer review”. But the more we talked, the more she had me on board; sure, writing the articles is part of our job, but the reviews we are also expected to perform for free? Those are often seen as over and above, certainly not counting nearly as heavily towards one’s tenure & promotion package but often time consuming and, at least for some, undertaken quite painstakingly and conscientiously. I argued that the ‘good’ guys, those truly open access publications which are operating at a break-even level, wouldn’t be able to afford that; but I couldn’t deny that charging the Big Five a hefty fee for at least some of the free labour we provide as academics would be incredibly satisfying. We fleshed out the idea, devising a scheme in which universities would administer the funds on behalf of their researchers; the funds would go towards the cost of research (RAs, equipment, etc) rather than being paid directly to reviewers, and a portion of it would be held back in a central fund to ‘reimburse’ those who were reviewing for non-profit, open access journals. We got ourselves so worked up about the issue that we decided right then and there to write a blog-post/style article about it, and within hours of getting home she’d sent me an outline of the arguments we’d use.

We did write the article; it’s currently being reviewed by one editor and we hope to get confirmation of publication this month. [Editor’s note: the article has been published and can be found at University Affairs.] But as I floated the idea past my colleagues, I was met with several objections, most of which were philosophical. I get it – the idea of monetizing any element of publication seems inherently wrong. But as long as we are stuck in a world in which some people (read: publishers) are getting rich off of the free labour of others (read: academics), shouldn’t we try to balance the tables at least a little bit? Another objection was that publishers would simply raise the prices of their subscriptions to compensate for their increased expenses. We countered this by calling for government intervention on subscription prices, much the way they currently cap the price of pharmaceuticals. We also noted that this harebrained idea would only work if everyone, worldwide, insisted on payment for their reviews – as soon as someone caves and does it for free, the system falls apart.

So do I think it will work? Not really. I’m sure it’s just another of the many, many ideas out there about how to transform academic publishing that won’t pan out. And in the process, it will probably raise the ire of many a librarian. But what I loved about this collaboration was that another academic – not a librarian, and a business faculty member to boot – cared as much about the unfairness of it all as I, a librarian, did. And I loved the idea that we would put this crazy idea* out there and potentially raise the conversation in a wider forum among academics – not just among librarians and other scholarly communication gurus, but among scholars from all disciplines. We intentionally submitted it to general academic, blog-type sites to gain as wide a readership as possible. And if it does nothing more than stir the pot among a larger audience of scholars and make a few of them think twice about feeding the oligopoly, then I will view that as a success.

So what do you think? Will I be outed as a traitor if this thing gets published? Will all librarians think I’m crazy, and shun me at conferences? Or is any idea a good idea if it raises awareness of the broken scholarly communication system?

*To be fair, once we started writing the paper we discovered that others have flirted with paying for peer review, with some success; see for examples.

This article gives the views of the author and not necessarily the views the Centre for Evidence Based Library and Information Practice or the University Library, University of Saskatchewan.

Revising a manuscript: Thoughts on how to organize your response to peer reviewer and editor comments

Lorie Kloda
Editor-in-Chief, Evidence Based Library & Information Practice
Associate University Librarian, Planning & Community Relations, Concordia University

Rebekah (Becky) Willson
Associate Editor (Research Articles), Evidence Based Library & Information Practice
Lecturer, Department of Computer & Information Sciences, University of Strathclyde, Glasgow, United Kingdom

Lisl Zach
Associate Editor (Research Articles), Evidence Based Library & Information Practice
Managing Partner, Informatics Insights, LLC, Philadelphia, PA, USA

Red Pen

CC BY-NC-SA 2.0 (Some rights reserved by cellar_door_films)

The process of revising and resubmitting a manuscript for further review can be a long and sometimes challenging process. As editors for the journal Evidence Based Library and Information Practice, we see a range of responses to requests for revisions or to revise and resubmit manuscripts. Based on our experience, there are things that you as an author can do to help both yourself and your reviewers to ensure a smooth process and to increase the likelihood of having your revised manuscript accepted for publication.

The first is to read the editor’s and reviewers’ comments and suggestions carefully. These comments are aimed at improving your manuscript. Not all of the editor’s and reviewers’ comments may be applicable or relevant to your work, and therefore you may have good reasons for disagreeing with their suggestions. Not all of the changes suggested have to be made, but all of the editor’s and reviewers’ comments do have to be addressed.

One challenging thing for editors and reviewers is the number of submissions we see. The time lag between when a manuscript is originally submitted and the time when it is revised and resubmitted can be many months. To help the editors and reviewers see whether you have addressed the comments made about your manuscript and revised it accordingly, you need to identify the changes you have made (or not made) and explain why.

One way to do this is to submit a revised version of the article with the changes clearly highlighted. Using Word’s Track Changes feature can be a useful tool to do this. By scanning through a manuscript with Track Changes, it is very clear what changes have been made. However, it must also be clear why you have made the changes that you did, how you approached those changes, and why you decided not to make the changes suggested. To keep track of your revisions – which are based on the input of at least two separate reviewers and the editor – a separate document that includes a table can be very useful. (See Rebekah Willson’s template for an example of how to organize this information.)

This table should consist of:

  1. The first column should contain the reviewers’ comments. Take each comment that requires a response (either a revision or an explanation) and paste it into a separate box.
  2. The second column should contain any revisions that you have made. With a brief description of what you’ve done, copy and paste the changes you’ve made to the manuscript as a result of the reviewers’ comments. Include page numbers (and, if helpful, table or figure numbers, paragraphs, or section names), so that it is easy to flip between the revised manuscript and the table of revisions. If the changes are too long to fit into the table, just provide the description and page numbers.
  3. The third column should explain your actions. If you have made revisions, explain how your changes have addressed the reviewer’s comments. If you have not made revisions, explain why you have made that decision.

Filling out the table once the revisions have been made can be very challenging. If you start by going through the editor’s and reviewers’ comments and putting them into the first column of the table, this will help you to identify the work that must be done. Then working between the table and your revised manuscript you can proceed in a step-by-step manner to complete your changes.

When working on a manuscript with multiple authors, a helpful strategy is to create the table and add all the editor’s and reviewers’ comments and suggestions, and then add another column to assign each of these to an author to address. Following this, a meeting to reach consensus on the changes to make and to assign the work can streamline the revision process and ensure the manuscript is resubmitted in a timely fashion.

See another example of how a PhD student organizes her manuscript revisions.

What questions do you have about the peer review and revisions process?

This article gives the views of the author and not necessarily the views the Centre for Evidence Based Library and Information Practice or the University Library, University of Saskatchewan.

Lessons Learned: The Peer Review Process

by Tasha Maddison
Saskatchewan Polytechnic
Maha Kumaran
Leslie and Irene Dubé Health Sciences Library, University of Saskatchewan

We are currently in the final stages of editing a book on distributed learning. We initially received 27 chapter submissions on October 31, 2015 and set up a peer review process shortly thereafter. Each chapter was reviewed by two external reviewers. Our first challenge was to find enough reviewers so that each chapter could be reviewed in a timely manner. We sought reviewers from our immediate network of colleagues and later from acquaintances and individuals that we met at conferences. Finally we had to extend our search and seek out individuals out for that purpose. Once the first few chapters were finalized, we requested assistance from those chapter authors with reviewing chapters, and if they were not available, asked if could they recommended others from their institutions. Reviewers were invited to comment directly on the document and/or provide comments using a template that we provided. It was a learning process for everyone involved; the authors, the reviewers and also the editors. Here is what we have learned thus far in this process:

Reviewers don’t always agree. In cases like these, it is very helpful to have a third opinion and this is where the editors play a critical role. They can ask the following questions and make a decision on the chapter: Does the review seem overly critical, or unjust? Is the reviewer actually providing suggestions help to improve the chapter? Or are they unnecessarily picky? Should the author(s) be given a chance to review and significantly revise their work, or is it feasible to reject it outright?

Lesson Learned: Use your judgement in accepting or intermediating the reviewer’s comments

Reviewers are too nice. There were occasions when reviewers did not make any comments on the document, and/or had only positive comments on the template. Upon reading the same documents, editors had questions or needed clarification.

Lesson Learned: The reviewer’s comments are not the only quintessential element to use towards bettering a chapter.

Reviewers and deadlines. Deadlines don’t mean the same thing to everyone. Some reviewers demonstrate tremendous discipline and always submit their work on time. Others, use deadlines more as a guideline than a hard and fast rule. Editors should count on these potential delays and build in a significant contingency plan for time.

Lesson Learned: Be prepared to be flexible and give reviewers 3 weeks to return their evaluation, but expect at least two weeks lag time for some. Also, build in a time contingency for the entire project.

Reviewers as copyeditors: Reviewers are tempted to take on the role of copyediting when reviewing a text, but the primary job here is to review the content and comment appropriately. The more detailed a reviewers’ suggestions can be, the more helpful it is to the authors, and ultimately, the more successful the final chapter will be. General sweeping statements are not useful. Specific detailed comments are more helpful. If you are a peer reviewer, think of yourself as a most valued intermediary in the process of publishing a chapter. You take the work, and help to elevate it to the next level.

Lesson Learned: Provide reviewers with a template posing specific questions to present their comments and an area where they can include general comments for the editors, which will not be shared with authors.

Rejections after reviewing: Unfortunately rejections are part of the peer review process. It is important that all parties are gracious and respectful if this is the outcome. The reviewers and editors should provide suggestions that strengthen the chapter and have it fit for publication upon revision. The authors should be left feeling that their submission and their participation in the process was worthwhile, and hopefully they too learned a lot.

Lesson Learned: Be prepared to listen to authors’ justifications about their chapter and then make final decisions.

The peer review process, regardless of the fate of the document, should noticeably improve the quality of the final product. Unbiased feedback from experts notes the successes or shortcomings of each chapter’s argument, the validity of results, the flow of the discussion, and the sound foundation of research. All members involved will benefit if they come in with a positive attitude and with a generosity towards accepting criticism.

For more information on the peer review process, check out these recent Brain Work Blog posts:

This article gives the views of the author(s) and not necessarily the views of the Centre for Evidence Based Library and Information Practice or the University Library, University of Saskatchewan.

The author’s side of peer review

by Kristin Hoffmann
Associate Librarian, University of Western Ontario

In the last few months, Brain-Work has featured two discussions of peer review: How to be an effective peer reviewer and Peer reviewing as a foundation of research culture, both aimed at librarians who might be serving as reviewers. In this post, I want to look at peer review from the perspective of the author who is reading and responding to peer reviewers’ feedback.

I get butterflies in my stomach every time I see the subject line in my inbox announcing an email that contains reviewer comments. Reading reviewer feedback feels like the closest I come these days to getting a grade back on a test or an essay, and I still desperately want that A. What I have increasingly come to realize is that reviewers’ feedback isn’t going to determine my final grade in the course, and that it can really be a process of giving supportive and formative feedback.

Here are some suggestions I have that will hopefully make the process of reading and responding to peer review feel less daunting and more supportive:

1. Ask someone else to read your paper before you submit it. It’s always a good idea to get a fresh perspective on your work. Also, getting feedback from someone you know will help prepare you for getting more feedback from the reviewers.

2. When you get the reviewers’ comments, particularly if they include lots of suggestions for revision, let yourself complain and vent about it – for a day. Then put the complaining behind you and move on.

3. Remember that the reviewers’ feedback is intended to improve your paper. Read it with that in mind. In my experience, reviewers have always provided at least one helpful suggestion. (Exception: a review that says simply “this was terrible and shouldn’t ever be published.” That review isn’t going to improve your paper, so go ahead and complain about that terrible review that should never have been written, and then move on.)

4. You don’t necessarily need to take all of the reviewers’ suggestions or address all their questions. The reviewers don’t know your research as well as you do, and it may be that their suggestions would change the focus of your paper beyond what you intended. It could also be that they’re asking for changes because they didn’t clearly understand your intent as you had presented it in the paper—and that should be a sign to you that you need to change something, even if the change is perhaps not exactly what the reviewers asked for.

5. Stay in contact with the editor. Let them know that you are working on changes. If the editor had sent a “revise and resubmit” decision and you’ve decided not to resubmit, let them know that too. Ask the editor for advice if the reviewers’ suggestions aren’t clear, or if the reviewers have provided conflicting suggestions.

For more advice about reading and responding to peer review, the following offer more good suggestions:

Annesley, Thomas M. 2011. “Top 10 Tips for Responding to Reviewer and Editor Comments.” Clinical Chemistry 57 (4): 551–54. doi:10.1373/clinchem.2011.162388.

McKenzie, Francine. 2009. “The Art of Responding to Peer Reviews.” University Affairs.

The Open Source Paleontologist. 2009. “Responding to Peer Review.”

This article gives the views of the author(s) and not necessarily the views of the Centre for Evidence Based Library and Information Practice or the University Library, University of Saskatchewan.

Peer Reviewing as a Foundation of Research Culture (Or, how not to be that peer reviewer)

by Heidi LM Jacobs
Leddy Library, University of Windsor

When you talk to published scholars, everyone has a horror story to share about a bad peer reviewer. Stories can range from the amusing to the incredulous. Over the years, I’ve had some excellent peer reviewers: these are readers who take the time to read my work carefully, see what I’m trying to do, and then find ways to make my article better. My work is always better because of those reviewers and I am grateful for them. But I’ve also had reviewers who lean more toward hurtful or harmful rather than helpful. Here are some of the comments I’ve personally received in the past few years that would fall into that “less than helpful” category:
• “Maybe you could talk to a faculty member on your campus who could teach you how to do research.”
• “Writing down your own thoughts about teaching is not research. You can’t just make statements about what you think. You need to use surveys and data to make claims.”
• Entire review: “Uninteresting. Unpublishable. Reject.”
• “I know very little about this area and don’t work in the area writer is talking about. However, if the writer explored [an entirely different topic] it would be a much more interesting article.”
I should note that all of the responses above are from peer reviews of articles that went on to be published and well-received elsewhere; one even made a “best instruction article of the year” list. As someone who has been sending papers out for over twenty years, I’m able to shrug those awful reviews off, send my work out again, and move on. Increasingly, however, what bothers me most about getting these reviews is imagining what it would be like to be a brand new scholar sending her or his work out to reviewers and receiving feedback like the above. It takes courage and trust to send our work out and when we receive harmful or hurtful feedback, the will and courage to send it out again diminishes considerably.

Our goal as reviewers should be—above all—to be helpful: helpful to the writer by giving him or her concrete strategies to make their work better and helpful to the profession by encouraging, nurturing, and mentoring our peers to publish top-notch work.

By being “helpful,” I want to underscore that I do not mean that reviewers can only say positive things and avoid anything that points out limitations of the work. Not pointing out, for example, that the literature review needs work or that a paper lacks focus might seem “nicer” but is not helpful either to the writer if she or he wants to continue to publish certain kinds of articles or to the profession in whose best interests it is for us to produce strong, rigorous, well-crafted scholarship. That said, we need to be cognizant of the ways in which we give feedback and the impact of that feedback.

In recent months, I’ve talked to colleagues about what makes a good peer review. Here are a few things we have come up with:
• Respond to the article that the writer submitted and respond to the piece in front of you. Do not try to get authors to rewrite their article into what you would have written, explore a different topic, or use a different methodology.
• Remember you are responding to a person who was as nervous, anxious, and vulnerable as you were when you sent out your first article. Don’t hide behind anonymity and blind reviews. Never say anything in a blind review that you wouldn’t say if your name were known or you were talking to this author in person.
• Peer reviewing is not only about evaluating, it is also about mentoring, nurturing and building a community of strong scholars.
• Think about the peer review process as a teachable moment. Give writers concrete, specific feedback they can use to make their articles better. If the article lacks focus, say something like, “Your article could be better focused around a central argument. On page four, you write _____. This strikes me as a solid summary of your piece, perhaps you could organize the article around this central idea.” No one ever wants to be told they need to do more work but giving someone concrete suggestions on how to make improvements makes revisions seem much more do-able.
• Understand that, increasingly, library research uses diverse methods and approaches. It is true that quantitative and qualitative research methods have traditionally been the dominant mode of scholarship in our field but there is a place in our scholarly literature for all kinds of methods. Don’t reject a piece simply because of its theoretical or methodological approach. Our profession will be stronger if we embrace the diversity of methods and approaches.
• Use criteria that are both appropriate to the journal and to the method the author employs. If the author submitted a quantitative paper, by all means, examine their statistical methods and their findings. If the author submitted a theoretical piece, consider their ideas fully and the logic of their argument.
• A peer reviewer is not a copy editor. If you notice the article is full of awkward sentences and typos, it is not your job to go through and fully edit and proofread the piece. Rather, you should note to the author and to the journal editor that there are a significant number of typos and/or awkward sentences and that these must be addressed.
• Have and maintain high standards but find ways to help authors reach those high standards.
• Finally: generosity is key. Always find something good to say about a piece. Be generous with your expertise and your understanding of what makes an article great. Help others achieve excellence.

As librarians become more active in scholarship, more and more of us will be taking on roles as peer reviewers. In this development, we have an opportunity to build a strong, supportive network of reviewers who can help us build a stronger body of published scholarship and a strong research culture.

This article gives the views of the author(s) and not necessarily the views of the Centre for Evidence Based Library and Information Practice or the University Library, University of Saskatchewan.

How to be an effective peer reviewer

by Kristin Hoffmann
Associate Librarian, University of Western Ontario

Peer reviewing is one of my favourite ways to participate in the librarian research community, because it combines two things that I find professionally rewarding and interesting: editing and mentoring.

In our academic writing and publishing system, peer review is a key part of the process, so it’s important that we do it well. However, there isn’t always a lot of guidance for those who are new reviewers. I’ve reviewed for some journals that have detailed reviewer guidelines, and for others that give reviewers a form to complete and not much else. I’ve reviewed book proposals that came with minimal direction, and I’ve reviewed conference proposals where the reviewer’s form was almost as long as the proposals themselves.

In all of those cases, reviewing was an individual activity. I’ve received almost no feedback on my reviewing, and I have had only occasional conversations with others about the process of reviewing.

In this post, I’ll give some suggestions for how to be an effective peer reviewer, in the hopes of helping novice reviewers and starting a conversation about reviewing:

  1. Think of peer reviewing as asynchronous, (usually) anonymous mentoring. A mentor is “an experienced and trusted advisor” (New Oxford American Dictionary), and a peer reviewer is an experienced and trusted advisor for someone else’s research. This is especially clear in open review processes, where the reviewer and author know each other’s identity, but it’s true of blind reviewing, too.
  1. Always find something positive to say about the piece you’re reviewing. Tell the author what worked well in their paper.
  1. Give specific, constructive suggestions. If you think the piece isn’t organized well, give the author ideas of how to re-structure it. If you think the analysis could be more robust, suggest additional aspects for the author to consider.
  1. When making substantive suggestions or pointing out flaws or concerns, do so in a way that shows you want the end result to be a better paper, and not that your goal is to assert your own superior research abilities. For example, “it would help if the Introduction included a stronger rationale for why this study is significant,” is better than, “this study is pointless and a monkey could have written the paper.”
  1. Remember that reviewers recommend, and editors decide. If you aren’t sure about your recommendation, tell the editor why you’re uncertain. They will look at your feedback along with that of the other reviewer(s) and make the best overall decision for the piece.
  1. Resist the urge to copy-edit. The editor needs you to comment on the submitted article as a whole, not the individual sentences. If the writing is hard to follow, or if the author consistently makes the same grammatical mistakes, then give a general comment about that. But remember that punctuation or sentence structure might change as a result of other, more substantial changes to the paper, so don’t put unnecessary work into copy-editing.

These are the main principles that I keep in mind as I’m writing reviews, but this isn’t an exhaustive list. I’d love to hear from Brain-Work readers: What other suggestions would you give to peer reviewers? What questions do you have about peer reviewing?

This article gives the views of the author(s) and not necessarily the views of the Centre for Evidence Based Library and Information Practice or the University Library, University of Saskatchewan.