The first time I reviewed a paper, I felt like I was back in school again. There was a timer, a proctor, a paragraph I had to read and then I had to choose the ‘correct’ answer or I got the question wrong.
Or back taking a Literature class. I’d read a short story or a book and then I’d have to write a paper about it. I’d have to choose the ‘right’ interpretation of the short story or book or… I’d fail that essay. My favorite English Literature professor in college told us that as long as we could justify it from the work, we could have any interpretation we wanted. There was no ‘right’, which was really nice. I wasn’t trying to guess what interpretation the professor was looking for, I was just trying to justify my opinion.
The same thing is true in writing a review, there’s no right. There’s you reading the paper and using your best judgement. No professor is going to take your review and go to town on it with a red pen. The first person to read your review will be the Associate Editor and they’re going to treat it like your opinion, not something that needs to be graded. You aren’t back in class, you aren’t worried about failing, you’re just giving your opinion on a paper.
And I do have some tips about the process. Things that will make it easier for you to review.
First, read the paper from beginning to end. I usually print it out and make notes on it, or use my iPad and make notes on it, or even read it on my computer and (you guessed it) make notes on it. These are notes for you for when you actually write the review, no one is going to read them. I look for things like ‘This is interesting!’ or ‘Wait… I read a paper on that’ (I make a note to go look up the paper if necessary), or ‘Um… wait, that data set has problems, I’ve tried to use it’. I”m looking for things to catch my eye. I might circle entire paragraphs and make a note to double check what it said.
The first read through is mainly to get an idea of what it said and what interests me about the paper or what problems I see immediately. If the author(s) neglected to tell me the origin of the data they used, I make a note. If they give me a URL, I might go look. My main goal in that first read is to answer the question ‘what can this paper teach me that I don’t already know?’.
Then I go look at the review questions. DTRAP has several and since I’m co-EiC of DTRAP, well, I’ll just use those as my example.
- What is this paper about?
- How does this paper contribute to the field of Digital Threats? What are the strengths of this paper?
- How can the paper be improved? What are its weaknesses? How can it be strengthened?
- Is this paper of potential interest to developers and engineers?
Using my notes, I consider each question. If I’m not sure, I go back and read the paper again. The first time I spent a lot of time going back and forth, by the third time I had enough experience to make my notes on my first read more useful for the actual review.
I hope those questions are straight forward. The DTRAP Reviewer Guidelines discusses more about what we expect from the answers to them, please refer to it if you aren’t sure. If you’re still unsure, contact either the AE in charge of the paper (he or she is the person that asked you to review) or one of the co-Editors-in-Chief of the journal. We want to help. We want you to feel comfortable reviewing, but we know it’s a new skill to pick up.
I do want to say whatever you write in answer to those questions, remember you’re speaking to the author. Be polite and respectful, always.