I’ve been asked exactly what the board does and how the Associate Editors help shape the journal. This post answers that.
ACM Digital Threats: Research and Practice has two Editors-in-Chief (co-EIC) and a Board of Associated Editors (AEs) that shape the journal through soliciting papers, creating special issues, and managing the review and acceptance process. Since the quality and value of a research journal is defined by the quality of its paper, reviewing papers is at the core of what makes a journal valuable. An AE’s primary operational responsibility then is to shepherd the review of papers.
This document describes the job of an AE in shepherding papers through the review process.
The process begins by a co-EIC assigning a paper to the AE for review. While it is most desirable for an AE to be assigned a paper in their chosen field, due to the nature of the field and composition of the AE Board, this may not always be possible. Hence, an AE may sometimes be called to shepherd the review of a paper in an area different from their expertise..
The AE’s first task is to do a preliminary review to determine if the paper is worth shepherding. Based on their expertise, experience, and judgement, an AE may recommend to the EiCs that the paper, in their opinion, should be “immediately rejected”, and need not be sent for further review. There may be any number of reasons for this. The paper is not aligned with the theme of the journal, the paper is written poorly, the paper very obviously doesn’t contain any material worthy of publication in DTRAP are just some such reasons..
Next the AE finds qualified reviewers and invites them to review the paper. In the entire shepherding process, this is the most important, challenging, and, often, frustrating part. The choice of reviewers is important because it defines the quality of the reviews. It is challenging because in a field as broad as cybersecurity, an AE may not readily know all the qualified researchers or research groups. It is frustrating in that the researchers who are most knowledgeable about a subject are often also very busy, and are more likely to deny the request. The good news is that between ACM and Google Scholar there are plenty of resources available to make it rather easy to find qualified reviewers. It is then a matter of inviting twice as many reviewers than needed, expecting that half may not be available.
Once a prospective reviewer has accepted to review a paper, the AE’s job is to watch the schedule, and make sure the reviews are completed in a timely manner. This is as important for academic researchers, whose career is defined by their publications, as for practitioners. We don’t want to leave the authors wondering what happened to their paper, we want to return a review to them as soon as we can.
Finally, after the reviews are all completed, the AE then carefully reviews the reviews, and makes a recommendation to the co-EIC. This is where the AEs professional judgment comes in. It is quite rare that all the reviewers return the same recommendation. It is the responsibility of the AE to sift through potentially conflicting recommendations and arrive at a recommendation of their own.
The AE (as also each reviewer) can make one of four recommendations for a paper:
- Reject — The paper isn’t ready for publication and the AE doesn’t think the problems with it can be fixed easily. This doesn’t necessarily mean it’s a bad paper, just that there are too many problems with the paper. It could also mean that it isn’t a good fit for DTRAP. For an extreme example, a paper on the threats of an abacus isn’t a digital threat, so we’d have to reject that. It could also mean that the problems with the research are such that to fix them, the entire paper would have to be redone. That’s creating a new paper, not editing an old paper.
- Major Revision — The paper has some good ideas but is incomplete in some significant ways, as for instance the experiment performed may not be convincing enough. Once the authors return a revised version, the paper should undergo a full review, if possible, by the same reviewers. Think of it like a boat that has many leaks but if those are plugged, it will still sail. It’s not a case of rebuilding the boat, just plugging the leaks. However, after the boat is fixed you want to do a full inspection..
- Minor Revision — The paper is mostly good but there’s a few things that could be repaired to make it the best paper it could be. The repairs are minimal, would not alter the conclusions of the paper, and do not merit a full review. The revised version may be reviewed by the AE or one reviewer just to confirm that the recommended changes were addressed..
- Accept — The paper needs no significant changes. It is good just the way it is, aside for some editorial corrections that need no further review.
The AE then returns their recommendation to the co-EIC who then makes the final decision, which is communicated to the authors, the AE, and all the reviewers.
To boil that down, shepherding a paper involves these requirements for the AE:
- Be an expert in their field and be open to other fields in Digital Threats
- Be able and willing to reach out to experts they may not know and request a review
- Be able and willing to follow up with people for reviews
- Be willing to resolve conflicting recommendations from reviewers
- Be willing to do this entire process in a timely manner.