About the Speaker

Prof. Dr. Niclas Böhmer is one of the newest Professors at HPI. After finishing his PhD at TU Berlin in 2023, Prof. Böhmer worked as a Postdoctoral Researcher at Harvard University and joined HPI in September 2024 where he started the research chair for Algorithmic Decision Making and Society. His research deals with the design and analysis of algorithms that aggregate the individual interests of groups of people to make collective decisions or to fairly allocate resources. After having published more than 50 papers in the span of 6 years, his lecture in the series is dedicated to the process of scientific publishing and reviewing.

Overview of Talk

As researchers, publishing is the final frontier where months, even years, of hard work face the ultimate test. But how do we turn our results into a compelling story that stands out? What happens behind the scenes when it comes to conferences, journals, and peer reviews? How does reviewing work? Why should you review? Prof. Dr. Niclas Böhmer shares insights into these topics in his talk, "Reviewing and Academic Publishing." With real-life examples and practical tips, his lecture gave a more detailed look at the academic publishing process—the victories, the challenges, and the unexpected twists along the way.

Before we start: please be aware that much of this information comes from Prof. Böhmers personal experience in specific subcommunities (mainly theory or AI focused). Every area (even just in computer science) has its own processes and publishing and reviewing may work very differently there. Take everything written here with a grain of salt, there may be some things that can't be applied to your field of study.

From Scientific Result to Writing a Paper

Before we can focus on conferences, journals, and reviews, we first have to discuss the act of creating a publishable result. Most of the students present in the lecture might soon face a situation like the following: Your Master's thesis is done and handed in and in it are some results that could prove useful to other researchers. The first step of transforming a scientific result into a paper is identifying what the main results actually are. A thesis, for example, usually contains a thorough introduction and related work, or repeats prior work to give additional context. To identify key findings, questions like "What do I want to say?" need to be raised. Does the result provide a new perspective on existing topics? How does a novel technique improve upon the state of the art? What is "the thing that I contributed that was not there before"?

These questions not only help with organizing one's own thoughts, but are also crucial for setting a narrative. Prof. Böhmer says that writing papers is two thirds science and one third storytelling. A paper needs to motivate others to read it, it needs to make people want to engage with it. A good paper should clearly relate itself to prior work and authors should make sure to ask themselves:

• How do the results differ from the expectations?
• How do they improve on the state of the art?
• Relating work to existing research
• How can I tell random people on the street why my research is relevant?

and include these thoughts in their narrative.

To be fair, while the ability to "motivate random people on the street" is quite the achievement already (depending on your area of research), it might still not impress your not-so-random reviewers. In fact, choosing which venue to submit your work to is on its own a major step of publishing.

Finding the Right Venue is an Art

There are countless conferences every year that one can submit research to. As an example, Prof. Böhmer counted the number of distinct venues that he himself, Prof. Dr. Tilmann Rabl (Data Engineering Systems), and Prof. Dr. Tobias Friedrich (Algorithm Engineering) have successfully published with. Prof. Böhmer says he worked with 20 venues so far and has on average two papers per venue. Prof. Rabl and Prof. Friedrich, both academically 20 years older than him, appear as authors on papers at 40 and 100 (!) venues, respectively. In most cases, you will first do the research and then find a fitting venue. As conferences are held all over the world, for many people the destination is the most important factor when choosing one. Prof. Böhmer lists six (more practical) factors to consider when choosing a venue:

1. Scope fit: Conferences are usually scoped to specific topics or areas. In particular, the definition of what qualifies as, say, "machine learning" differs between venues.
2. Reputation: Conferences are ranked based on several factors (this was covered in the talk by Stefan Neubert and Timo Kötzing). Depending on one's confidence in the relevance of their results, one should maybe aim towards one or the other end of the spectrum.
3. Timing: Conferences happen usually once a year and have strict submission deadlines. If your result fits a venue whose deadline just passed, it is probably not worth it to wait a full year.
4. Quality level: Naturally, more prestigious venues require a higher quality from submitted papers.
5. Audience: For the case that one's work is not only included in the proceedings of a conference, but should be presented at the conference in some way, one needs to consider whether their work is suited for the audience that is expected to visit the event.
6. Mannerism: Some conferences have "strange implicit rules". These rules can include peculiar things like the numbering of sections, or what sections there are to begin with. To Prof. Böhmer's dislike, it is impossible to know these rules unless one infers them from previous publications or reviews.

Figure 1: Factors to consider when selecting a venue for paper submission

Conference vs Journal

Generally there are two different ways to publish your scientific work: at a conference or in a journal. Conferences often require a fee, they are live events, they include talks, posters, presentations and go all day. The application process is very fast-paced and there are strict deadlines for everything. To apply you have to submit a (typically short) paper, which is normally subject to a lot of rigid guidelines. This artificial shortening often means that essential parts of your research get added to the appendix, where few people will actually read them.

Journals on the other hand typically don't include fees, and are no live events. This also means there are basically no deadlines and the whole process takes a lot longer. Formally, you have a lot more freedom when writing your paper. They tend to be a lot longer, better written and are a lot more accessible (may include examples and further explanations).

Journals and Conferences don't have to be mutually exclusive. You can submit your paper to a journal after attending a conference with it. This means there are commonly multiple versions of a paper: a short version for a conference and a long version for a journal.

Conferences have a better prestige in the Computer Science area. As it is perceived as a fast-paced field, a fast-paced process seems fitting. More theory-focused fields tend to see journals as more prestigious.

Peer Reviewing

Peer reviewing is the evaluation process of some work by other specialists in this field. Why do we need this? The practical reason is of course that the space at a conference is limited and there needs to be some selection process. The maybe more formal reason is that it functions as a quality stamp. There are two sides to this, on one hand you want to be sure the content of the work is correct. On the other hand you want the work to be adequately written and a good fit for the specific venue. (Later on are some example criteria regarding this)

The specific process after submitting depends on the venue, but generally it is as follows. The reviewers for this conference will get papers to review assigned. This may be based on some bidding process, but generally papers get assigned to reviewers who have some expertise in the area of the paper. The reviewers then give feedback and a score. There may be a second round after some initial rejections and revisions. After hearing back for the final decision, you revise your paper based on the feedback regardless of the outcome. If the paper got accepted, you want to do some final revisions. If it got rejected, you may want to apply to some other conference and revise it for that.

Coin Flip Tale

When it comes to submitting a paper, there's always an element of uncertainty—what Prof. Böhmer calls the "coin flip" tale. Even after all the effort put into writing a solid paper, there's still a chance it could be rejected. Think of it like flipping a coin: no matter how well you've crafted your ideas, the outcome isn't always in your control. This randomness is an inherent part of the peer review process, where factors like reviewer preferences, submission volume, and even the physical capacity of the conference or journal venue can influence the decision.

Now, that doesn't mean you're totally powerless. Authors can certainly stack the odds in their favor by carefully selecting the right venue, presenting their work in an engaging way, and clearly articulating the novelty and significance of their contributions. But the fact remains, some of the acceptance or rejection still depends on things you can't predict or control.

This is where the coin flip metaphor really resonates. While you can't guarantee success, you can certainly improve your chances. But even in the most prestigious conferences, rejection is always a possibility. It's just part of the game—and it's worth remembering that the process isn't always about the quality of your work; sometimes it's just about the luck of the draw.

Study about Inconsistencies

One interesting study Prof. Böhmer mentioned during the talk explored how often decisions about the same paper differ when reviewed by independent committees. The researchers set up two separate review pools for the same paper to see how often the decisions made by each pool aligned or conflicted. The question at hand was: What is the % of the papers that yield inconsistent decisions? [1]

Before diving into the results, it's important to keep in mind that about 70% of papers are typically rejected, with only 30% being accepted at most conferences. This makes the maximum potential for inconsistent decisions roughly bounded by those acceptance rates. The research found that around 25% of decisions were inconsistent, meaning that about one-quarter of papers that were accepted by one committee were rejected by the other.

Figure 2: Table showing the percentage of inconsistent decisions in paper reviews

This is a bit unsettling, considering that these committees were working independently, with no knowledge of each other's decisions. But what's even more surprising is how this inconsistency seems to persist over time. A follow-up study in 2021 showed that these results didn't change much. In fact, the numbers stayed pretty much the same—about 25% inconsistency and 50% of decisions differing between committees.

Figure 3: Results from the 2021 follow-up study on review inconsistencies

So, what does this tell us? First, it means that even if your paper is accepted by one committee, there's a parallel universe where it might have been rejected. Or, if your paper gets rejected, there's another universe where it could have been accepted. It's a reminder that the peer review process, while essential, is still inherently subjective and unpredictable. Decisions aren't always based purely on the quality of the work; they're influenced by a range of factors, including the biases or priorities of the reviewers.

This study also highlights an important point about conference awards and highly selective venues: the more selective the venue, the higher the likelihood of inconsistent decisions. For example, if you're aiming for the "top 5%" of papers out of thousands, there's no guarantee that everyone will agree on what constitutes the best. This adds a layer of complexity to conference decisions, especially when committees are choosing award-winning papers. Ultimately, what we learn from this is that every decision in academic publishing should be taken with a grain of salt. Even if you get rejected or accepted, it's not the end-all-be-all. The process is a bit random, and there's always a degree of uncertainty in how papers are evaluated.

Overview of an AI Conference

Prof. Böhmer presented the structure of an AI conference as an example. The committee there consists of multiple layers:

• PC Members (Reviewers): They are the workforce that get assigned the papers to review. Typically one PC Member gets assigned 2-7 papers. They are expected to write comprehensive feedback and make a justified recommendation.
• Senior PC Members: They lead the review teams for approx. 10-20 papers. They verify review assignments and may validate (and discard) reviews. They also organize and moderate the discussion with the PC Members. Afterwards they write a meta-review containing a justified recommendation based on the discussion.
• Area Chairs: They coordinate all papers of a specific area (around 100 papers). They validate and calibrate meta-reviews. They also handle unresponsive S(PC)s and difficult cases, e.g. in the case of plagiarism.
• Program Chairs (PCs): They are the organizers of the conference. They set the rules for reviewing; find (and approve) PC Members, ACs and PCs; oversee selection of accepted papers; and oversee the award selection.

Figure 4: Hierarchical structure of an AI conference review committee

Roles of a Review

Reviews are an opinion piece. They are mainly your subjective opinion and should therefore convince others of your thoughts about a paper. They should also provide constructive feedback that will hopefully help better the paper in future iterations. Finally, a review is supposed to contain a summary (not only to proof you read the paper) to provide SPCs and ACs with the information they need to decide about a paper.

Reviewing for Journals

In journals, the process begins when a paper is assigned to an editor, who acts as a paper manager. The editor then selects a group of experts, usually five, to review the paper. Review requests are sent out individually, inviting experts to evaluate a specific paper. This is an invitation-based process, where reviewers are recruited specifically for a single paper. The journal review process is typically slower compared to conferences. Editors, instead of Associate Chairs (ACs) or Senior Program Committee members (SPCs), manage the process. In journals, there are revision rounds instead of rebuttals, meaning that after the initial review, authors are asked to revise their paper based on the feedback they receive before resubmitting. This process can take much longer, with the overall timeline often being slower than that of conferences.

Figure 5: Days from submission to decision for papers that are sent to reviewers at the Journal of Machine Learning (JMLR). [2]

Evaluation Criteria

Some general criteria to consider when reviewing a paper:

1. Correctness: Are there any errors in the models, theory or the experiments?
2. Clarity: How well is it written? Is it accessible for a larger audience?
3. Novelty: Are the results presented new? What do they contribute to the field?
4. Significance: Are the findings significant? Will they have future impact?
5. Reproducibility: Can the results presented in this work be reproduced?

Prof. Böhmer also presented some personal criteria he likes to consider when reviewing:

1. Interestingness: What did I learn while reading the paper?
2. Importance of the problem
3. Convincingness
4. Honesty: Does the paper deliver what it claims to achieve? Are the authors overestimating the importance of their results?
5. Depth: How much time was put into this?
6. Positioning in related work
7. Fit to the venue

Parts of a Review: Score

When it comes to reviewing papers, the overall assessment is the big takeaway–the score that sums up how a reviewer feels about your work. At the top the scale is Award Quality which means your paper is groundbreaking and could win the best paper award. At the bottom is Strong Reject, which is a clear "no." In between, you've got everything from Strong Accept (inspiring, excellent work) to Weak Reject (not quite ready yet), with Borderline Accept/Reject sitting in the middle, reflecting uncertainty.

The middle scores, like Borderline or Weak Accept, often mean the paper has potential but might not stand out enough to make the cut. In competitive conferences, even decent scores like Weak Accept might not be enough to get accepted. For authors, the score is important, but it's not everything. The written feedback that comes with it usually holds the real value—explaining what worked, what didn't, and what could be improved.

Reviews are frustrating

For authors, it is often a frustrating experience when they submit their paper and encounter reviews. Reviews range from insightful and constructive commentary to rushed, shallow, uninformative, and sometimes hurtful comments. And negative reviews occur quite frequently, sometimes even without specific or actionable criticism, whereby the author then does not even know how to move forward with improvement.

On the other hand, reviewers often invest much time and energy in their reviews and then see authors disregard their suggestions, adding to the frustration. The mutual dissatisfaction stems from the fact that authors and reviewers are often the same people wearing different hats, yet the system doesn't seem to satisfy anyone. The process has its flaws, but it is a part of academia, and for that respect, politeness, and constructive discourse is to be expected from both sides.

Why Should I Review?

By now reviewing papers is a requirement in academia—but it's also a chance to grow as a researcher and contribute to the community. In the past, it was understood that if you want a career in academia, you can't skip the peer review process. Reviewing papers is key to advancing in the field. The people who ask you to review today could write your recommendation letters tomorrow. It's all part of the network that shapes your future.

But beyond the professional side, reviewing papers can be rewarding. You get early access to the latest research in your field, sometimes months ahead of everyone else. You'll see work before it's published, giving you a chance to think critically about it and develop ideas for your own research. It's like getting a head start on trends and knowledge, putting you ahead of the curve.

Reviewing papers also helps improve your own work. As you critique others, you learn what makes a paper clear, well-structured, and engaging. This can guide you in becoming a better writer yourself. In the early stages of your career, reviewing papers can open your eyes to what works and what doesn't, helping you shape your own papers.

While it's true that submission quotas in fields like computer science can make reviewing feel more like a chore, the benefits remain. It pushes you to read papers carefully, evaluate research critically, and give feedback in a constructive way. It's also a way to give back to the academic community. Someone reviewed your work, so reviewing others helps maintain the collaborative nature of research.

References

1. Corinna Cortes and Neil D. Lawrence. "Inconsistency In Conference Peer Review: Revisiting the 2014 NeurIPS Experiment." (2021) arXiv preprint. DOI: https://doi.org/10.48550/arXiv.2109.09774
2. Fabian Pedregosa et al. "Retrospectives from 20 Years of JMLR." (2022) Blog post URL: https://www.jmlr.org/news/2022/retrospectives.html. Accessed at: http://jmlr.org.s3-website.eu-central-1.amazonaws.com/news/2022/retrospectives.html