Introduction the of speakers
Dr. Timo Kötzing is a lecturer and chair representative at the chair for Algorithm Engineering of Prof. Dr. Tobias Friedrich. Some HPI students may also know him from one of his undergraduate courses, such as Mathematics I or Mathematics II.
Stefan Neubert is a Ph.D. student, also at the Algorithm Engineering Group. Both have written numerous research papers in the area of theoretical computer science which have been published in academic journals or presented at conferences. Two selected examples are the Incremental Ordering for Scheduling Problems [1] and Multiplicative Up-Drift [2] and have thus gained a broad understanding of the inner workings of the scientific field, with a bias towards the German theoretical computer science area. Timo even taught a course at HPI focusing on research.
Overview
After last week's talk about the meaning and theoretical foundations of science, Timo and Stefan went into detail about how these ideas are put into practice. They began with a description of the different scientific communities, asking why people organize themselves in science-oriented groups. Then, they continued with an interactive discussion about the characteristics of good scientific work and how we can control its quality. They also explained how to choose the scientific venue where you want to publish your research. In addition, they discussed what a scientist must have as a formal qualification to publish papers and shared some insights into this process. Finally, they traced the money in science: where it comes from and how a scientist obtains funding.
Problem statement
Last week we talked about the importance of science, but it remained unclear how a scientist can make money or publish their findings. This week's talk focuses on the sometimes sub-optimal practical implementation of science. As Timo and Stefan said last week, science can be done just for the beauty of it. People want to follow their curiosity and create something new, but it needs to be distributed to reach other scientists. You need to find out what others are doing, connect with them, and understand their work. The first part of the talk focuses on how a scientific community can work.
In addition, many scientific advances cannot be directly exploited economically. This raises the question of who funds this science. The second part of the talk focuses on how science can be funded.
Summary
What scientific communities exist (in CS)?
A scientific community can be defined as a group of people who come together to achieve a goal related to science. The most obvious goal is to do research, as in universities or research institutions. In most universities, there are different fields of research that allow researchers to foster collaboration between fields. Most importantly, research institutions produce new researchers and provide them with an introduction to the scientific field, like in this course.
Various activities are undertaken by the scientific communities. As a starting point, they promote collaboration within the research discipline. There are organizations such as the Association for Computing Machinery (https://www.acm.org/) and the Gesellschaft für Informatik (https://gi.de/) that specialize in a single field, here computer science, to connect people from different universities, academic levels, and research interests. By organizing conferences and curating publications, they advance scientific research by facilitating the determination of the state of the art (SOTA). In addition, they motivate researchers by awarding prizes such as the best paper/poster award, the Turing Award, the best Ph.D. thesis, and so on. Scientific communities also establish scientific standards and agreements on common ethics. Recently, the community in the field of generative AI has reached an agreement on when and how to use generative AI. There are also agreements on broader issues, such as ethics in science, and violating these rules can lead to people being kicked out of the community. However, the diversity of research domains within these broad scientific communities also means that they cannot form tightly-knit groups, and not all research is relevant to every researcher. This is why there is a community for a common research interest, such as the Association for the Advancement of Artificial Intelligence (https://aaai.org/), which organizes conferences only for that field.
In addition to research communities, there are organizations that represent researchers, such as the national academies (in Germany it is the Leopoldina (https://www.leopoldina.org)). They represent all the researchers of a country to lobby for their interests and disseminate news. Unions represent researchers to their employers, allowing them to obtain better working conditions.
Scientific Quality Control
Publications are crucial, as they provide the community with new insights and allow other scientists to build on the results, but how can you be sure that a published result is correct? Timo and Stefan present heuristics to judge the quality of papers. One of the most important signs is the number of citations. The idea is that if many people base their research on a publication, it must be correct. Of course, using this heuristic alone has drawbacks. It is essential to look at the quality of the citations.
Other statistics about the authors are also relevant, such as previous work or scientific standing, although people can be misled and trust a highly respected scientist more easily. For authors, there are a plethora of different rankings (see en.wikipedia.org/wiki/Author-level_metrics), all of which have drawbacks in one way or another. The most common metric is the h-index, you get an h-index of n if the person has n publications with at least n or more citations. The main problem is that the citations of the citations do not matter, i.e. one could game the system by publishing many papers that cite each other.
Another important indicator is the publication venue. If a venue includes peer review, this usually builds more confidence in the quality of the paper. However, since reviewers don't get paid for their work, superficial reviews are also common. Also, reviewers may be biased and value a paper more if, for example, it was published by Berkeley or a Turing Award winner. Therefore, reviews should be double-blind (see Niclas Böhmer's talk on January 20 for more details). In general, journal publications are preferred over conference publications, as they go through more iterations of reviewing.
Researchers from lesser-known universities or with more fringe research may experience the gatekeeping of conferences that try to conform to some internal customs. Thus, the paper will be published at a lesser conference, which does not mean it is bad research.
Furthermore, the quality of the paper's writing is an easy indicator of the overall quality of the paper. If the grammar and spelling are bad, it has not gone through many revision cycles (more details in Holger Karl's talk on January 20).
In addition to the writing style, you can look at the methods used in the paper; if the methods presented make sense and fit the type of work they are building on, it gives credibility to the paper. If the authors use only a few sources to build their work or do not cite important papers, this can be a sign of poor academic work.
Ultimately, a good paper should be able to withstand one's own review, since none of the quality control mechanisms are guaranteed to determine paper quality with absolute certainty.
What are good venues?
A scientific publication derives much of its credibility from the venue in which it is published. Therefore, it is important to know the quality of a venue. There are rankings like the ICORE that classify conferences into categories. The problem with these rankings is that they are somewhat subjective. Another way to measure the quality of a venue is to look at the publications that are presented there and if the publishers are credible. Here we can use the metrics from the Scientific Quality Control section to rate the individual submissions at a conference. If the submissions get many citations or the authors have a high h-index, it seems to be a good conference.
Therefore, one should pay attention to the venues one wants to submit to. A website that helps with this is thinkchecksubmit.org for journals and thinkcheckattend.org for conferences.
Formal requirements to participate in science
Timo and Stefan show the formal requirements a publication has to have to not be instantly rejected (desk reject) based on the AAAI 2025 (https://aaai.org/conference/aaai/aaai-25/) conference. The academic career of the authors is not important, meaning they do not need to have studied or achieved any other training. This could not be checked by the reviewer as the paper is not allowed to mention the author’s name in the review process to keep it double-blind. Also, full proofs for the theorems or sources are not required. The only formal requirement is that it should adhere to their style-template so that authors stay within the content limit.
Of course, just because it is not a desk reject does not mean it is accepted and many of the voluntary points will boost the acceptance chance dramatically. The authors will need academic training in the research field, to know the important papers to cite and write in the preferred style. In addition, there can be legal requirements (like for experiments with humans) that need to be fulfilled.
For researchers trying to submit to a certain conference, it is a harsh reality that most conferences have a set number of publications they can accept, meaning their publication can be just as good as all others and still get rejected due to the conference’s threshold on number of papers being reached.
Where does the money come from?
Research can be very expensive. Scientists require salaries and the cost of equipment and research facilities is not to be underestimated. Another curiosity regarding this topic is that journals do not pay authors royalties; instead, many journals charge authors a page fee. This creates a high barrier to entry for researchers.
As shown in Fig. 1, the majority of the funding is sourced from industry, which provides financial support for research and development. They will mostly fund applied research related to their products. Basic research is mainly funded by the public sector, such as the DFG or BMBF in Germany. A small percentage of funding comes from foundations such as the Stifterverband.
