AI-assisted programming tools like GitHub Copilot promise increased productivity – but how do developers actually experience working with Generative AI (GenAI)? A recent study by HPI PhD Candidate Charlotte Brandebusemeyer investigates how professional developers interact with GenAI in their real-world work environment.
To tackle the question and get a realistic impression, Charlotte Brandebusemeyer conducted an empirical mixed-methods study at two SAP locations in the US. She combined real-world behavioural data with developers’ personal experience of working with GenAI. The research is part of a collaborative project between the HPI and SAP. Together with co-authors Prof. Bert Arnrich, head of the "Digital Health – Connected Healthcare" department, and Prof. Tobias Schimmer, head of Developer Experience at SAP, Charlotte Brandebusemeyer’s findings from the study were accepted for publication at the flagship conference in software engineering: International Conference on Software Engineering (ICSE 2026) in the Software Engineering in Practice (SEIP) track. In April, she will present her work at the conference in Rio de Janeiro, Brazil.
Results of the study show that moderate use of GenAI assistance - such as in-code suggestions or chat prompts - increases efficiency and reduces perceived workload during working tasks. However, overly intensive or combined use can diminish these benefits. The study offers valuable insights for both academic research and industry contexts, highlighting the importance of a developer-centred, holistic view of productivity and developer experience.
The preprint of Charlotte Brandebusemeyer’s study "Developers’ Experience with Generative AI – First Insights from an Empirical Mixed-Methods Field Study" can be found here.
We interviewed Charlotte Brandebusemeyer about her research project.
Hasso Plattner Institute: What surprised you most in the study regarding how developers actually use AI?
Charlotte Brandebusemeyer: I was surprised by the diversity in how developers use GenAI in their day-to-day life and how choosing the right mode of interaction for each task is key for efficiency gains and reduced perceived workload.
HPI: What approach did you take in your study?
Charlotte: It was important to me to put the developer into the focus of the study. A lot of research concentrates on productivity gains and output quality of models, but the experiences and well-being of the people using these tools are often overlooked. To get a realistic impression, we conducted a study with professional developers directly in their everyday work environment. By giving them structured tasks, as well as letting them document their natural workday, and combining data from surveys, screen recordings and physiological data from wristbands, we are able to capture a broad and detailed view of how developers interact with GenAI in real life.
HPI: Why does moderate use of AI assistance lead to better results than intensive or combined use?
Charlotte: We found that how developers use Copilot makes a big difference. Intensive use of in-code suggestions often reflects a trial-and-error approach with a lot of code being generated but also deleted. Heavy use of the chat dialogue that problems with the code remain or further explanations are needed. Using both in-code suggestions and chat prompts for the same task can indicate that the initially chosen interaction type was suboptimal to solve the problem at hand, leading to more cognitive overload by changing the way of interacting with Copilot and creating small interruptions in the workflow. The key takeaway: choosing the right interaction type for each task upfront can help developers get the most benefit from AI-assisted programming tools.
HPI: How can I decide which Copilot interaction type is best suited for a task at hand?
Charlotte: As a rule of thumb, simple coding tasks that need only a small portion of the code as context and don’t require long explanations work well with in-code suggestions. For more complex or non-coding tasks - like brainstorming, writing summaries, debugging or exploring a large codebase – chat interaction tends to be more helpful. In general, tasks that require higher-level thinking, context, or creative problem-solving benefit from the dialogue-like nature of the chat.
HPI: What concrete lessons can companies take from the results of using Generative AI in everyday programming?
Charlotte: It is important to train employees on how to interact effectively with GenAI. Companies could use study setups similar to the one we designed to test new AI tools for their specific use cases and see how developers interact with them, as well as determine if they are helpful. This helps identify the best ways to apply an AI tool for different tasks and maximise its benefits, leading to a more satisfactory and productive experience and interaction for the developers.
HPI: Charlotte, thank you very much for the interview.
