SAP - Developer Experience in times of Generative AI
SAP is a German software company that aims to provide the best-possible environment to collaboratively deliver enterprise applications. This includes increasing development productivity without compromising on developer experience and satisfaction. In this SAP-HPI research project, we conduct empirical studies directly at SAP to collect real-world, multimodal data on developers' experiences at work including their interaction with Generative AI. A developer-centered holistic perspective is taken.
Contact: Charlotte Brandebusemeyer
Funding
This research project is funded by the SAP-HPI research program.
Empirical Studies at SAP
Developer Experience During Generative AI Interaction
SAP Newport Beach, SAP Palo Alto 2025
The aim of this study was to analyse the software developer – Generative AI (GenAI) interaction with respect to the programming flow, cognitive load, stress, emotions, work satisfaction and developer experience (DevEx) of a professional software developer in a realistic setting. A holistic approach was taken by considering physiological data, behavioral data (screen, keyboard and mouse recordings), code metrics and subjective questionnaire data. This way, one receives a broad, multifaceted picture of the interaction with GenAI. Controlled and uncontrolled phases in this study enable reliable data recording whilst also capturing the natural work environment of software developers.
Contact: Charlotte Brandebusemeyer
Cognitive Load During Simulated Software Development Tasks
SAP Innovation Center Potsdam, SAP Berlin, SAP Signavio 2024/2025
Whilst programming, one can either be in the flow and everything works smoothly or struggle to concentrate and spend a large amount of time solving error messages. One reason for not being in the flow can be cognitive overload and stress. The aim of this study was to make the flow measurable by examining cognitive load and stress during software developers' everyday tasks. The programming flow can be measured by recording a person's physiological activity with body sensors during a programming task. Reactions of the eyes, the brain, the skin and the heart give insights into experienced cognitive load and stress. Making physiological activity visible in source code makes it possible to determine problematic sections in the code. This study involved the use of an IntelliJ Integrated Development Environment (IDE) plugin, CognitIDE. The primary function of this plugin is to record physiological data and link the physiological activity to specific parts of the code.
Contact: Fabian Stolp, Charlotte Brandebusemeyer
Developer Experience
SAP Newport Beach 2024
Developer experience (DevEx) is receiving more attention in software organizations. Developer experience comprises how a developer feels, thinks about and values his/her work. A positive developer experience encompasses satisfied employees who are highly productive. Noda et al. [1] identified three key dimensions of the developer experience: the feedback loop, cognitive load and the flow state. The feedback loop refers to feedback both from the tools one is working with as well as from other people. Slow and qualitatively low feedback interrupts the development process, which leads to delays and frustration for the software developer [2]. Cognitive load can be defined as the amount of mental processing required to perform a task. Problems such as poorly organized and documented code lead to extra time and effort needed to complete tasks successfully and therefore impose a high cognitive load. The flow state is considered an optimal state of productivity. The person in this mental state is fully immersed and involved in a task, is focused and has a positive experience [3]. Interruptions and delays in the feedback loop are factors that hinder a person from being or getting into the flow state [4]. Optimizing these three main factors of the developer experience therefore increases a software developer’s satisfaction and productivity.
So far, developer experience has mainly been assessed via questionnaires. Developer experience is highly personal and contextually dependent and subjective experiences can be captured in this manner. However, the concept of developer experience has not yet been systematically and objectively assessed with physiological measures. Physiological measurements have the advantage of measuring cognitive and emotional states in real-time. Such data helps to better understand what programmers go through at work, and measures well-being and productivity. Especially heart rate and perspiration, but also skin temperature and movement have been proven to be indicative of stress, cognitive load and the flow state [5]. A non-invasive way of collecting physiological data which can at the same time be easily integrated in studies outside of the laboratory, are with the help of wristbands or smartwatches. These devices capture the individual physiological reactions of software developers. This study motivates the use of wearables in a firm context to enable a holistic evaluation of developer experience by combining subjective and objective measures in a real-world setting. Efficient task scheduling and detecting first signs of work exhaustion could be enabled, which increases the software developer’s well-being and productivity and longterm reduces sick days and turnover rates in firms.
[1] Noda, A., Storey, M. A., Forsgren, N., & Greiler, M. (2023). DevEx: What Actually Drives Productivity: The developer-centric approach to measuring and improving productivity. Queue, 21(2), 35-53.
[2] Forsgren, N., Kalliamvakou, E., Noda, A., Greiler, M., Houck, B., & Storey, M. A. (2023). DevEx in Action: A study of its tangible impacts. Queue, 21(6), 47-77.
[3] Csikszentmihalyi, M. 2008. Flow: The Psychology of Optimal Experience. Harper Perennial Modern Classics.
[4] Janssens, S., Zaytsev, V. 2022. Go with the flow: software engineers and distractions. In Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings, 934–38;
[5] Charles, R. L., & Nixon, J. (2019). Measuring mental workload using physiological measures: A systematic review.Applied ergonomics,74, 221-232.
Contact: Charlotte Brandebusemeyer
