Explainable AI: Shedding light on the black box

10/22/2025

AI is a black box. This is something you often hear when talking about complex applications of artificial intelligence. The more challenging the task that AI is supposed to perform, the more difficult it becomes to understand exactly how AI arrives at its result. The field of explainable AI research therefore focuses on better understanding the inner workings of AI in order to increase trust.

HPI master's student Marius Dörbandt spent two summer months at the Massachusetts Institute of Technology investigating what happens in an AI model during the training phase. The project is part of the research collaboration between HPI and the MIT Morningside Academy for Design (MAD). Part of Marius' team at MIT includes PhD student Rem Yang and postdoctoral researcher Charles Jin. Their research is led by Prof. Martin Rinard at MIT and Prof. Robert Hirschfeld at HPI. 

During his time at MIT, Marius not only conducted his research but also experienced the start of the semester and the welcome events for new students. In this interview, he talks about his project—and his personal experiences beyond research.

Hasso Plattner Institute (HPI): What is your project about, explained in simple terms? 

Marius Dörbandt: Modern AI models such as ChatGPT usually go through several training phases. The largest is pre-training, during which most of the knowledge is acquired. This is followed by smaller, so-called fine-tuning phases. In these phases, the models are taught, for example, how to solve certain tasks. I am researching what happens within the models as they go through fine-tuning phases.To do this, I use so-called probes.   

HPI: What are probes, and how do they work?  

Marius: Probes are comparatively tiny AI models. They receive the “nerve impulses” that flow within a large model as input. They are then supposed to extract certain information from this. In my case, the model programs a small robot to navigate it through a virtual world full of walls to a destination point. A probe is designed to find out, for example, whether the robot is currently standing in front of a wall. If the probe manages to extract this information from the impulses, then this is an indication that the model has an internal understanding of the virtual world and actually knows what it is doing when it programs the robot.   

HPI: What question do you want to answer in your project?  

Marius: What happens to probes while the associated AI model is undergoing a fine-tuning phase?  

HPI: In what contexts could what you are working on be used? What challenges could it help with?  

Marius: I hope to contribute in my own small way to a better understanding of the inner workings of large AI models. AI is often described as a black box, and research into interpretability/explainable AI helps to make AI more of a white box. 

HPI: You got to experience the start of the semester at MIT. What is it like when a new semester starts and the campus fills up with new and returning students?   

Marius: First of all, there are lots of welcome events, e.g., from my dorm or the various student clubs. We also went to two fraternity parties. That's probably where the most bizarre thing happened: We were standing a little way away from the entrance to the fraternity, where lots of (presumably) first-year students were trying to get into the party. Out of nowhere, a fraternity member came up to us and asked if we were MIT students. He then guided us past the long line and through the back entrance. This gave us a little insight into what fraternities are like, but we didn't stay at the party for very long.  

Of course, lectures have also started, and I've attended a few of them. You also just see a lot more people when you walk across campus or sit in one of the many public spaces. A great side effect of a busier campus is that you can spontaneously do things with other students. For example, I was on my way home when I noticed a group of students playing volleyball in front of my office. I asked if I could join them and ended up playing with them for quite a while. 

HPI: What did you do in your free time at MIT?  

Marius: I tried to make the most of my short time there and take part in as many activities as possible. I enjoyed getting to know different parts of Boston, initially via the Freedom Trail, but I also visited various neighborhoods, such as the Italian North End and Chinatown. I also took a weekend trip to New York City with the three other students from HPI. In addition to the classic sightseeing activities, we also went to see a football game. In Boston, I also went to see a baseball game. In both NYC and Boston, I enjoy visiting art, science, and natural history museums – one highlight was the whale watching tour from the New England Aquarium. On the social side, I enjoyed attending events organized by Vista, MIT's Visiting Student Association, including a biweekly meeting at an MIT pub and weekly lunches. I also enjoyed taking advantage of the many sports activities offered by MIT and my dorm.   

HPI: What was your absolute highlight outside of work?   

Marius: The trip to New York City, closely followed by the whale watching tour and the trip to Provincetown.  

HPI: What would you like MIT to adopt from HPI?  

Marius: Friday Beers!   

HPI: And vice versa?  

Marius: I think MIT and HPI are quite similar, except that everything is bigger at MIT. For example, there is a large sports center for many different sports. What also struck me—perhaps because I was part of it—was the size of the international community at MIT. At HPI, the international community is still smaller.  

HPI: What will happen with the project after your stay? 

Marius: We were working on a paper. The plan was for me to finish the experiments on site and for us to start writing the paper towards the end of my stay. Now that I'm back, we'll finish the paper.   

HPI: What is the most important thing you learned during your stay? 

Marius: AI training takes a really long time, even when working on 12 GPUs simultaneously. 

Thank you very much for the interview, Marius!  

Learn more about HPI's research collaborations here.