Prof. Dr. Patrick Baudisch

Adding Force Feedback to Mixed Reality Experiences using Electrical Muscle Stimulation

(click to watch video)

Summary: We present a mobile system that enhances mixed reality experiences and games with force feedback by means of electrical muscle stimulation (EMS). The benefit of this fully mobile setup is that it adds physical forces while keeping the users’ hands free to interact unencumbered—not only with virtual objects, but also with physical objects, such as props and appliances.


Motivation: why is AR haptics so ellusive?

Unfortunately, not much of the existing haptic technology applies to Mixed Reality (MR) or Augmented Reality (AR). While Virtual Reality users have access to haptic gloves and exoskeletons with tactile and force feedback, the main challenge for MR is that users may encounter not only virtual objects, but also physical objects. This means that haptic technolo- gy for AR/MR must leave the users’ hands unencumbered. Furthermore, MR users may want to avoid any kind of bulky technology, as these tend to be visible in MR and in extreme cases might even occlude the real-world objects users are trying to interact with.

While some researchers have proposed ways to simulate the tactile qualities of objects in MR, e.g., by vibrating the user’s fingernails instead of the fingertips, simulating the physical resistance of objects continues to be a challenge. The reason is that the standard solutions based on mechanical actuators such as exoskeleton gloves and wearable pulley systems tend to be large and cumbersome.
We demonstrate, by means of four simple mixed reality experiences built for the Microsoft HoloLens, how EMS supports force feedback not just in VR (as our previous research demonstrated) but also in the broader spectrum of Reality. This includes adding force feedback on a variety of situations rooted in the Reality-Virtuality continuum, ranging from interacting with purely virtual objects, to passive props and augmented physical devices.

Our approach: using EMS to provide AR/MR haptics while keeping the hands free

In this research project, we propose adding force feedback in MR games and experiences using electrical muscle stimulation instead—an approach that is small, light, and fits under the user’s clothing. The figure below illustrates this at the example of our Mixed Reality balance marble game using a physical tray as a game prop, which our approach augments via EMS-based force feedback. Using EMS our simple game is able to, for instance, render shifts in the board’s center of gravity, caused by the “weight” of the moving virtual marble, by actuating the user’s arms towards the heavier side.

Figure (top) User is playing a "balance the marble game" with a physical tray, the EMS renders force feedback effects such as the moving weight of the marble on top of the game board. (lower) View from what the user sees in the HoloLens. (Marble Game is an homage to Ohan and Feiner's research.)

Applications: adding haptics to MR and AR experiences

1. A pure virtual experience (Shooting the catapult): The figure below shows our simple MR game featuring a virtual catapult that appears in the user’s physical surroundings. (a) The user arms the catapult by pulling the catapult’s bucket backwards. As the user pulls the bucket backwards, our system provides force feedback that simulates the increasing tension of the catapult—this force feedback also serves as an indicator for the user on how far the projectile can be expected to fly. Our system achieves this by stimulating the antagonist muscle (triceps) proportionally to the catapult’s arm angle. (b) When the user opens the hand (which terminates the HoloLens gesture) the catapult releases. Our system abruptly ceases to stimulate the triceps ceases and the remainder force in the user’s biceps, which now does not have a counter force, creating a sensation of actual recoil.

(top: rendered Mix Reality view. Lower: HoloLens view from user's perspective.)

2. Augmented physical props: balance marble game. Figure 1 (see top) depicted our MR marble maze, which was in fact inspired by that of Ohan and Feiner and complemented with EMS-based force feedback for added realism. As previously explained, our marble game renders the shift in gravity caused by the moving marble by actuating the user’s arms towards the heavier side. Note that even if the marble perfectly balanced on the center of the tray we render a constant pull down of the user’s arms (triceps) to represent the marble’s weight under gravity. Furthermore, we render a number of extra haptic effects: (1) when the ball collides with any of the maze walls, we render a short bump in the user’s triceps, depending on the ball’s velocity vector; (2) when the marble falls out of the tray in one of the openings in the obstacle walls, the user feels the relief in the weight; and, (3) when the game starts, the marble falls from the sky onto the tray and we render this “extra weight” by quickly pulling down the user’s arms, as depicted in the Figure below.

3. Augmented appliances (escape room experience). We implemented a simple Escape Room [14] in Mixed Reality. In the traditional version of these experiences, users must first find the solution to a puzzle, and only then they can escape the current room. The Figure below illustrates the user solving this room’s puzzle. The user sees a virtual message on the wall next to the virtual door that provides a clue to the puzzle: “the lights will illuminate your path, but only in one special way”. The user now explores one of the three suspicious gooseneck lamps in the room. As shown in he figure, the user finds that when moved the lamps have detents, rendered using our system, hence they can be only in one of three positions. By testing different positions, the user finds the secret combination of the lamps’ gooseneck positions that opens the virtual door.

EMS provides haptics to the Reality-Virtuality Continuum

We demonstrated how our system adds haptics to three classes of applications along the Reality-Virtuality Continuum: (1) entirely virtual objects, such as furniture with EMS friction when pushed or an EMS-based catapult game. (2) Virtual objects augmented via passive props with EMS-constraints, such as a light control panel made tangible by means of a physical cup or a balance-the-marble game with an actuated tray. (3) Augmented appliances with virtual behaviors, such as a physical thermostat dial with EMS-detents or an escape-room that repurposes lamps as levers with detents.


Pedro LopesAlexandra Ion, Sijing You and Patrick Baudisch
Adding Force Feedback to Mixed Reality Experiences and Games using Electrical Muscle Stimulation
In Proceedings of CHI'18. Fullpaper.
PDF (11Mb) | video (youtube)

Source code

Code: You can find the source code to our HoloLens applications (made with Unity3D) here.

Hardware: if you are searching for an open-source interactive EMS kit, we suggest you have a look at openEMSstim.


Press Photos

Presskit (hi-res images, etc) for this project.


We would like to thank our colleagues Arthur Silber, Lung-Pan Cheng and Sebastian Marwecki for their feedback.