Summary: We present a mobile system that enhances mixed reality experiences and games with force feedback by means of electrical muscle stimulation (EMS). The benefit of this fully mobile setup is that it adds physical forces while keeping the users’ hands free to interact unencumbered—not only with virtual objects, but also with physical objects, such as props and appliances.

In this research project, we propose adding force feedback in MR games and experiences using electrical muscle stimulation instead—an approach that is small, light, and fits under the user’s clothing. The figure below illustrates this at the example of our Mixed Reality balance marble game using a physical tray as a game prop, which our approach augments via EMS-based force feedback. Using EMS our simple game is able to, for instance, render shifts in the board’s center of gravity, caused by the “weight” of the moving virtual marble, by actuating the user’s arms towards the heavier side.

1. A pure virtual experience (Shooting the catapult): The figure below shows our simple MR game featuring a virtual catapult that appears in the user’s physical surroundings. (a) The user arms the catapult by pulling the catapult’s bucket backwards. As the user pulls the bucket backwards, our system provides force feedback that simulates the increasing tension of the catapult—this force feedback also serves as an indicator for the user on how far the projectile can be expected to fly. Our system achieves this by stimulating the antagonist muscle (triceps) proportionally to the catapult’s arm angle. (b) When the user opens the hand (which terminates the HoloLens gesture) the catapult releases. Our system abruptly ceases to stimulate the triceps ceases and the remainder force in the user’s biceps, which now does not have a counter force, creating a sensation of actual recoil.

2. Augmented physical props: balance marble game. Figure 1 (see top) depicted our MR marble maze, which was in fact inspired by that of Ohan and Feiner and complemented with EMS-based force feedback for added realism. As previously explained, our marble game renders the shift in gravity caused by the moving marble by actuating the user’s arms towards the heavier side. Note that even if the marble perfectly balanced on the center of the tray we render a constant pull down of the user’s arms (triceps) to represent the marble’s weight under gravity. Furthermore, we render a number of extra haptic effects: (1) when the ball collides with any of the maze walls, we render a short bump in the user’s triceps, depending on the ball’s velocity vector; (2) when the marble falls out of the tray in one of the openings in the obstacle walls, the user feels the relief in the weight; and, (3) when the game starts, the marble falls from the sky onto the tray and we render this “extra weight” by quickly pulling down the user’s arms, as depicted in the Figure below.

3. Augmented appliances (escape room experience). We implemented a simple Escape Room [14] in Mixed Reality. In the traditional version of these experiences, users must first find the solution to a puzzle, and only then they can escape the current room. The Figure below illustrates the user solving this room’s puzzle. The user sees a virtual message on the wall next to the virtual door that provides a clue to the puzzle: “the lights will illuminate your path, but only in one special way”. The user now explores one of the three suspicious gooseneck lamps in the room. As shown in he figure, the user finds that when moved the lamps have detents, rendered using our system, hence they can be only in one of three positions. By testing different positions, the user finds the secret combination of the lamps’ gooseneck positions that opens the virtual door.

We demonstrated how our system adds haptics to three classes of applications along the Reality-Virtuality Continuum: (1) entirely virtual objects, such as furniture with EMS friction when pushed or an EMS-based catapult game. (2) Virtual objects augmented via passive props with EMS-constraints, such as a light control panel made tangible by means of a physical cup or a balance-the-marble game with an actuated tray. (3) Augmented appliances with virtual behaviors, such as a physical thermostat dial with EMS-detents or an escape-room that repurposes lamps as levers with detents.