Hasso-Plattner-Institut
Prof. Dr. Patrick Baudisch
 

DualPanto: A Haptic Device that Enables Blind Users to Continuously Interact with Virtual Worlds

Oliver Schneider, Jotaro Shigeyama, Robert Kovacs, Thijs Jan Roumen, Sebastian Marwecki, Nico Boeckhoff, Daniel-Amadeus Gloeckner, Jonas Bounama and Patrick Baudisch
 

DualPanto is a haptic device that enables blind users to track moving objects while acting in a virtual world.

As shown in above figure (b), the device features two handles. Users interact with DualPanto by actively moving the me handle with one hand and passively holding on to the it handle with the other. DualPanto applications generally use the me handle to represent the user’s avatar in the virtual world and theit handle to represent some other moving entity, such as the opponent in a soccer game.

Interacting with DualPanto

me handle: As shown in the figure above, DualPanto’s applications generally use the me handle to represent the user’s avatar in the virtual world.

(a) A user moves around and explores the virtual environment, such as a series of rooms in a shooter game, by moving the me handle. The device uses a direct 1:1 mapping, similar to the mapping of a touch screen; returning the handle to the same location in the physical world returns the avatar to same location in the virtual world.

(b) The knob itself allows users to rotate their avatar. The pointed end of the handle represents the direction the avatar faces.

(c) When the user pushes against a wall, the me handle resists by providing force feedback.

(d) The me handle plays back haptic icons, for example, a short knockback when the player is hit by a projectile.

 

it handle: As shown in the figure above, DualPanto applications use the it handle to render one selected moving object, such as the opponent in a first-person shooter.

(a) If the object represented by the it handle moves, so does the handle. The handle is actuated only by the device, i.e., it will resist users trying to move it. By allowing the it handle to guide their hand, users can track the associated object, e.g., feel how the opponent moves and obtain a sense of where it is currently located with respect to the user’s avatar.

(b) At the same time, the actuated knob conveys to the user what direction it is facing. The it handle may also display haptic icons, such as an impulse when the opponent is shot.

Application

Shooter is a first-person survival shooter game. In this game, the player must fight a horde of zombie bunnies by shooting at each zombie as it approaches while avoiding their touch.

Implementation

Pantographs

DualPanto implements the haptic pantograph design. We chose this design because it is planar and therefore appropriate for a single registered workspace without collisions; the two pantographs operate independently, unlike constructions where two pantographs control a single end-effector. We based our implementation on the open-source Haply platform.

The two pantographs are mounted individually onto a tripod using a 3D-printed bracket. We typically weight the tripod’s legs to improve stability. The linkages are made of 3 mm laser-cut aluminum, which is rigid enough to avoid deflection. This maintains the vertical distance between the two handles.

Handles

The me and it handles are mounted to the arms of the pantographs by a 3D-printed bracket. The me handle contains a small motor (Pololu, 6V) with a gear ratio of 10:1, which is easily back-drivable for user input. The it handle has a higher gear ratio of 75:1 to provide enough force for system output.

To represent direction, we mounted a 3D-printed, asymmetric tip onto each of DualPanto’s handles. After experimenting with several designs, the “flattened teardrop” design (8x10x15 mm) performed best, in that holding this design between index finger and thumb clearly conveys its orientation any time.

User Study

We conducted a study to explore whether DualPanto enables blind users to interact with motion in spatial virtual environments. Participants played the 1-on-1 soccer game and reported on their experience. We recruited six participants (ages 14-45, 1 woman) who had visual impairments. Participants’ self-reported level of vision was between completely blind and 16% in one eye. Participants played one or more matches against a trained player (one of the researchers) for 3-10 minutes and then provided their feedback in an interview. 

DualPanto enabled spatial awareness of the game state

DualPanto enabled continuous interaction with moving objects. All six participants tracked their opponent and responded to their location, either to pass to them during practice (P2-6) or to shoot past them during gameplay (P1,2,4-6): “yes I could definitely move the player to the ball and everything” (P4). Participants could detect opponents’ actions across the field: P3 remarked, “that was on my goal” when our trained player scored, and P5 noted when our trained player scored on his own net: “own goal?”

Inclusive multiplayer experiences

Participants were surprised and excited about the prospect of multiplayer experiences: “[multiplayer] is really significant, I have fun crushing people” (P3), “ah, cool that there are 2 players” (P4). Participants also requested more multiplayer capabilities: “it would be interesting to have more than 2 players” (P3).

DualPanto as an interactive audio-haptic gaming console

Audio provided context and engaged: “I like…that you can instantly hear sounds, I really like it” (P2). More sounds could improve the experience: “it could be better if more sounds were added” (P1), suggesting cheers and gasps from the crowd and sound effects from blind soccer.

Haptic feedback complemented the context provided by audio. Participants thought the combination could work for different games: “you could explore levels with force feedback and show the next enemy with the it-handle, story is about audio anyways, so yeah I could imagine this working well for role play games” (P3). Adding haptics improved interactivity: “[another game] has different story lines that you can go into, but you always have to wait” (P4).

Conclusion

We presented DualPanto, a haptic device that enables blind users to interact with spatial virtual environments that contain objects moving in real-time, as is the case in sports or shooter games. The key feature is that its two haptic in/output handles are spatially registeredto each other, which enables blind players to navigate, track, dodge, and aim. In our user study, blind participants reported very high enjoyment when playing a soccer game (6.5/7).