me handle: As shown in the figure above, DualPanto’s applications generally use the me handle to represent the user’s avatar in the virtual world.

(a) A user moves around and explores the virtual environment, such as a series of rooms in a shooter game, by moving the me handle. The device uses a direct 1:1 mapping, similar to the mapping of a touch screen; returning the handle to the same location in the physical world returns the avatar to same location in the virtual world.

(b) The knob itself allows users to rotate their avatar. The pointed end of the handle represents the direction the avatar faces.

(c) When the user pushes against a wall, the me handle resists by providing force feedback.

(d) The me handle plays back haptic icons, for example, a short knockback when the player is hit by a projectile.

it handle: As shown in the figure above, DualPanto applications use the it handle to render one selected moving object, such as the opponent in a first-person shooter.

(a) If the object represented by the it handle moves, so does the handle. The handle is actuated only by the device, i.e., it will resist users trying to move it. By allowing the it handle to guide their hand, users can track the associated object, e.g., feel how the opponent moves and obtain a sense of where it is currently located with respect to the user’s avatar.

(b) At the same time, the actuated knob conveys to the user what direction it is facing. The it handle may also display haptic icons, such as an impulse when the opponent is shot.

We conducted a study to explore whether DualPanto enables blind users to interact with motion in spatial virtual environments. Participants played the 1-on-1 soccer game and reported on their experience. We recruited six participants (ages 14-45, 1 woman) who had visual impairments. Participants’ self-reported level of vision was between completely blind and 16% in one eye. Participants played one or more matches against a trained player (one of the researchers) for 3-10 minutes and then provided their feedback in an interview. 

DualPanto enabled spatial awareness of the game state

DualPanto enabled continuous interaction with moving objects. All six participants tracked their opponent and responded to their location, either to pass to them during practice (P2-6) or to shoot past them during gameplay (P1,2,4-6): “yes I could definitely move the player to the ball and everything” (P4). Participants could detect opponents’ actions across the field: P3 remarked, “that was on my goal” when our trained player scored, and P5 noted when our trained player scored on his own net: “own goal?”

Inclusive multiplayer experiences

Participants were surprised and excited about the prospect of multiplayer experiences: “[multiplayer] is really significant, I have fun crushing people” (P3), “ah, cool that there are 2 players” (P4). Participants also requested more multiplayer capabilities: “it would be interesting to have more than 2 players” (P3).

DualPanto as an interactive audio-haptic gaming console

Audio provided context and engaged: “I like…that you can instantly hear sounds, I really like it” (P2). More sounds could improve the experience: “it could be better if more sounds were added” (P1), suggesting cheers and gasps from the crowd and sound effects from blind soccer.

Haptic feedback complemented the context provided by audio. Participants thought the combination could work for different games: “you could explore levels with force feedback and show the next enemy with the it-handle, story is about audio anyways, so yeah I could imagine this working well for role play games” (P3). Adding haptics improved interactivity: “[another game] has different story lines that you can go into, but you always have to wait” (P4).