Prof. Dr. Patrick Baudisch

Muscle-plotter: an Interactive System based on EMS that Produces Spatial Output

Existing interactive EMS systems output a single 1D output variable. Since subsequent output overwrites earlier output, users never see more than a single value. With muscle-plotter we explore how to create more expressive EMS-based systems. Muscle-plotter achieves this by persisting EMS output, allowing the system to build up a larger whole. More specifically, (1) muscle-plotter spreads out the 1D signal produced by EMS over a 2D surface by steer-ing the user's wrist, while the user drags their hand across the surface. Rather than repeatedly updating a single value, this renders many values into curves. (2) By adding the pen, we persist this signal, allowing the system to build up a larger display, which in turn enables longer and more meaningful interactions.

Muscle-plotter is a system that provides users with input and output access to a computer system while on the go. Using pen-on-paper interaction, muscle-plotter allows users to engage in cognitively demanding activities, such as writing math. Users write formulas using a pen and the system responds by making the users’ hand draw charts and widgets. While Anoto technology in the pen tracks users’ input, muscle-plotter uses electrical muscle stimulation (EMS) to steer the user’s wrist so as to plot charts, fit lines through data points, find data points of interest, or fill in forms. We demonstrate the system at the example of six simple applications, including a wind tunnel simulator

Walkthrough: testing the aerodynamics of a car with only pen and paper

We think of muscle-plotter as a tool for mobile sense­making in that it allows users to interact (input and output) with an intelligent backend.

Here, a car designer is iterating on the body of a new car, sketching it and analyzing implications of its design on the car’s aerodynamics. The designer wrote “wind tunnel” onto the paper and has drawn crop marks around the car. Since the user does so using a pen that offers built-in tracking (Anoto), muscle-plotter “sees” this input. It recognizes the handwriting using a handwriting recognizer (Tesseract ) and forwards its output to a wind tunnel simulator running in our custom backend. The system computes the wind velocity field and makes it available to the pen frontend. Muscle-plotter plots the resulting streamlines by actuating the user's wrist. This way, the designer sketches two different car's and compares the results of the wind simulation, entirely on pen and a paper. 

Surprisingly, the hatchback’s streamlines look straighter that the sedan’s, this suggests an aerodynamic advantage. The designer decides to drill down by plotting aerodynamic profiles of the tail winds of the two car designs. The designer now sketches a blank coordinate system, writes “plot sedan”, sets down the pen down left of the coordinate system, and drags the pen into it. Muscle-plotter responds by plotting wind speeds across the cross section into the coordinate system. For comparison, the user now writes “plot hatchback” and plots the wind speed function for the hatchback into the same coordinate system.

Finally, this designer wonders whether the improved turbulences will really manifest themselves in lower wind resistance; hence better gas mileage. Hence, using muscle-plotter's scale widget, it is possible to compare drag coefficient of both car designs. 

Application scenarios

To demonstrate muscle-plotter's to render significantly more complex data than previous EMS-based systems, we implemented five additional applications: an RC circuit simulator, equation plotter, I/O forms, a simple statistics plotter and a optical lenses simulator.

Implementation (open-source)

We believe post-wearable interfaces such as muscle-plotter, pose-io and affordance++ will redefine the way we conceptualize the boundary between user and the computer interface.

To inspire other researchers to build on these ideas we provide an open-source implementation of muscle-plotterhere (download code as zip here). 


Lopes, P., Yüksel, D., Guimbretière, F., and Baudisch, P.
Muscle-plotter: an Interactive System based on Electrical Muscle Stimulation that Produces Spatial Output
In Proc. UIST'16. Full Paper.

 PDF (6 MB) |  Video (63 MB) |  Source code (zip file 5 MB)

Press Photos


We thank our colleagues at the HPI, in particular Ludwig Wall and Alexandra Ion, for their assistance in piloting our prototypes.