Hand-gesture interface and

physical therapy games




16 weeks, 2018


2 Engineers

3 Designers



acquired by Facebook in 2019


Electromyography wearable device,

Product ver.2017



Adobe Suite

G Suite


  1. Developed competitive analysis to prioritize features that leverage the strengths of CTRL-labs's 2017 neural-signal device.

  2. Created product backlog and prioritization methods

  3. Organized 70+ user testing sessions and analyzed feedback

  4. Wrote design documentation for CTRL-labs's on-boarding developers


Three physical therapy games that showcase the strengths of client's Machine-Learning wearable device. 

Disclaimer: The device used in this project is an early prototype from 2017, which has no indication of CTRL-labs's current product.

Snowboarding is an adventure game. Users define their own subtle hand or finger movements to steer the snowboard through the gates and avoid the Yetis. 

Cake Master is a cake-making game. Users use different forces to squeeze cream in 3 cake layers. 

Sound Mosaic is a synthesizer and visualizer experience, integrating the device with Kinect. Users make micro hand gestures to add sound filters to the background ambience.


Early prototype with technical issues

(e.g: high latency)

No documentation

My team of 6 only had 1 Machine-Learning focused programmer

[2] Product backlog and tasks prioritization

I created Product backlog (sample below) to drive the design directions for our prototypes.

After spending 1 month to research and building simple demos, we arrived at 3 use cases that showcase the device's strengths: continuous gesture recognition, force detection, and micro-gesture recognition. We spent each month designing an interactive game for each use case, following a 1-week Scrum cycle. The product roadmap was updated weekly after every build and user test. User stories were categorized into sections of priorities and placed into a vertical slice.


For this project, game mechanics were not as prioritized as design for usability and giving users a sense of control. 

[3] User interviews and testing

I organized weekly playtests with adult gamers, non-gamers and created success metrics for each. I documented all user feedback in Usability Testing Report 1  Usability Testing Report 2​  and many similar reports (images below). 

Recognizing individual fingers

Walking mechanism

Rotating gestures

Squeezing different forces

[4] Technical design document

Since our team was the first external developers for CTRL-labs and they did not have any documentation on their past experiments, I took it upon myself to write a Technical Design Document (images below). CTRL-labs later told me that they have used this document for their on-boarding engineers.

[1] What is a "done" task?

Delayed responses from the device made it challenging for me to understand whether my users' negative feedback was caused by our design or by the device's technical constraints. Our team struggled to define a task being "done." We learned to break down our goals into smaller milestones per week and record our progress to show to stakeholders. 

[2] Product management lessons
  • Have a clear product backlog and priorities so the team can focus on what's most important. Revisit it weekly.

  • Visually display thought process and prototyping goals in the project room to keep everyone's vision aligned.

  • Paper storyboarding does not always communicate user interactions well.

  • I learned how to communicate technical concepts, such as machine learning, in a non-technical way that artists, designers, and users would be able to understand.

Designed by Tera Nguyen