Problem

Long and Tedious Physical Recovery Process

Physical therapies on body and hand often involves long, repetitive exercises that can feel tedious and demotivating, making the recovery process less engaging and enjoyable for patients.

Limited Engagement in Gaming for Children

Children love superheroes, but merely using keyboard and controller in PC games doesn’t let them immerse themselves in their favorite characters' iconic moves, limiting their engagement with superhero-themed games. Usage of VR or AR kits could be useful, but they require additional costs and are served for specific game platforms.

Concept of the System

To make repetitive body movements more engaging, we introduced fun gameplay into the process. I developed the concept of a "Gesture Recorder," a software system that uses a webcam to capture hand shapes and full-body movements, converting them into a structured data format for storage and processing. When playing compatible games, a "Gesture Detector" layer runs in real-time, analyzing the webcam feed. If a recorded gesture is detected, the system triggers the corresponding game inputs, making it feel as though the gesture directly controls the game.

Process of Design & Implementation

Hand Shapes Recording & Detection

Together with Abid Ali, we implemented hand shape detection based on the Google MediaPipe. The algorithm captures coordinates between hand joints and normalizes them as ratios of the shape.

By comparing these ratios instead of absolute coordinates, the detection algorithm can accurately recognize the hand shapes with rapid response, regardless of orientation, rotation, size, or skin tone. This allows a hand shape stored by one user to be used by anyone.

Body Movements Recording & Detection

Using the body detection from MediaPipe, which provides coordinates of the body "skeleton", the recorder captures up to 2 seconds of dynamic movement. The recorded motion path is then processed and stored in a specially designed data format.

By comparing the stored body movement data with live performance data using the Dynamic Time Warping algorithm, the software could identify the most similar movement being performed.

Client-side User Interface

The two main segments of the app—recording and detection—each come with dedicated client-side applications coded in Microsoft Foundation Class in C++. I designed and implemented both applications, streamlining the process of recording, loading and configuring gestures, as well as additional option settings used for the game key bindings.

Outcome

The final system was tested in the Code Ninjas children school in London, where kids played superhero action games using gesture-based controls.

The system was also showcased at the Microsoft Reactor and Great Ormond Street Hospital in London, where guests and investors explored its features through gesture-based games, and how it could serve physical therapy.

We explored a variety of game genres and recorded dedicated gestures for each. Playing these familiar games in a completely new way was not only fun but also a great means of workout.