Control a Skiing Game with Your Body Using AI Pose Recognition Sensor
Recap: AI penalty shootout goalkeeper challenge based on gesture recognition
1: Project Background
In the previous Project, I attempted to use AI Vision Posture and Gesture Sensor, combined with custom gesture learning functionality, to create an AI penalty shootout goalkeeper mini-game based on gesture recognition without requiring any additional wearable devices. During the hands-on process of this Project, I discovered that this Sensor supports not only custom gesture recognition but also custom pose estimation.
This sparked a new idea: if hand movements can be used to control a game, could the overall posture of the human body become an even more natural and intuitive interaction method?
So, I went a step further and used the custom pose estimation function of the AI Vision Posture and Gesture Sensor to create a motion-sensing skiing mini-game.
In this game, players don't need a keyboard, mouse, or game controller. Instead, they can directly control the virtual skiing character through body movements such as leaning left, leaning right, flipping, and accelerating. Through this Project, I also hope to further explore a more natural interaction method: making human body movements the control interface between people and the virtual world.
2: Quick Experience of the Skiing Game
Before starting the game, you first need to use the software that comes with the Sensor to learn the target poses used in the game, so that the Sensor can recognize the actions set by the player.
After completing pose learning, open the corresponding HTML game page and connect the SEN0670 AI Vision Posture and Gesture Sensor. Once in the game, select the mode you want to challenge, perform the corresponding actions according to the pose prompts on the page, and you can control the virtual skiing character through your body posture.
The entire process doesn't require wearing any additional motion capture devices. You just need to stand in front of the Sensor and complete the game operations through body movements.
Understanding the Game Actions Corresponding to Different Poses
Before officially starting the challenge, let's first understand which operations correspond to the different poses in the game.
This game mainly uses 6 human body poses:
| Human Pose | Game Action |
| Lean Left | Character glides left |
| Lean Right | Character glides right |
| Front Flip | Front flip 360° |
| Left-Side Front Flip | Left-side front flip 360° |
| Right-Side Front Flip | Right-side front flip 360° |
| Accelerate | Enter acceleration state |
Once you've mastered these poses, you can perform the corresponding actions based on the prompts in the game scene, thereby controlling the skiing character to complete the levels.
Gameplay Demonstration
After selecting the game mode you want to challenge, the player simply needs to stand within the recognition range of the Sensor. In actual use, a distance of about 2 meters from the Sensor is appropriate.
Once the game starts, strike the corresponding pose according to the prompts on the screen. The Sensor will detect the player's body movements in real time and return the currently recognized pose ID. After the program receives the pose estimation result, it converts the different poses into corresponding game control commands.
For example: Body leaning left → recognized as left → game character glides left; Body leaning right → recognized as right → game character glides right; Striking a front flip pose → recognized as power → triggers a front flip. In this way, the complete interaction process from human body movement → AI pose estimation → game command → virtual character action is achieved.
Below is the entire gameplay experience. You can download the attachment at the end of the article and follow the steps in this video to quickly experience the skiing level challenge based on pose estimation.
3: How to Implement a Skiing Level Challenge Game Based on pose estimation
To map human body movements to game operations, I first needed to solve a problem: How can game operations be designed as poses that humans can naturally perform while also being easily distinguishable by the AI vision Sensor?
After experimentation, I ultimately chose 6 custom poses with relatively obvious differences, corresponding to:
- Glide left
- Glide right
- Front flip 360°
- Left-side front flip 360°
- Right-side front flip 360°
- Accelerate
Using the custom pose learning function, the Sensor can remember these actions and assign a corresponding recognition ID to each pose. Afterwards, by simply establishing a mapping relationship between pose IDs and game actions in the program, the AI recognition results can be converted into game control commands.
What is pose estimation
pose estimation, simply put, is about enabling machines to "understand human body movements." The system first captures human images through a camera, then detects key points on the human body, and finally determines which pose the current human body is in based on the positions, distances, angles, and change relationships between different key points.
Currently, common human pose estimation methods mainly fall into the following categories:
| Recognition Method | Implementation Approach | Principle | Limitations |
| pose estimation based on computer vision | Ordinary camera + human pose models such as OpenPose, MediaPipe Pose, YOLO Pose | Captures human images through a camera, uses AI models to detect human key points, and determines actions and poses based on positional relationships | Requires running vision models on a computer, which demands a certain level of computing performance |
| pose estimation based on depth Sensor | RGB\-D cameras, depth cameras (such as Kinect, RealSense, etc.) | Obtains the human body's spatial position through depth information, achieving more accurate 3D human body structure compared to ordinary cameras, then performs pose analysis | Higher hardware cost, larger device size, and certain limitations on usage environment |
| pose estimation based on inertial Sensor | IMU, accelerometers, gyroscopes, wearable devices | Analyzes human pose and movements by detecting acceleration, angular velocity, and directional changes during human motion | Usually requires wearing devices, limiting movement range and user experience |
Unlike solutions that require running pose estimation models on a computer, the SEN0670 integrates AI vision algorithms directly within the Sensor, allowing it to complete human pose detection and recognition on its own. The computer doesn't need to deploy complex vision models separately; it only needs to receive the recognition results returned by the Sensor and complete the subsequent game logic based on those results.
More importantly, the SEN0670 supports custom pose learning. Users can have the Sensor learn different human movements based on their own Project needs and establish corresponding recognition results for each action. This also makes it highly suitable for creating Project projects that require human movements as input, such as motion-sensing games, smart control, and robot interaction. Therefore, this game uses the SEN0670 AI Vision Posture and Gesture Sensor:https://www.dfrobot.com/product-3135.html
How to Use the AI Vision Posture and Gesture Sensor to Learn Poses
When designing the control method for the skiing game, my first consideration was: How can the movement, trick, and acceleration operations in the game be converted into human movements that players can easily perform while also being easily distinguishable by the Sensor?
In the previous AI Gesture Goalkeeper Project, I used the method of the same gesture + joint point coordinate changes, calculating coordinate changes in different directions to determine the ball-saving direction. This time, I directly used the SEN0670's custom pose learning function to learn 6 clearly distinct human poses.
Then, I mapped the different pose IDs returned by the Sensor to different actions in the game: Pose → ID → Game Command → Character Action
This achieves a direct conversion from human poses to game control commands.
Step 1: Open the Official Learning Tool
First, download the official software that comes with the SEN0670 AI Vision Posture and Gesture Sensor, which is Attachment 1 at the end of the article.
After extracting, double-click the .exe file to launch the software. Connect the computer to the SEN0670 using a USB cable, and the software will automatically detect the serial port corresponding to the Sensor. Select the appropriate serial port and open it.
Step 2: Enter Pose Mode and Learn Custom Poses
Select pose estimation mode in the software. Then stand about 2 meters away from the Sensor and strike a custom pose. Here, we'll use the "lean left" pose as an example. After striking the pose, you can see the human key points and the human detection box in the left-side view. Before learning is completed, the human key points will be displayed in green.
After striking the "lean left" pose, click the learning button and hold this pose still for about 3 seconds. When the color of the body joint points changes from green to another color, learning has succeeded. You can then customize the gesture name and name it "left". The real-time data display below shows the coordinates of each joint point.
Using the same method, you can continue learning: right, power, left360, right360, fast, ultimately completing the learning of all 6 game control poses.
Below is the specific pose learning process. You can also refer to the official pose learning tutorial: https://wiki.dfrobot.com/sen0670/docs/23979
Implementation of the Pose Skiing Game Program
After completing pose learning, the next step is to enable the web game to receive the pose estimation results returned by the SEN0670. The complete pose skiing game program can be found in Attachment 2 at the end of the article. Open the HTML file to view the complete source code.
The entire pose skiing game program code can be divided into four parts: Access Layer, Perception Layer, Decision Layer, and Presentation Layer.
Access Layer: Obtaining Sensor Data
In the access layer: First, the program needs to establish communication with the SEN0670 and receive the data returned by the Sensor. After the Sensor recognizes a human pose, it returns the corresponding pose ID. Once the program obtains this ID, it can proceed to the next step of decision-making.
Perception Layer: Obtaining the Current Pose
For example, let's assume the pose ID currently recognized by the Sensor is:
let poseId = detectedPoseId;
Here, detectedPoseId represents the human pose currently recognized by the Sensor. However, for the game program, directly using numeric IDs isn't intuitive. Therefore, I created a pose ID → game action name mapping table.
// Create a mapping between SEN0670 pose IDs and game actions
// Each ID corresponds to a learned human body pose
const POSE_MAP = new Map([
[1, "left"], // ID 1: Move left
[2, "right"], // ID 2: Move right
[3, "power"], // ID 3: Front flip 360°
[4, "left360"], // ID 4: Left-side front flip 360°
[5, "right360"], // ID 5: Right-side front flip 360°
[6, "fast"] // ID 6: Accelerate
]);
// Get the current pose ID detected by SEN0670
let poseId = detectedPoseId;
// Find the corresponding game action based on the pose ID
// Example: ID 1 → left, ID 6 → fast
let currentAction = POSE_MAP.get(poseId);
Through this mapping table, the numeric IDs returned by the Sensor are converted into more intuitive game action names. For example:
ID 1 → left → glide left; ID 2 → right → glide right; ID 6 → fast → accelerate
This step essentially serves as the bridge connecting AI perception results with game control logic.
Decision Layer: Executing Game Logic Based on Poses
After obtaining the current action name, the program can further determine what operation the player wants to perform.
For example, left and right gliding can be implemented with the following code:
// ===== Left and right steering control =====
// When a left or right pose is detected and the confidence meets the threshold
if(steeringName && conf >= 35){
// left → move left
// right → move right
Game.setSteer(
steeringName === 'left' ? 1 : -1
);
}
Here, the program first checks whether a valid steering pose has been detected, while also requiring the recognition confidence conf to reach a certain threshold.
When the recognition result is left: the character moves left.
Game.setSteer(1);
When the recognition result is right: the character moves right. In this way, the player's left and right body movements are converted into left and right movements of the game character.
Game.setSteer(-1);
In this manner, different human poses correspond to different game behaviors. Ultimately, the entire game forms a complete data chain like this: This is also the core implementation concept of the entire motion-sensing skiing game.
The player performs a corresponding pose
↓
The AI sensor recognizes the human pose
↓
Returns the pose ID
↓
The pose ID is mapped to a game action
↓
The program checks confidence and action state
↓
Executes the game control logic
↓
The virtual skiing character performs the corresponding action
Core Program Analysis
The core program first reads the pose ID returned by the SEN0670, then uses the pre-established pose mapping table to convert different poses into corresponding game control commands. Among these, IDs 1 and 2 are used to control the character gliding left and right respectively, IDs 3–5 correspond to different flip actions, and ID 6 is used to trigger acceleration. This achieves a direct mapping from "self-learned human poses" to "virtual character actions."
// Create a mapping between SEN0670 pose IDs and game actions
// Each ID corresponds to a learned human body pose
const POSE_MAP = new Map([
[1, "left"], // ID 1: Move left
[2, "right"], // ID 2: Move right
[3, "power"], // ID 3: Front flip 360°
[4, "left360"], // ID 4: Left-side front flip 360°
[5, "right360"], // ID 5: Right-side front flip 360°
[6, "fast"] // ID 6: Accelerate
]);
// Get the current pose ID detected by SEN0670
let poseId = detectedPoseId;
// Find the corresponding game action based on the pose ID
// Example: ID 1 → left, ID 6 → fast
let currentAction = POSE_MAP.get(poseId);
When the player strikes the corresponding pose and the Sensor reads it, the program will enter the corresponding game control logic based on the different actions. Among these, left and right are used to change the skiing character's movement direction, fast is used to control the acceleration state, and power, left360, and right360 enter the trick action decision logic. This completes the conversion from "pose estimation results" to "actual character actions."
// ===== Left and right steering control =====
// When a left or right pose is detected and the confidence meets the requirement
if(steeringName && conf >= 35){
// left → steer the skiing character to the left
// right → steer the skiing character to the right
Game.setSteer(
steeringName === 'left' ? 1 : -1
);
}
// ===== Acceleration control =====
// Enable acceleration when the "fast" pose is detected
if(name === 'fast' && conf >= 40){
Game.setFast(true);
}else{
Game.setFast(false);
}
// ===== Trick action control =====
// Three poses that can trigger aerial tricks
const trickPose = [
'power',
'left360',
'right360'
];
// Execute the corresponding trick action when a valid pose is detected
if(trickPose.includes(name)){
Game.performTrick(name);
}
4: Summary
Through this Project, I gained a more intuitive understanding of the application of pose estimation in motion-sensing interaction. The SEN0670 AI Vision Posture and Gesture Sensor can not only recognize different human poses but also build its own action categories through custom learning. In this skiing game, I designed 6 poses based on the actual gameplay — lean left, lean right, front flip, left-side front flip, right-side front flip, and accelerate — and mapped the pose IDs returned by the Sensor to corresponding game control commands. Combined with stability checks, debouncing, and action trigger logic, I achieved real-time conversion from real body movements to virtual skiing character actions.
From gesture recognition to pose estimation, I've continuously learned new things about interaction methods using AI Vision Posture and Gesture. In the future, if more pose data and application scenarios can be combined, human body movements could become the control method for even more devices and systems, such as motion-sensing games, smart device control, and robot interaction. Perhaps the way humans and machines communicate in the future won't necessarily rely on keyboards, mice, or touchscreens, but can be accomplished through more natural movements.
5: Attachments
Attachment 1: SEN0670 AI Vision Posture and Gesture Sensor companion learning tool
Attachment 2: Pose skiing game HTML file
