No Controller — Save Penalties with Hand Gestures! AI Goalkeeper Challenge
1: Project Background
Once while playing a soccer game with a friend, I noticed something interesting: even though penalty kicks and saves are the most intense moments in the game, the actual controls still boiled down to pressing direction keys or moving a joystick. At that moment, I thought — what if the goalkeeper's movements weren't triggered by button presses, but directly controlled by the player's own physical actions? Wouldn't that be more immersive?
Coincidentally, I had an AI vision gesture and posture Sensor that supports gesture recognition and hand keypoint detection, so I started experimenting with turning "human movements" directly into "game commands." Players don't need to memorize complex button combinations or hold a game controller — they just open their palm and change its orientation based on the direction the ball is flying, and the goalkeeper on screen makes the corresponding save. This way, the penalty kick game is no longer just "watching the character move," but becomes "I see the ball coming, and I physically move to make the save." It was from this idea that I created this AI penalty kick goalkeeper mini-game based on gesture recognition.
2: Quick Experience of the Save Game
The gameplay is very simple — players don't need a keyboard, mouse, or game controller, and certainly don't need complex program or computationally demanding Board. First, open the software that comes with the Sensor (Attachment 1) on your computer to train the save gesture, then open my HTML link webpage (Attachment 2), connect to the gesture recognitionSensor, choose the game mode you want to play, and you can experience the immersive feeling of a penalty kick.
Understanding the Correspondence Between Save Direction and Gesture
Before entering the game, you can first learn how palm orientation corresponds to the goalkeeper's save position: when the palm tilts left, the goalkeeper dives to the left; when the palm stays vertical, the goalkeeper defends the center; when the palm tilts right, the goalkeeper dives to the right.
After choosing a mode you want to play, simply place your hand in front of the gesture Sensor and open one palm to participate in the penalty kick goalkeeping challenge. The left side displays the Sensor real-time feed and hand keypoints, so players can see whether their palm is within recognition range and what direction their palm is currently judged as. The bottom-left corner also provides real-time feedback on the corresponding action based on your palm orientation.
Game Vedio
Below is the full gameplay experience. You can download the attachment at the end of this article and follow the steps in this video to quickly experience the gesture-based penalty kick goalkeeper game.
3: How to Implement a Save Game Based on gesture recognition
What is gesture recognition
gesture recognition is the technology that allows machines to determine what hand gesture a person is currently making through a camera or Sensor. The system first captures hand images or keypoints, then determines which gesture it belongs to based on the motion characteristics.
Common gesture recognition approaches:
| Recognition Method | Implementation Approach | Principle |
| gesture recognition based on Sensor | Data gloves, flex Sensors, IMU, EMG Sensors, etc. | Directly collect finger bending, hand posture, acceleration, or muscle electrical signals, then determine the gesture based on this data |
| gesture recognition based on computer vision | Regular camera + OpenCV, MediaPipe, AI models, AI vision Sensors, etc. | Capture hand images through a camera, then use image processing or AI algorithms to recognize gestures, keypoints, and orientation |
From the comparison above, it's clear that both Sensor-based and computer vision-based gesture recognition solutions have their limitations. I wondered if there was a Sensor that was relatively affordable and didn't require setting up a vision environment and processing power on the computer — that's when I noticed the SEN0670 AI Vision Gesture and Posture
It belongs to the computer vision-based gesture recognition solution: it captures images through a camera and then uses AI vision algorithms to analyze human hand movements. The difference is that it integrates gesture recognition, hand keypoint detection, and other functions directly into the Sensor itself, combining the advantages of both approaches. This means the computer doesn't need to build a vision recognition environment from scratch — the local AI processing makes it much more convenient to use. Additionally, this Sensor comes with three recognition modes: custom gesture recognition (supports up to 8), custom pose estimation (supports up to 8), and fixed gesture recognition (13 types). Learners can easily get started, and the gesture recognition process outputs coordinate information for hand joint keypoints.
How to Use the Gesture Sensor to Train Gestures
When I started designing the game control scheme, my first consideration was: how can I establish a natural correspondence between the player's hand movements and the goalkeeper's save direction?
Initially, I considered training multiple separate gestures, such as using different gestures to correspond to the three save directions: left, center, and right. This approach was relatively straightforward to implement, but after actual testing, I found that players had to deliberately switch between different gestures while simultaneously tracking the ball's movement direction — the correspondence between actions and real saves didn't feel natural. In reality, actual save movements typically involve opening the palm, with the main difference being the palm orientation and movement direction.
Therefore, I shifted my approach from "recognizing multiple different gestures" to "recognizing one base gesture, then analyzing the directional changes of that gesture."
After choosing "open palm" as the base gesture, I found that even when tilting or swinging the palm in different directions, the model still recognized it as the same "open palm" gesture. This way, I could keep the gesture category constant and then further analyze the palm orientation changes, mapping them to three different save directions: left, center, or right.
After learning more about the DFRobot SEN0670 AI Vision Gesture and Posture Sensor, I discovered that it can not only recognize trained gestures but also obtain coordinate information for each hand keypoint during recognition. This gave me an idea: instead of training multiple different gestures, I could use the coordinate relationships between these hand keypoints to calculate the palm's tilt direction, thereby establishing a "palm posture — save direction" correspondence.
So in the end, I chose "open palm" as the base control gesture. When the Sensor recognizes an open palm, the program first confirms that the player has entered an active control state; it then reads the palm keypoint coordinates, determines the palm's tilt direction by analyzing the positional relationships between keypoints, and maps this to one of three save directions: left, center, or right. With this design, players don't need to learn or switch between different control gestures — they simply open their palm like a real goalkeeper and naturally move toward the corresponding direction to control the game.
1. download the official software that comes with the SEN0670 AI Vision Gesture and Posture Sensor (Attachment 1). After extracting, double-click the .exe file. Once opened, connect the computer to the Sensor — it will automatically detect the device port. Then open the serial port to see the camera feed.
2. Select gesture recognition mode from the modes. Position yourself about 50–60 cm away from the Sensor and make the "open palm" gesture. The left screen will display the joint keypoints and bounding box of the current gesture — they appear green before training.
3. After making the "open palm" gesture, click the train button and hold your palm still for about 3 seconds. When the finger joint keypoints change from green to another color, training is successful. You can then customize the gesture name — I named it "Plam" as the base gesture for subsequent save recognition. The real-time data display below shows the coordinates of each joint keypoint.
Below is the specific gesture training process. You can also refer to the official gesture training tutorial: https://wiki.dfrobot.com/sen0670/docs/23979
Gesture Save Game Program Implementation
The complete gesture save game can be found in Attachment 2 at the end of this article. You can view the HTML file's source code, as shown below.
The entire AI gesture goalkeeper program code can be divided into four parts: access layer, perception layer, decision layer, and presentation layer.
Implementing Save Direction Determination Based on Gesture Keypoints
In the game, after recognizing the "open palm" gesture, the program further reads the 21 hand keypoints output by the SEN0670 AI Vision Gesture and Posture Sensor. Keypoint 0 is selected as the wrist position, and keypoint 12 is selected as the middle fingertip position.
The core program establishes a 2D direction vector from "wrist → middle fingertip" and calculates the angle of this vector. Based on the angle range, the palm direction is then mapped to three save commands: left, center, and right.
// Extract keypoint 12, which represents the middle fingertip position
const middleTip = p[12];
// Calculate the 2D direction vector from the wrist point to the middle fingertip point
// dx and dy represent the horizontal and vertical components of the palm orientation
let dx = middleTip[0] - wrist[0];
let dy = -(middleTip[1] - wrist[1]);
The program divides the angle into three ranges: 22°–70° is determined as right, 70°–110° as center, and 110°–158° as left. For example, when the palm is basically vertical, the direction vector is close to 90°, and the program outputs "center"; when the palm tilts right, the angle approaches 45°, and it's determined as "right"; when tilting left, it approaches 135°, and it's determined as "left."
/* Direction control debounce processing: right / center / left */
// Define the reference angles for different palm directions:
// right = 45°, center = 90°, left = 135°
const PALM_ZONE_CENTER = {right:45, center:90, left:135};
/**
* Calculate the minimum difference between two angles (0~180 degrees)
*
* @param {number} a First angle
* @param {number} b Second angle
* @returns {number} The minimum angular difference
*/
function angleDiff(a,b){
// Calculate the absolute difference and normalize it within 360 degrees
let d = Math.abs(a - b) % 360;
// If the difference is larger than 180 degrees,
// use the shorter angle distance in circular angle space
return d > 180 ? 360 - d : d;
}
/**
* Determine the current palm direction zone based on the input angle
*
* Three wide upper-half zones are defined:
* right / center / left
*
* Hysteresis is applied to keep the detected direction stable
* and prevent rapid switching between adjacent zones.
*
* @param {number} angle Current detected palm angle
* @param {string|null} lastZone Previous detected zone for hysteresis control
* @returns {string|null}
* "right" - right direction
* "center" - center direction
* "left" - left direction
* null - outside valid range
*/
function palmZoneFromAngle(angle,lastZone=null){
// ===== Hysteresis logic =====
// If the previous zone remains the same,
// use a wider angle range to maintain stability
// and avoid frequent zone switching.
// Previous zone is right:
// keep right zone when angle is between 18° and 78°
if(lastZone==='right' && angle>=18 && angle<=78)
return 'right';
// Previous zone is center:
// keep center zone when angle is between 62° and 118°
if(lastZone==='center' && angle>=62 && angle<=118)
return 'center';
// Previous zone is left:
// keep left zone when angle is between 102° and 162°
if(lastZone==='left' && angle>=102 && angle<=162)
return 'left';
// ===== Initial detection / zone switching =====
// Use narrower thresholds for switching to a new direction.
// The palm must enter a new range before changing state.
// Right zone: 22° ~ 70°
if(angle>=22 && angle<70)
return 'right';
// Center zone: 70° ~ 110°
if(angle>=70 && angle<=110)
return 'center';
// Left zone: 110° ~ 158°
if(angle>110 && angle<=158)
return 'left';
// Return null if the angle is outside all valid zones
return null;
}
4: Summary
Through this Project, I gained a more intuitive understanding of gesture recognition. The SEN0670 AI Vision Gesture and Posture Sensor can not only recognize different gestures but also output hand keypoint and posture data. By further analyzing this data, we can convert information such as palm direction and motion trajectory into control commands for games.
In this penalty kick goalkeeper game, I used "open palm" as the base gesture and calculated palm direction through the 21 hand keypoints, achieving control over three save directions: left, center, and right.
This also made me realize that gesture recognition isn't just about "recognizing gestures" — it can serve as an interaction method that connects real physical actions with the virtual world.
This penalty kick goalkeeper game is just the beginning. You can also use the SEN0670's gesture recognition, pose estimation, and keypoint data to try turning more real-world actions into game controls and create your own interactive games.
5: Attachments
Attachment 1: SEN0670 AI Vision Gesture and Posture Sensor companion learning tool
