AI-Based XiaoZhi Fitness Assistant with Pose Recognition

auroraAA Nov 10.2026

0 46 Easy

I. Project Introduction

Have you ever had this experience: when you first go to the gym, you feel a bit overwhelmed by all the fitness equipment and exercises, many movements are still unfamiliar, and you want to ask someone for help but feel too embarrassed; when you follow along with videos on your own, you always worry that your form is off, and even after practicing for a while, you have no idea whether you are doing it correctly. If only you had a "personal trainer" by your side at times like this — someone who could understand your movements, promptly remind you what needs adjusting, and even talk with you directly — wouldn't that be so much more convenient?
Based on this idea, I tried using the UNIHIKER UNIHIKER K10 as the Board and interaction terminal, with the AI Vision Posture and Gesture Sensor pose estimationSensor responsible for capturing human pose and keypoint data, and combined it with Xiao Zhi AI voice interaction and MCP tool calling to build an AI fitness coach that can recognize human poses, correct movements, and provide real-time voice feedback. I hope it can not only help you "watch your form," but truly accompany your training, give movement correction feedback, and also let you talk directly with Xiao Zhi — for example, asking "Is this movement correct?" and Xiao Zhi will answer based on the actual detection results from the Sensor. This lets even fitness beginners train on their own.

II. Project Framework Diagram

In this Project, we use AI Vision pose estimationSensor + UNIHIKER K10 + Xiao Zhi AI together to build an AI fitness coach assistant that can "watch movements, make judgments, and communicate."
Among them, the AI Vision pose estimationSensor is responsible for real-time collection of human movement information and completes human detection, pose estimation, and keypoint output; the UNIHIKER K10 serves as the core control terminal of the system, responsible for receiving and parsing Sensor data, analyzing and processing the current human state, movement category, and training results, while displaying standard movement images and real-time feedback on the screen. And Xiao Zhi takes on the role of the "intelligent fitness coach," responsible for voice interaction with the user. Users can directly ask questions like "Am I in position?" or "Is this movement correct?" The UNIHIKER UNIHIKER K10 registers capabilities such as human state queries, movement feedback, and training mode switching as callable tools through MCP, and Xiao Zhi automatically selects the appropriate tool based on semantics, then feeds the analysis results back to the user in natural language.

What is MCP?

MCP (Model Context Protocol) aims to provide a standardized way for large language model (large language model (LLM)) applications to seamlessly integrate with external data sources and tools. You can think of it as a "bridge" connecting AI and external device capabilities. Large language models are inherently good at understanding user language and generating responses, but they do not know what the device is currently detecting, nor can they directly read Sensor data, so these capabilities need to be exposed to AI through MCP.
"Xiao Zhi MCP" refers to the integration and application of the Model Context Protocol (MCP) by the open-source AI chatbot Project "Xiao Zhi" (Xiaozhi). It enables Xiao Zhi, as a voice interaction terminal, to connect to and control the external world through the MCP standard. In this Project, the posture Sensor is responsible for collecting human pose and keypoint data, the UNIHIKER UNIHIKER K10 is responsible for parsing this data and analyzing movements, and registers functions such as human state queries, movement feedback, and training mode switching as MCP tools. When the user asks by voice "Is this movement correct?", Xiao Zhi automatically selects and calls the corresponding MCP tool based on the user's semantics, reads the real training data already analyzed by the UNIHIKER UNIHIKER K10, and then converts the results into natural language feedback for the user.

III. Project Demonstration

Homepage Effects and Features

After the system starts, it first enters the training movement selection interface, with the currently supported training movements listed in the center, such as "Wall Sit, Lunge Stretch, Plank." Users can quickly switch between the fitness movements they want to train using the buttons (A: short press — select mode upward, B: short press — select mode downward), and long-press the A button to enter the corresponding training mode.
You can also wake up Xiao Zhi by saying "Hello, Xiao Zhi," and then tell it: "I want to practice wall sits today," and Xiao Zhi will automatically enter wall sit correction mode.

Secondary Page Effects and Features

After selecting "Wall Sit" mode, the Sensor will provide prompts based on your movements, indicating where you can adjust. You can ask Xiao Zhi: "Hello, Xiao Zhi, please tell me the key technical points of the wall sit" — it will tell you how to position your legs, how much to tighten your core, and other key technical points, so you can adjust in time.
You can also tell Xiao Zhi: "I want to challenge a 30-second wall sit, please help me time it," and Xiao Zhi will automatically start a countdown and play an ending prompt tone when finished.

Demonstration: Play Recorded Video

IV. Project Production Preparation

HARDWARE LIST

1 UNIHIKER K10

Link

1 SEN0670 AI Vision Posture and Gesture Sensor

Link

1 PH2.0-4P Data Cable

Link

1 USB TYPE-C Data Cable

Link

Wiring Connections

Required Software

xiaozhi-esp32 is an open-source ESP-IDF Project. Since the ESP-IDF configuration process is quite cumbersome, you can directly download the xiaozhi firmware I have debugged and flash it onto the UNIHIKER K10.

Flashing tool: Attachment 1

firmware download: Attachment 2

V. Project Production Steps

During fitness training, we usually judge whether a movement is correct by observing body posture — for example, whether the thighs are close to horizontal during a wall sit, whether the front and back legs are positioned correctly during a lunge, and whether the body remains stable and the abdomen is tightened during a plank. So here comes the question: How does AI "understand" these movements and judge whether our posture is correct?

This is where human pose estimation comes in. pose estimation does not simply determine whether there is "a person" in the frame; instead, it analyzes the person's current posture through key positions such as the head, shoulders, arms, hips, knees, and ankles. Some common pose estimation solutions run vision models on computers, phones, or more powerful Board devices, and then determine what movement the person is currently doing based on the positional relationships of these keypoints. But these methods require significant computing power or good computer performance.

In this Project, the AI Vision Posture and Gesture Sensor used does not require deploying complex vision models. It only needs to receive the recognition results returned by the Sensor to detect the human body and output 17 keypoints. It can also "learn" to remember pre-recorded standard fitness poses through its learning function.

When the user is training, the Sensor compares the current human pose with the already-learned standard movements and provides the corresponding pose ID and similarity score. The UNIHIKER K10 then determines the current movement state based on these recognition results, displays the results on the screen, and provides them to Xiao Zhi for voice feedback.

Learning Fitness Poses

Before it can judge whether our movements are correct, we first need to let the AI Vision pose estimationSensor "recognize" the movements we want to detect. Although the Sensor can detect the human body and output keypoints, it does not know what pose in the Project represents "wall sit," "lunge stretch," or "plank." Therefore, we need to first use the Sensor's self-learning function to separately collect several standard fitness poses, so the Sensor remembers the features of these movements.

After learning is complete, each movement corresponds to an independent pose ID. When the user stands in front of the Sensor to train, the AI Vision pose estimationSensor matches the current human pose with the already-learned movements and outputs the corresponding pose ID, similarity score, and human keypoint data.

In this Project, a total of 3 fitness poses were learned:

- ID 1: Wall Sit — mainly observes squat depth, knee joints, and body posture;

- ID 2: Lunge Stretch — mainly observes front and back leg positions and body center of gravity;

- ID 3: Plank — mainly observes whether the shoulders, hips, and ankles maintain a good body line.

These three movements include both standing training and ground support movements, with relatively obvious differences in human pose, making them suitable for demonstrating pose estimation and movement correction features.

Step 1: Open the Official Learning Tool

Next, we first use the accompanying official learning tool to record these 3 standard fitness poses into the Sensor one by one. First, download the official software that comes with the AI Vision Posture and Gesture Sensor, which is Attachment 3 at the end of this article.

After extracting, double-click the .exe file to launch the software. Use a USB data cable to connect your computer to the AI Vision Posture and Gesture Sensor, and the software will automatically detect the Sensor's corresponding serial port. Select the corresponding serial port and open it.

Step 2: Enter pose estimation Mode and Learn Custom Poses

Select pose estimation mode in the software. Then stand about 2 meters away from the Sensor and strike a custom pose — here we use the "wall sit" pose as an example. After striking the pose, you can see the human keypoints and the human detection box in the left panel. Before learning is complete, the human keypoints will be displayed in green.

Click the learn button and wait 3 seconds; the keypoints will change color, indicating that learning was successful. You can also customize the name.

Below is the specific pose learning process. You can also refer to the official pose learning tutorial: https://wiki.dfrobot.com/sen0670/docs/23979

Project Flashing:

This Project is a secondary development based on xiaozhi-esp32 tailored for AI fitness scenarios. The original Xiao Zhi mainly handles wake word detection, voice dialogue, network connectivity, and other basic capabilities, while this version further adds AI vision posture Sensor integration, fitness movement analysis, UNIHIKER K10 dedicated training interface, and MCP fitness tools, transforming Xiao Zhi from a simple "voice assistant" into an "AI fitness coach" capable of obtaining real human pose data. In addition, training timers, continuous timing, and voice interruption features have been added, making feedback during training more timely.

Step 1: Flash Xiao Zhi into the UNIHIKER K10

Connect the assembled hardware to your computer via USB, extract the previously downloaded flashing tool and firmware, and first open "flashdownloadtool3.9.8w1" in the "Flash download tool" folder.

Then select:

ChipType: ESP32-S3

WorkMode: Develop

LoadMode: UART, then click OK

In the first blank field, press the "..." button, select the firmware downloaded above and extract it to get the merged-binary.bin file. Use 0x00 as the starting fill address.

Then select the correct COM port.

Set the baud rate to 1152000.

First press the "ERASE" button to erase the current firmware in the UNIHIKER K10.

After the erase step is complete, press "START" to flash the Xiao Zhi firmware into the UNIHIKER K10.

Step 2: Configure Wi-Fi

After successfully flashing the firmware, press the RST button on the back of the UNIHIKER K10. The UNIHIKER K10 will first enter Wi-Fi configuration mode.

Now use your computer or phone to connect to Xiao Zhi's hotspot (check the screen for the specific hotspot name), then configure the Wi-Fi SSID and password to let Xiao Zhi connect to the internet. (Note: The UNIHIKER K10 can only connect to 2.4GHz Wi-Fi.)

Wait for the Wi-Fi connection to succeed; the UNIHIKER K10 will restart within 3 seconds and display a six-digit device code on the screen.

Step 3: Add Device

Next, go to the Xiao Zhi configuration site to configure the language model and voice.

Use a browser to enter xiaozhi.me to open the Xiao Zhi configuration website, and click on the console.

Register a Xiao Zhi account using your phone number. Then after entering the console, enter the 6-digit device code shown on the UNIHIKER K10 display.

After successful addition, the UNIHIKER K10 will display the following, indicating successful addition.

Step 4: Custom Settings

You can enter the character configuration screen to customize and change xiaozhi's character identity.

Step 5: Wake Up Your AI Fitness Assistant

After completing the previous configuration, our AI fitness coach is officially ready to go on duty! Next, let's go through a simple fitness experience to see how the AI Vision Posture and Gesture Sensor SEN0670 and the UNIHIKER UNIHIKER K10 work together.

1. Get Ready for Exercise

First, fix the AI Vision pose estimationSensor in a suitable position, facing the exercise area. Then stand in front of the Sensor and adjust the distance to ensure that your body and the key parts that need to be recognized during exercise are within the Sensor's recognition range, avoiding body occlusion.

2. Select a Fitness Movement

Turn on the UNIHIKER UNIHIKER K10 and use the A and B buttons on the homepage to switch between different fitness movement modes. After selecting the movement you want to practice, long-press the A button for about 1 second to enter the corresponding fitness interface. You can also use "Hello, Xiao Zhi, enter WallSit mode" for real-time voice interaction to enter the page mode.

The button functions of the UNIHIKER UNIHIKER K10 are as follows:‘

Interface

A Button

B Button

Homepage

Short press: select mode upward;

Long press for about 1 second: enter the selected mode

Short press: select mode downward

Fitness Interface

Short press: return to homepage

3. Start Exercising and Experience AI Movement Feedback
After entering fitness mode, you can refer to the standard fitness pose on the fitness interface and start exercising. You can also consult Xiao Zhi by voice: "Hello, Xiao Zhi, I want to practice the WallSit. Can you tell me the key technical points of this fitness movement?" Then Xiao Zhi will tell you what to pay attention to, what angles to maintain for your hands and feet, how to tighten your back, and so on.
During exercise, the Sensor collects human pose information and passes the recognition results to the UNIHIKER UNIHIKER K10. The system analyzes whether the current movement meets the requirements based on the positions of human keypoints and movement judgment rules. When the movement meets the preset standards, the system gives the corresponding correct movement feedback; if the movement is not standard, the system also provides prompts to guide us in adjusting our posture.
In this way, we can not only complete movement practice, but also adjust our movements in time based on AI feedback.

4. Experience Voice Training Timing

In addition to movement correction, Xiao Zhi can also help us record training time. This Project provides two timing methods:

(1) Countdown Training

If you want to do a set of training with a fixed duration, you can directly tell Xiao Zhi:

"Xiao Zhi, help me do a one-minute plank."

Xiao Zhi will call the countdown tool through MCP and start timing according to the specified duration. During training, it can also give periodic reminders based on the timing status, such as "30 seconds left, hold on!" "Last 10 seconds, keep going!" etc. When the countdown ends, Xiao Zhi will play a prompt tone indicating that this set of training is complete.

(2) Count-Up Training

If you want to challenge your endurance and see how long you can hold a movement, you can also tell Xiao Zhi:

"Xiao Zhi, help me see how long I can hold a plank."

At this point, Xiao Zhi will start the count-up function and continuously record the training duration. When we can no longer hold on, we just need to say "stop timing," and Xiao Zhi will end the timing and report the actual duration of this training session.

For example: "This plank lasted 58 seconds, not bad!"

Through these two timing methods, we can do daily training with a fixed duration or challenge our endurance at any time.

5. End Training

After training is complete, you can press the B button to return to the UNIHIKER K10 homepage and select other fitness movements to continue, or tell Xiao Zhi by voice to "end training."

VI. How Does Xiao Zhi "Understand" Your Fitness Movements?

When we stand in front of the AI Vision pose estimationSensor and perform a fitness movement, how does Xiao Zhi judge whether our fitness movement is "correct"? Obviously, Xiao Zhi does not directly "see" our movements; instead, behind the scenes it goes through an entire process from human pose collection, data parsing, movement judgment, to AI obtaining results and providing voice feedback.

Pose Data Parsing

When we stand in front of the AI Vision pose estimationSensor and perform movements, it does not directly tell the UNIHIKER K10 "whether your movement is correct." What it does first is turn the human body into a set of computable data. For example, it determines whether there is a person in the frame, which learned pose the current pose most resembles, and where body parts such as the shoulders, hips, knees, and ankles are located. This information is returned in the form of pose_keypoints, which includes the human detection box, pose ID, similarity score, and 17 human keypoints.

CODE


 // Parse human pose data returned by SEN0670
static bool ParseFitnessPose(
    const char* json,
    FitnessPoseSnapshot& pose
) {
    pose = FitnessPoseSnapshot();

    // Locate the pose_keypoints data
    const char* pose_key =
        FindLastKey(json, "\"pose_keypoints\"");
    if (!pose_key) return false;

    // Find the start of the current JSON object
    const char* object_start = pose_key;
    while (object_start > json && *object_start != '{') {
        --object_start;
    }

    cJSON* root =
        cJSON_ParseWithOpts(object_start, nullptr, false);
    if (!root) return false;

    cJSON* keypoints =
        cJSON_GetObjectItemCaseSensitive(
            root, "pose_keypoints"
        );

    if (!cJSON_IsArray(keypoints) ||
        cJSON_GetArraySize(keypoints) == 0) {
        cJSON_Delete(root);
        return false;
    }

    // Read the detected human body data
    cJSON* person = cJSON_GetArrayItem(keypoints, 0);
    cJSON* header = cJSON_GetArrayItem(person, 0);

    // Data format:
    // [x, y, width, height, score, pose_id]
    pose.person  = true;
    pose.x       = cJSON_GetArrayItem(header, 0)->valueint;
    pose.y       = cJSON_GetArrayItem(header, 1)->valueint;
    pose.width   = cJSON_GetArrayItem(header, 2)->valueint;
    pose.height  = cJSON_GetArrayItem(header, 3)->valueint;
    pose.score   = cJSON_GetArrayItem(header, 4)->valueint;
    pose.pose_id = cJSON_GetArrayItem(header, 5)->valueint;

    // Parse the 17 human body keypoints
    cJSON* points = cJSON_GetArrayItem(person, 1);

    if (cJSON_IsArray(points) &&
        cJSON_GetArraySize(points) >= 17) {

        for (int i = 0; i < 17; ++i) {
            cJSON* point =
                cJSON_GetArrayItem(points, i);

            int x =
                cJSON_GetArrayItem(point, 0)->valueint;
            int y =
                cJSON_GetArrayItem(point, 1)->valueint;
            int confidence =
                cJSON_GetArrayItem(point, 2)->valueint;

            FitnessPoint p =
                PointFromKeypoint(x, y, confidence);

            // Store the main joints required for motion analysis
            switch (i) {
                case 5:  pose.lshoulder = p; break;
                case 6:  pose.rshoulder = p; break;
                case 11: pose.lhip = p; break;
                case 12: pose.rhip = p; break;
                case 13: pose.lknee = p; break;
                case 14: pose.rknee = p; break;
                case 15: pose.lankle = p; break;
                case 16: pose.rankle = p; break;
                default: break;
            }
        }

        pose.keypoints_available = true;
    }

    cJSON_Delete(root);
    return true;
}

Judging Movements Based on Pose ID and Keypoint Data

After obtaining the human pose data, the next step is to judge: Is this movement actually performed correctly?

Taking the wall sit as an example, the program uses the hip, knee, and ankle keypoints to calculate the knee joint angle, and combines knee spacing, ankle spacing, and upper body tilt to determine whether there are issues such as squatting too high, knees caving inward, or body tilting.

CODE


 // Evaluate the wall sit exercise
static FitnessFeedbackCode EvaluateWallSit(
    const FitnessPoseSnapshot& p
) {
    // No person detected
    if (!p.person)
        return FITNESS_FB_NO_PERSON;

    // Insufficient keypoint data
    if (!p.keypoints_available || p.height <= 0)
        return FITNESS_FB_ADJUST_POSITION;

    // Calculate the average hip-knee-ankle angle of both legs
    const float knee = AverageValid(
        Angle3(p.lhip, p.lknee, p.lankle),
        Angle3(p.rhip, p.rknee, p.rankle)
    );

    const FitnessPoint shoulder =
        Midpoint(p.lshoulder, p.rshoulder);
    const FitnessPoint hip =
        Midpoint(p.lhip, p.rhip);

    if (knee < 0 ||
        !shoulder.valid ||
        !hip.valid)
        return FITNESS_FB_ADJUST_POSITION;

    // A large knee angle indicates insufficient squat depth
    if (knee > 125.0f)
        return FITNESS_FB_WALL_SIT_TOO_HIGH;

    // Check whether the knees are excessively caving inward
    const float knee_width =
        fabsf(p.lknee.x - p.rknee.x);
    const float ankle_width =
        fabsf(p.lankle.x - p.rankle.x);

    if (ankle_width > 1.0f &&
        knee_width < ankle_width * 0.60f)
        return FITNESS_FB_WALL_SIT_KNEE_IN;

    // Check whether the upper body is tilted
    if (TiltFromVertical(shoulder, hip) > 22.0f)
        return FITNESS_FB_WALL_SIT_BODY_TILT;

    return FITNESS_FB_NEAR_STANDARD;
}

Organizing Movement Analysis Results into Data That Xiao Zhi Can Understand

After completing movement judgment, these results also need to be provided to Xiao Zhi. In the Project, GetFitnessFeedbackJson() is used to uniformly organize the current human state, movement category, similarity score, and movement feedback into JSON data. The actual code also includes information such as data freshness and training status.

CODE


 // Convert the current fitness state into JSON
std::string GetFitnessFeedbackJson() {
    // Get the latest training state
    FitnessEvaluatedState state =
        CopyFitnessState(true);

    char json[1200];

    snprintf(
        json,
        sizeof(json),
        "{"
        "\"exercise\":\"%s\","
        "\"person_detected\":%s,"
        "\"keypoints_available\":%s,"
        "\"detected_pose_id\":%d,"
        "\"stable_pose_id\":%d,"
        "\"similarity_score\":%d,"
        "\"feedback\":\"%s\","
        "\"feedback_zh\":\"%s\""
        "}",

        // Currently selected exercise
        ExerciseNameEn(state.exercise),

        // Whether a person is detected
        state.pose.person ? "true" : "false",

        // Whether human body keypoints are available
        state.pose.keypoints_available ?
            "true" : "false",

        // Current pose ID detected by SEN0670
        state.pose.pose_id,

        // Pose ID stabilized across multiple frames
        state.filtered_pose_id,

        // Current pose similarity score
        state.filtered_pose_id != 0 ?
            state.filtered_score : 0,

        // Motion feedback code
        FeedbackCodeName(state.feedback),

        // Chinese motion feedback message
        FeedbackZh(state.feedback)
    );

    return std::string(json);
}

Registering MCP Tools So Xiao Zhi Can Call Movement Analysis Results

With the movement analysis results ready, we also need to provide Xiao Zhi with an interface to access the data. In the Project, the self.fitness.get_feedback tool is registered through McpServer::AddTool(). The tool description in the actual code explicitly specifies that this tool should be called when the user asks whether a movement is correct, how the posture is, what the similarity score is, or what needs adjusting.

CODE


 // Register the MCP tool for fitness motion feedback
void InitializeTools() {
    auto& mcp_server =
        McpServer::GetInstance();

    mcp_server.AddTool(

        // MCP tool name
        "self.fitness.get_feedback",

        // Tool functionality description
        // XiaoZhi uses this description to determine
        // whether to invoke the tool based on the user's query
        "Query the current exercise posture, "
        "pose similarity score, and correction suggestions. "
        "Call this tool when the user asks whether their "
        "exercise posture is correct, how well they are "
        "performing the movement, or what needs to be adjusted. "
        "Provide feedback based on the actual sensor "
        "analysis results.",

        // No additional parameters required
        PropertyList(),

        // Callback executed when XiaoZhi invokes this tool
        [this](const PropertyList&)
            -> ReturnValue {

            // Read the motion analysis results processed by K10
            std::string result =
                GetFitnessFeedbackJson();

            ESP_LOGI(
                "FITNESS",
                "MCP get_feedback CALLED: %s",
                result.c_str()
            );

            // Return the result to XiaoZhi
            return result;
        }
    );
}

So the entire calling process can be summarized as:

CODE


User:
"Is my posture correct?"
        ↓
XiaoZhi understands the user's intent
        ↓
Selects the MCP tool
self.fitness.get_feedback
        ↓
K10 executes the tool callback
        ↓
GetFitnessFeedbackJson()
        ↓
Retrieves the current pose ID,
similarity score, and motion feedback
        ↓
Returns the JSON result
        ↓
XiaoZhi generates a natural language response
        ↓
"Your posture is quite good.
Keep it up!"

VII. Summary

At this point, we have successfully combined the AI Vision Posture and Gesture Sensor, the UNIHIKER UNIHIKER K10, and Xiao Zhi voice interaction to create an AI fitness coach that can recognize human movements, judge whether movements are correct, and provide real-time voice feedback.

Of course, the applications of this AI fitness coach go far beyond this. Imagine: at home, it can become our personal exercise partner, accompanying us through daily workouts and promptly reminding us of improper movements; on campus, it can assist physical education, help students practice standard movements, and even transform into a fun motion-sensing game device. And the capabilities of the AI Vision Posture and Gesture Sensor are not limited to fitness — by recognizing different human poses and gestures, we can also try using a wave to switch training Projects, using specific movements to control lights, or designing interactive games that require no screen touching. In the future, if combined with more Sensors and external devices, the UNIHIKER UNIHIKER K10 can transform these recognition results into even richer interactive experiences, letting AI not only understand our commands but also understand our movements, truly entering more everyday life scenarios.