1. Project Background
Recently, I started learning to cook Chinese dishes at home. Recipes and cooking videos were a great help, but I soon ran into a practical problem: just when I needed to check the next step, my hands were often wet, oily, or covered in ingredients. I kept having to stop, wipe my hands, and tap the screen—a small interruption that became frustrating when I was trying to keep up with a hot wok. That gave me an idea: what if I could follow a recipe without touching the screen? I built a touchless kitchen recipe assistant around UNIHIKER K10. An AI Pose & Gesture Recognition Sensor lets me choose a recipe and move between steps, while a speech synthesis module reads the instructions aloud. An LED flashes to confirm that a gesture has been recognized. Together, the screen, voice prompts, and gesture controls let me focus on cooking with fewer interruptions.
2. Project Objective
Build a kitchen recipe assistant that lets users select recipes, view ingredients, follow cooking steps, and receive spoken instructions and timer prompts through gesture control. The following video demonstrates the complete project experience.
Hardware naming note: Product names follow DFRobot's English catalog. This project uses the V1.0 speech synthesis module described in the original tutorial. The current DFRobot product page lists V2.0; the version here has been retained to match the program. Cable lengths and exact accessory SKUs are not specified in the source and have not been inferred.
4. Preparation
1. Set Up Custom Gestures for Recipe Navigation
Before writing the program, first try the core functions of the sensor using its companion desktop application. For detailed instructions on using the AI Pose & Gesture Recognition Sensor, see the following page:
wiki.dfrobot.com.cn/SKU_SNE0670_Gravity_AIAI Pose & Gesture Recognition Sensor
Download and extract the official desktop application for the AI Pose & Gesture Recognition Sensor, then double-click the .exe file.
Open the application and connect the sensor to the computer with a USB data cable. The interface shown below appears. Select the port and click Open to display the sensor camera feed. The screenshots in the original tutorial use Chinese, selected via the language menu in the upper-right corner.
The desktop application defaults to custom-learned gesture recognition mode. As shown below, make a hand gesture. The sensor recognizes it and outputs its ID and confidence score, together with real-time data for 21 hand keypoints. Click the Learn button to teach the sensor a gesture and record its gesture ID. This project uses two custom-learned gestures, assigned to ID1 and ID2, to navigate the recipe assistant.
Next, switch to fixed gesture recognition mode. As shown below, make one of the gestures illustrated on the right. The sensor identifies the gesture and outputs its ID and name.
The sensor also supports body pose recognition. This project does not use body pose learning, so that function is not covered here. After learning the gestures as described above, you can start writing the code for a touch-free kitchen recipe assistant using gesture recognition.
2. Add the Recipe Images
Before writing the program, copy the prepared image assets to the microSD (TF) card, then insert the card into UNIHIKER K10. The project uses 20 images, each a 140×128-pixel PNG. Note: For the microSD (TF) card used with UNIHIKER K10, the recommended maximum capacity is 16GB. A genuine SanDisk card sold by DFRobot is recommended. Cards from unknown brands may cause read/write failures or program crashes.
The table below lists the image filename prefixes and step numbers for the recipes used in this project.
| Recipe | Filename Prefix | Step Numbers |
| Stir-Fried Tomatoes and Eggs | recipe_tomato_ | 1～5 |
| Egg Fried Rice | recipe_fried_rice_ | 1～5 |
| Cola Chicken Wings | recipe_cola_wings_ | 1～5 |
| Stir-Fried Pork with Green Peppers | recipe_pepper_pork_ | 1～5 |
3. Download the Required Software and Libraries
(1) This project is programmed in Arduino C/C++. First, download and install Arduino IDE from https://www.arduino.cc/en/software/.
After downloading, follow the installation prompts to install Arduino IDE, then configure UNIHIKER K10 for the project. For detailed setup instructions, see: https://www.unihiker.com/wiki/K10/GettingStarted/gettingstarted_arduinoide/
(2) Download and extract the project resources attached at the end of this tutorial. Locate the DFRobot_HumanPose and DFRobot_SpeechSynthesis_M folders, which contain the libraries for the AI Pose & Gesture Recognition Sensor and the speech synthesis module, respectively.
5. Build the Project
1. Analyze the Workflow
The AI Pose & Gesture Recognition Sensor performs gesture recognition locally. UNIHIKER K10 reads the results and converts them into internal action IDs. A page state machine then determines the action according to the current page.
The diagram below shows the core functions of the project:
System Workflow:
The home page lets users cycle through four recipes and automatically adjusts the visible card area to keep the selected recipe in view. The ingredients page displays the ingredients for that recipe, and another gesture opens the cooking steps. During cooking, left and right gestures move to the previous or next step, and a fist gesture returns to the home page at any time. The display, spoken instructions, and timer update together according to the current recipe and step. The digital LED flashes whenever a gesture is successfully recognized.
The following table maps the gestures used in this project to their functions:
| Gesture | System Action | Recognition ID |
| Left gesture (custom-learned) | Previous recipe/step | 1 |
| Right gesture (custom-learned) | Next recipe/step | 2 |
| OK (fixed gesture) | Confirm selection | 8 |
| Fist (fixed gesture) | Return to home page | 0 |
Gesture labels: “Left” and “right” name the two gestures learned for this project. The SEN0670 sensor recognizes static hand gestures; these labels do not imply dynamic motion-path recognition. For English setup instructions, see the official getting-started guide.
2. Connect the Hardware
Connect the AI Pose & Gesture Recognition Sensor to the I²C port on UNIHIKER K10 and the Digital White LED Light Module to P0. Insert UNIHIKER K10 into the Robotics & Power IO Extender, and connect the Speech Synthesis Module V1.0 to the extender. Connect UNIHIKER K10 to the computer with a USB data cable when uploading the program.
Note: After the program has been uploaded from the computer to UNIHIKER K10, the project can also be powered directly by a power bank.
Once the devices are connected, write the complete program.
3. Write the Program
Core Program Logic
The program initializes the devices and then opens the recipe home page. During operation, the main loop continuously handles the LED state, speech playback status, gesture recognition, and step timing. It determines the required action from the current page and the recognized gesture.
Core Gesture Recognition Logic:
- The program alternates between the eHand and eGesture models. After a successful model switch, the sensor may still hold inference results from the previous model, or its image data may not yet be stable. The program therefore waits 200 ms before reading results.
- After the model settles, the program reads gesture results from the current model for approximately 650 ms. If no valid gesture is established during this window and the system is not locked, it switches to the other model to continue recognition.
- When the program first detects a valid gesture, it records it only as a candidate and stores its appearance time in candidateSince, without immediately performing an action. The gesture is confirmed only after the same gesture has been recognized continuously for 250 ms.
- After confirming a stable gesture, the program first sets gestureLocked to true, then calls handleGesture() to perform the page action. The digital LED lights up for 200 ms at the same time to indicate that the system has accepted the gesture.
- After performing the gesture action, the system locks. It unlocks and allows the next action only after the sensor has detected no hand continuously for approximately 600 ms.
Project File Structure
The project file structure is shown below. RecipeAssistant_K10.ino is the main program file. Most changes to recipe names, ingredients, cooking steps, timer durations, interface text, or colors are made in this file.
RecipeAssistant_K10/
│
├─ data/ [Images and debugging resources]
│ ├─ recipe_images_preview/ [Step images; replace using the naming rules]
│ │ ├─ recipe_tomato_1.png ～ recipe_tomato_5.png
│ │ ├─ recipe_fried_rice_1.png ～ recipe_fried_rice_5.png
│ │ ├─ recipe_cola_wings_1.png ～ recipe_cola_wings_5.png
│ │ └─ recipe_pepper_pork_1.png ～ recipe_pepper_pork_5.png
│
│
│
└─ RecipeAssistant_K10_c/ [Arduino program folder]
│
├─ RecipeAssistant_K10.ino [Main program; editable]
│
└─ src/ [Local dependencies; normally leave unchanged]
│
├─ unihiker_k10_local/ [UNIHIKER K10 adaptation library]
│ ├─ unihiker_k10.h
│ ├─ unihiker_k10.cpp
│ ├─ GT30L24A3W.h
│ └─ DFRobot_GT30L24A3W.cpp
│
├─ DFRobot_HumanPose/ [SEN0670 gesture sensor library]
│ ├─ DFRobot_HumanPose.h
│ ├─ DFRobot_HumanPose.cpp
│ ├─ Result.h
│ ├─ Result.cpp
│ ├─ library.properties
│ └─ LICENSE
│
└─ DFRobot_SpeechSynthesis_M/ [Speech module reference library]
├─ DFRobot_SpeechSynthesis_M.h
└─ DFRobot_SpeechSynthesis_M.cppSelected Key Code:
(1) Initialize the Devices and Interface
setup() starts serial communication and UNIHIKER K10, configures the LED output, creates a landscape canvas, and initializes the microSD (TF) card. Create the background object before the canvas to obtain a valid display object. Screen orientation 1 corresponds to 320×240 pixels. The relevant code excerpt follows:
void setup() {
Serial.begin(115200);
delay(500);
k10.begin();
digitalWrite(GESTURE_LED_PIN, GESTURE_LED_ON == HIGH ? LOW : HIGH);
pinMode(GESTURE_LED_PIN, OUTPUT);
k10.initScreen(SCREEN_DIR);
k10.setScreenBackground(COLOR_BG);
// This K10 library requires a background object first so Canvas has a valid parent.
k10.initBgCamerImage();
k10.creatCanvas();
k10.setBgCamerImage(false);
// Use the user-provided MindPlus TF card initialization API once at startup.
Serial.println(F("[TF] Initializing SD file system..."));
k10.initSDFile();
Serial.println(F("[TF] SD initialization returned."));
Next, initialize the V1 speech synthesis module, connect the gesture sensor, and select the eHand model. Both devices share Wire, but use different addresses and access clock speeds.
const uint8_t SENSOR_ADDR = 0x3A;
const uint8_t SPEECH_ADDR = 0x40;
const uint32_t GESTURE_I2C_CLOCK = 400000;
const uint32_t SPEECH_I2C_CLOCK = 100000;
(2) Map Gesture IDs and Alternate Recognition Models
In the custom-learned eHand model, ID1 and ID2 correspond to the left and right gestures. In the fixed eGesture model, ID8 and ID0 correspond to OK and fist. mapGestureResult() converts the raw ID according to the current model. The internal values GESTURE_OK=3 and GESTURE_FIST=4 are action IDs; there is no need to teach additional gestures as ID3 or ID4. ID0 in the custom-learned model does not trigger a return action. The relevant code follows:
uint8_t mapGestureResult(uint8_t rawId) {
if (readingLearnedModel) {
if (rawId == 1) return GESTURE_LEFT;
if (rawId == 2) return GESTURE_RIGHT;
} else {
if (rawId == 8) return GESTURE_OK;
if (rawId == 0) return GESTURE_FIST;
}
return GESTURE_NONE;
}
The two models run alternately. After a successful switch, the program waits 200 ms before entering a sampling window of approximately 650 ms. If the switch fails, it pauses reading and waits 1000 ms before retrying the same target model. Old results and candidate gestures are cleared when switching, preventing IDs from different models from being mixed. The relevant code follows:
void switchGestureModel() {
const bool nextLearned = !readingLearnedModel;
restoreGestureI2CClock();
const DFRobot_HumanPose::eCmdCode_t status = sensor.setModelType(
nextLearned ? DFRobot_HumanPose::eHand : DFRobot_HumanPose::eGesture);
if (status != DFRobot_HumanPose::eOK) {
// A failed return leaves the device model uncertain; do not read results, and retry the same target.
modelSwitchFailed = true;
modelRetryAt = millis() + MODEL_RETRY_MS;
Serial.println(F("[Gesture] model switch failed; retrying."));
return;
}
readingLearnedModel = nextLearned;
modelSwitchFailed = false;
while (sensor.availableResult()) sensor.popResult();
candidateGesture = GESTURE_NONE;
candidateSince = millis();
modelReadyAt = millis() + MODEL_SETTLE_MS;
modelWindowSince = modelReadyAt;
}
(3) Confirm Stable Gestures and Prevent Repeated Triggers
First, read the results from the current model, map them to actions, and select the valid action with the highest confidence score. If communication fails, clear the candidate, but do not treat the failure as the hand having been removed. processGestureFrame() requires the same candidate to persist for at least 250 ms before executing the action and locking. Another trigger is allowed only after the hand has been removed for approximately 600 ms.
The relevant code follows:
void processGestureFrame(uint8_t id) {
unsigned long now = millis();
if (id == GESTURE_NONE) {
if (now - lastGestureSeen >= 600) {
gestureLocked = false;
candidateGesture = GESTURE_NONE;
}
return;
}
lastGestureSeen = now;
if (gestureLocked) return;
if (id != candidateGesture) {
candidateGesture = id;
candidateSince = now;
return;
}
if (now - candidateSince >= 250) {
gestureLocked = true;
// Flash once after 250 ms of stable recognition; holding the same gesture does not flash repeatedly.
// Indicate recognition even if the gesture has no action on the current page.
flashGestureLed();
handleGesture(id);
}
}
4. Run the Program
After completing the wiring, gesture learning, and image transfer to the microSD (TF) card, compile and upload the complete program to UNIHIKER K10. After power-on, wait for the speech module and sensor to initialize. Begin operating when the prompt meaning “Please select a recipe” appears. Open the program file in Arduino IDE and run it on the board to see the result.
The image below shows the project in operation:
Note: See the attachment for the complete program.
Debugging and Notes
- Sensor connection failure: Check that the sensor is in I²C mode, its address is 0x3A, SDA/SCL are connected correctly, the devices share a common ground, and power is supplied correctly. Power-cycle the sensor after changing the DIP switch settings.
- Inaccurate or unresponsive gesture recognition: Check that the gesture IDs are correct. You can first verify the recognition results in the desktop application, then adjust hand orientation, lighting, and background. After each action, move your hand completely out of the recognition area and wait approximately 0.6 seconds before making the next gesture, so the trigger lock does not interfere with successive actions.
- No speech when changing pages: Check the speech module wiring, address 0x40, V1 mode, and volume settings. If the previous speech is still playing, the program skips the new announcement. Wait until playback finishes before performing the next action.
- Ingredient quantities, cooking steps, and timings in this project are for reference only. Adjust the recipe data in the code according to your cooking experience and needs, and update the corresponding step images at the same time.
- Keep the DFRobot_HumanPose gesture sensor library and the project-specific unihiker_k10_local library in the src folder of the project. Arduino IDE automatically compiles these dependencies when the main program is opened; They do not need to be installed again through Library Manager.
Google Drive: https://drive.google.com/file/d/1Bgt-ubCy1Ho3Ez5XHkbWvqP4BTaXNqDu/view?usp=sharing
