Multi-Model Based Ordering and Points System

1.1 Project Introduction

In a beverage shop, when customers queue up to order during peak hours and want to check their points, they usually have to take out their phones, launch a mini program, find their personal QR code, and then ask the staff to scan it to check their points and complete the payment. To solve this, we built this "Multi-Model Based Ordering and Points System". It uses a dual-model parallel technology that combines face recognition and gesture recognition to effectively address these pain points. Customers only need to stand in front of the device; the camera instantly identifies the customer, eliminating the need to find a QR code, and the screen automatically displays the customer's exclusive points. The whole process requires no touching of the screen or taking out a phone: customers simply make a "victory" sign in the air to switch beverages, and when they see the one they want, they make an "OK" gesture and the system completes the order and awards points.
For customers, this delivers truly "contactless, seamless point accrual"; for managers, it improves ordering efficiency. In addition, the solution can be flexibly adapted to scenarios such as smart canteens, corporate pantries, and gyms, making internal benefits management and automatic point accumulation easy.

1.2 Demo Video

2.Project Implementation Principle

Its core logic is as follows: First, we need to "enroll" face information in the HUSKYLENS 2 camera in advance and preset each person's initial points in the UNIHIKER K10. When a customer stands in front of the device, multiple models on HUSKYLENS 2 run simultaneously to perform different visual recognition tasks, capturing faces and gestures in real time, and transmit the recognized "face ID" and "gesture ID" to the UNIHIKER K10 via I2C communication.
The UNIHIKER K10 processes and evaluates the transmitted data: if a learned, valid face is recognized, the screen immediately displays that customer's exclusive points. At this point, if the customer makes a "victory" sign, the system switches the number of the beverage to be purchased; if the customer makes an "OK" gesture, the system completes the purchase, awards the corresponding points, and refreshes the screen with the message "Purchased!". If no customer face is recognized, the screen displays "Please stand in front of the camera", and no purchase can be made.

3. Hardware and Software Preparation

3.1 Equipment List

Software: Mind+ Programming Software (Version: 2.0)

Note: The Mind+ Programming Software is version v2.0, and the HUSKYLENS device is HUSKYLENS 2.

3.2 Hardware Connection

Connect the computer, UNIHIKER K10, and HUSKYLENS 2 according to the wiring diagram below.

3.3 Software Platform Preparation

Download and install the Mind+ 2.0 or later installer from the official website. After installation, double-click the icon to launch it.

4. Project Development

4.1 HUSKYLENS 2 Operation

First, select the communication mode of HUSKYLENS 2.
HUSKYLENS 2 supports two communication modes, UART (serial port) and I2C. This tutorial uses I2C communication mode.
Click System Settings -> Protocol Type -> select I2C communication mode, then return to the main menu interface.

Then, swipe the screen to find the "Face Recognition" function.

Point HUSKYLENS 2 at a scene containing a face. When a face is detected, the screen draws a white box around the face and marks the positions of the eyes, nose, and corners of the mouth with white dots. If multiple faces appear in the frame, multiple boxes will be drawn.

Aim at the face you want to learn. Once the face is enclosed by a white box, adjust the angle at which HUSKYLENS 2 "looks" so that the crosshair in the center of the screen is inside the white box, then press button A at the top right of HUSKYLENS 2 to learn that face.

After learning is complete, if a learned face is recognized, the screen draws a colored box around the face and displays the name, ID, and confidence above it, for example "Face: ID1 97%". The name defaults to "Face"; to set the name, see "Parameter Settings" in this function. "ID1" is the face of the first person learned; "97%" means the confidence is 97%. In machine learning, confidence refers to how "certain" the model is about its prediction. For example, face ID1 at 97% here means the model "believes there is a 97% probability that the face in the current frame is the learned ID1 face".

HUSKYLENS 2 RGB light auto indication: There is an RGB light on the back of HUSKYLENS 2 that provides feedback to the person in front of the camera. It lights up blue when a face is detected and green when a learned face is recognized. Learn more faces in the same way.
Follow the same steps above to learn the faces of all members, which completes the face model preparation.

Finally, swipe the screen again to find the "Gesture Recognition" function.

Aim at the palm you want to learn and adjust the angle at which HUSKYLENS 2 "looks" so that the crosshair in the center of the screen is inside the white box. To learn a gesture from multiple angles, press and hold button A, adjust the angle at which HUSKYLENS 2 "looks", and learn the gesture from different angles, as shown below.

After learning is complete, if a learned gesture is recognized, the screen draws a colored box around the gesture and displays the gesture name, ID, and confidence above it, for example "Gesture: ID1 93%". The name defaults to "Gesture"; to set the name, see "Parameter Settings". "ID1" is the first gesture learned; "93%" is the confidence, indicating the probability that the target gesture recognized by the model belongs to a certain gesture under the Gesture Recognition function. "ID1 93%" means the model believes there is a 93% probability that this gesture is the ID1 gesture. Learn more gestures in the same way.

For more detailed usage of HUSKYLENS 2, see the following website:
Gravity: HUSKYLENS 2 - 6 TOPS LLM MCP AI Vision Sensor (20+ Models, Deploy Custom Models) | DFRobot Wiki

Once the model preparation is complete, you can start writing code to build a multi-model ordering and points system

4.2 Code Implementation

Open the Mind+ Programming Software and select "Upload Mode" under "Coding".

First, we need to add the corresponding extensions in Mind+, including UNIHIKER K10 and HUSKYLENS 2.

Go to the "Extensions" page and search for "UNIHIKER K10".

Click the "Download" button on the extension package and wait for it to finish downloading.

Click the extension package to finish loading.

Follow the same steps used to load "UNIHIKER K10" to load "HUSKYLENS 2 AI Camera" from the extension libraries.

After the extension libraries "UNIHIKER K10" and "HUSKYLENS 2 AI Camera" have finished loading,
click the "Back" button to return to the coding interface.

Click "Connect" under "Connect Device".

The "Connection successful" screen is shown below. Click "×" to return.

After the extension libraries finish loading, write the program as follows:

The core code is explained below:

The attachment contains the complete program file. Expand it and click "Load Project".

Select the program file from the attachment and click "Open".

Click "Upload" to run the program.

The result is as follows:

5. Attachment List