1.1 Project Introduction
In restaurants or beverage shops, traditional paper menus or screens are often limited by layout space and can only display names and prices. When customers want to learn more about a drink, such as its calories and ingredients, flipping through or searching for the information is very tedious. To solve this, we created the "AI Gaze Interactive Menu"—customers only need to "take a look" at the drink they are interested in, and the screen automatically pops up a detailed introduction with images and text, making information access intuitive and efficient. For customers, this is a contactless and more convenient way to query information; for merchants, this novel interaction not only immediately catches customers' attention but also displays product information intuitively and vividly, making products easier to be "seen."
This project, which originated in food service, can also flexibly "transform" to fit more commercial and daily-life scenarios: in smart retail, it can count which products customers are more interested in, providing data to help merchants optimize displays; in smart cockpits, it can monitor whether the driver's gaze deviates from the road ahead (fatigue monitoring); in medical assistance, it can help patients with limited mobility control devices with their eyes, improving their quality of life.
1.2 Demo Video
2.Project Implementation Principle
Its core logic is as follows: first, have the HUSKYLENS 2 camera "learn" the user's gaze direction, which is equivalent to creating a "gaze direction whitelist" for the device. Write a program to determine the gaze direction ID and display the corresponding beverage information on the UNIHIKER K10. When the user looks at a dish on the menu again, if it is a valid learned gaze direction, the screen will automatically display the corresponding beverage's info card.
3. Hardware and Software Preparation
3.1 Hardware and Software List
SOFTWARE
Mind+ Programming Software (Version: 2.0)
Note: The Mind+ programming software version is v2.0, and the HUSKYLENS version is HUSKYLENS 2.
3.2 Hardware Connection
Please connect the computer, UNIHIKER K10, and HUSKYLENS 2 according to the connection diagram below.
3.3 Software Platform Preparation
Download and install the Mind+ 2.0 or later installation package from the official website. After installation, double-click to open it.
4. Project Development
4.1 HUSKYLENS 2 Operation
First, select the protocol type for HUSKYLENS 2.
Tap System Settings → Protocol Type → select I2C communication mode, then return to the main menu interface.
Next, swipe the screen to find the "Eye Gaze" function.
Point HUSKYLENS 2 at a scene containing a face. An arrow appears near the human eye; the direction of the arrow represents the gaze direction, and the arrow points to the gaze landing point. At the same time, an angle value and a numerical value appear on the screen, which are the angle and length of the gaze point projected onto the plane. For example, the screen shows (350°, 247), meaning that after the gaze direction is projected onto the detection plane, the angle between the projection and the reference line is 350°, and the normalized length value in that projection direction is 247.
After completing the parameter configuration and mode selection in HUSKYLENS 2, please follow the steps below to collect and learn gaze direction data:
Aim at the gaze direction you want to learn, then press the A button in the upper-right corner of HUSKYLENS 2 to learn that gaze direction.
After learning is complete, if a learned gaze direction is recognized, a colored arrow appears, and the ID and confidence appear after the angle and length data. For example: 350° 246.
Repeat the above steps to learn the user's gaze directions; the gaze preparation is then complete.
Note: Because everyone's eye shape and lighting conditions differ, if you find during testing that looking down is recognized as looking up, appropriately lower the detection threshold and recognition threshold at the bottom of the camera screen.
For more detailed usage of HUSKYLENS 2, see the following website:
https://wiki.dfrobot.com.cn/_SKU_SEN0638_Gravity_HUSKYLENS_2_AI_Camera_Vision_Sensor
Once the model is ready, you can start writing code to create an AI smart menu with automatic gaze recognition.
4.2 Code Implementation
Open the Mind+ programming software and select "Upload Mode" under "Coding".
First, we need to add the corresponding extensions in Mind+, including UNIHIKER K10 and HUSKYLENS 2.
Go to the "Extensions" page and search for "UNIHIKER K10".
Click the "Download" button on the extension package and wait for the download to complete.
Click the extension package to finish loading.
Follow the same steps used to load "UNIHIKER K10" to load the module library "HUSKYLENS 2 AI Camera".
After the "UNIHIKER K10" and "HUSKYLENS 2 AI Camera" module libraries are loaded,
click the "Back" button to return to the programming interface.
Click "Connect" in the Connect Device section.
The connection success screen is shown below. Click "×" to return.
After the extension libraries are loaded, write the program as follows:
The core code is explained below:
The complete program file for this project is provided in the attachment. Expand the project and click "Load Project".
Select the project in the attachment and click "Open".
Click "Upload" to run the program.
The result is as follows:
5. Attachment List
Code.zip 
