Object Recognition-Based Visually Impaired-friendly Item-finding Assistant

1.1 Project Introduction

In our daily lives, finding items is a seemingly trivial task—when an umbrella goes missing, we can quickly locate it by looking around. However, for people with visual impairments, this process is quite challenging. Unable to rely on vision to determine orientation, they often need to fumble around or ask for help constantly.

What if technology could help them "find"? This project is designed with this in mind. By leveraging the object recognition capability of HuskyLens 2 and integrating it with the computing and voice interaction features of UNIHIKER K10, we’ve created an "item-finding assistant" that can understand commands, automatically scan, and indicate direction.

When a user issues the command "Find the umbrella", the servo rotates the HuskyLens 2 module to scan the environment. Upon recognizing the target, UNIHIKER K10 plays an audio segment to indicate the orientation, such as "This item is right in front of you". This solution helps visually impaired individuals quickly locate frequently used items, making technology truly their "second eyes".

1.2 Demo Video

2 Project Implementation Principle

This project leverages the Object Recognition capability of HuskyLens 2 combined with the main control computing power of UNIHIKER K10 to construct a visually impaired-friendly object locator. Users can trigger object recognition through voice commands. A servo motor drives HuskyLens 2 to rotate, expanding the search range. HuskyLens 2 performs object learning and real-time inference locally, sending recognition results to UNIHIKER K10 via I2C communication. Upon receiving the data, the main controller determines whether an object exists and the servo's orientation: if detected, it feeds back to the user through a voice synthesis module (e.g., "This item is on you front left"); if no object is recognized, it continues controlling the servo to rotate the camera for continuous searching.

3. Hardware and Software Preparation

3.1 Equipment List

Note: Huskylens requires Version 2

3.2 Hardware Connection

Make connections by referring to the diagram below.

3.3 Software Preparation

Download and install the Mind+ installation package (Version 2 or above) from the official website. Double-click to open it after installation.

4. Project Making

4.1 Huskylens 2

First, select the protocol type for HuskyLens 2.
Tap System Settings -> Protocol Type -> Select I2C communication mode, then return to the main menu interface.

Second, swipe the screen to find the "Object Recognition" function.

When HuskyLens 2 "looks" at an identifiable object, observe its screen. All recognized objects will be enclosed in white bounding boxes, with their corresponding object names and confidence levels displayed. In machine learning, confidence level refers to the model's "certainty" in its prediction. For instance, "apple 36%" indicates that under the object recognition function, the model calculates a 36% probability that the object in the current frame is an "apple".

Please complete the incremental learning process by following these steps:

Direct HuskyLens 2 at the target object. After the object is enclosed in the white box, adjust HuskyLens 2's angle so the "+" cursor lies within the white box. Press the top-right Button A to confirm learning this object.

Upon successful learning, if a previously trained object is detected, the screen will highlight it with a colored bounding box, and display "name: IDx Confidence" above the object. For example, "cup: ID1 85%". Repeat this process to train the target objects of this project including the umbrella as ID1 and the cup as ID2.

For more detailed usage of HuskyLens 2, please refer to the following URL:
https://wiki.dfrobot.com/_SKU_SEN0638_Gravity_HUSKYLENS_2_AI_Camera_Vision_Sensor

After completing the parameter configuration and training, you can start writing code to implement the visually impaired-friendly item-finding assistant based on object recognition technology.

4.2 Programming

Open the programming software Mind+, then select "Upload Mode" in the "Coding" tab.

Next, add the required extensions in Mind+, including UNIHIKER K10 and HuskyLens 2.
Enter the "Extensions" page, switch to "Board" tab, search for "K10", and click "UNIHIKER K10".

Load the "HuskyLens 2 AI Camera" and "Micro Servo" libraries by the same way from "Module" page.

Click the "Back" button to return to the programming interface.

Click the "Connect Devices", choose your device and "Connect".

Audio preparation:
Prepare an SD card (16GB or 32GB) for the audio files. Save 3 audio files, which you can find in the attachment, to your SD card. Insert this SD card into your UNIHIKER K10 SD card slot.

Then write the program as follows:

The analysis of the core code is as follows:

There is a complete program file for this project in the attachment. (Note: The .mp file is compatible with both Mind+ v1.x (e.g., v1.8.1) and v2.x (e.g., v2.0), while the .mpcode file only works with Mind+ v2.0)
Open Project->Open Local File to load project.

Select the project in the attachment and click "Open".

Click "Upload" to run the program.

The effect is as follows:

5 Attachment