Pose-Recognition-Driven Musical Instrument Variation with HUSKYLENS 2

1. Project Introduction

1.1 Project Design

This project, based on HUSKYLENS 2 and Arduino UNO, develops a "Musical Instrument Variation Device." The system uses the classic music "Jingle Bells" as material, pre-separates the performance tracks of three instruments (Guitar, Flute, Violin) from it, and through real-time recognition of the user's instrument-playing postures (such as violin, flute, guitar), synchronously plays the corresponding instrument's performance clips to achieve real-time synchronization between posture and sound. For example, when the system recognizes a violin-playing posture, it automatically plays the violin part of "Jingle Bells," enabling users' movements to resonate vividly and intuitively with familiar music.

The system aims to transform users' perception of musical instruments into an engaging interactive experience, via real-time auditory feedback, elevating "posture imitation" to "simulated performance." This gamified design encourages users to shift from passive observation to active participation, naturally stimulating interest in music and exploration within an intuitive and relaxed operating environment.

1.2 Demo Video

2. Implementation Principle

The core logic is as follows:
HUSKYLENS 2 captures and recognizes the user's instrument-playing posture and transmits the recognition result to Arduino UNO. Arduino UNO, based on the recognized posture, command HUSKYLENS 2 to play audio matching this pose through the onboard speaker.

3. Hardware and Software Preparation

3.1 Equipment List

3.2 Hardware Connection

Follow the connection diagram below to connect the computer, HUSKYLENS 2, and Arduino UNO.
 

Store the audio materials in the path shown in the figure below; they can be downloaded from the "Attachment" at the end of the article.

3.3 Software Preparation

Download and install the Mind+ installation package (Version 2 and above)  from the official website. Double-click to open it after installation.
 

4. Project Making

4.1 HUSKYLENS 2

First, select the protocol type for HUSKYLENS 2.  
Tap System Settings -> Protocol Type -> Select I2C communication mode, then return to the main menu interface.

Swipe the screen to find the "Pose Recognition" function.

Aim HUSKYLENS 2 at an image containing humans. Upon detecting humans, the screen will display white bounding boxes enclosing all humans in the scene, and 17 key points on the human bodies will be marked using dots.

Position the human pose you want to learn, adjust the viewing angle of HUSKYLENS 2 so that the crosshair in the center of the screen is within the white box, then press Button-A on the top-right corner of HUSKYLENS 2 to learn this pose.

After learning a pose, when a learned pose is recognized, the screen will frame it with a colored box and display "Pose:IDx 90%" above it, for example, "Pose:ID1 78%". Here, "name" defaults to "Pose" by default (customize the name via "Parameter Settings"). "ID1" refers to the first pose learned, and "78%" represents the confidence level, indicating the model's confidence in the recognized pose belonging to the learned set (e.g., "ID1 85%" means the model is 85% confident it is Pose ID1). The same logic applies to learning additional poses.

For more detailed usage of HUSKYLENS 2, please refer to the following URL:
https://wiki.dfrobot.com/_SKU_SEN0638_Gravity_HUSKYLENS_2_AI_Camera_Vision_Sensor

4.2 Programming

Open the programming software Mind+, choose "Coding" mode, then click "Upload" to create a new project.
 

Next, add the required extensions in Mind+, including Arduino UNO, HUSKYLENS 2.
Enter the "Extensions" page, switch to "Board" tab, search for"Arduino UNO", and click "Arduino UNO".
 

           

Load the "HUSKYLENS 2 AI Camera" library by the same way from "Module" page.
 

     

Click the "Back" button to return to the programming interface.
 

Click the "Connect Devices", choose your device and "Connect".
 

     

After the device is successfully connected, write the program as follows:
 

The analysis of the core code is as follows:
 

There is a complete program file for this project in the attachment. (Note: The .mp file is compatible with both Mind+ v1.x (e.g., v1.8.1) and v2.x (e.g., v2.0), while the .mpcode file only works with Mind+ v2.0)
Open Project->Open Local File to load project.
 

Select the project in the attachment and click "Open".
 

Click "Upload" to run the program.
 

The effect is as follows:
 

5. Attachment