Creating a Marionette Powered by HUSKYLENS 2 Hand Recognition

auroraAA Feb 05.2026

0 22 Easy

Introduction

This project uses the HUSKYLENS 2 AI vision sensor to build a human-computer interaction system centered on gesture recognition. The system detects the 3D coordinates of key hand points (finger tips) in real time, and converts the movement paths of finger tips into the marionette's actions on the screen. It lets us experience the easy and direct control of "using hands as the strings for the marionette".

What You’ll Need

Hardware Preparation

To follow this tutorial, here’s the hardware you’ll need to get ready:

HARDWARE LIST

1 HUSKYLENS 2

Link

Check HUSKYLENS 2 firmware version: System Settings → Device Information.

If the firmware version is too old, update it to System Version 1.2.1 and above in time. For firmware updates, check out this guide:
https://wiki.dfrobot.com/_SKU_SEN0638_Gravity_HUSKYLENS_2_AI_Camera_Vision_Sensor#7.1%20%20Latest%20Version

Software Preparation

Ensure Python and pip (Python package manager) are installed and configured in your system/IDE (works for PyCharm, VS Code, IDLE, Jupyter Notebook, etc.).

How it works

Step 1: Setup RNDIS Driver for PC

To use the wired real-time video streaming function, install the RNDIS driver first. If the RNDIS driver is not already installed on your computer, you will need to install it first. The steps for Windows 11 are as follows: for other systems, if the operation fails as described below, please refer to the system version for guidance or contact technical support.

Step 2: Make Connections

Step 3: Setup video streaming to get result from HUSKYLENS 2

Swipe the HUSKYLENS 2 screen to find "Video Streaming". If not found, perform a firmware update first. In the Video Streaming menu, toggle the WebRTC Streaming switch to ON and select YES.

This will automatically return to the main interface. The top-right status bar will show a real-time video streaming icon, indicating that you are currently in video streaming mode.