Musical Shoes are smart insoles that transform your foot pressure into interactive music and engaging multiplayer games.
Software apps and online services
- TouchDesigner- Microsoft Visual Studio Code Extension for Arduino
Hand tools and fabrication machines
- 3D Printer (generic)
StoryMusical Shoes
Musical Shoes are smart insoles transform your foot pressure into interactive music and engaging multiplayer games.
Introduction
Musical Shoes is a project that transforms ordinary footwear into interactive musical instruments. The system detects pressure at specific points on the feet by embedding three Force Sensitive Resistors (FSRs) into custom-designed insoles. An ESP32 microcontroller then processes these pressure readings to generate dynamic music and synchronized visuals. The project offers two distinct modes of interaction:
- One-PlayeMode: In this mode, users can freely create music by applying pressure to different points on their feet. The pressure data is visualized in real-time using TouchDesigner, while Ableton Live generates corresponding musical outputs, allowing for an immersive and creative musical experience.
- Two-Player Game Mode: This competitive mode involves two players who compete by pressing designated pressure points as quickly as possible. A web-based interface guides users on where to step and apply pressure, and the system tracks and scores their performance based on speed and accuracy. The player with the most points at the end is declared the winner, making it an engaging and interactive game.
The integration of wearable technology with music and gaming not only showcases the potential of interactive wearables but also provides a unique platform for both creative expression and competitive gameplay.
Components
Hardware for1 foot (x4)
- Force Sensitive Resistors (FSRs) x3
- ESP32 Microcontroller
- 3D-Printed Casing for ESP32 and Battery
- Batteries
- Standard Insoles
Software
- VSCode & Platform.io
- TouchDesigner
- Ableton Live
- HTML, CSS, JavaScript
- WebSockets
Project PhasesMilestone 1: Ideation and Planning
During the initial phase, our team engaged in extensive brainstorming to explore various project ideas, including flip dots for visual displays, bio knitting with sensory data, haptic integration with TouchDesigner, and robotic arms for automated tasks. Despite the creativity of these concepts, none fully aligned with our feasibility criteria. After thorough discussions and evaluations, we unanimously decided to pursue the "Musical Shoes" project
Milestone 2: Initial Design and Prototyping
In the second month, we focused on developing an initial prototype for the Musical Shoes. Our preliminary approach involved integrating vibration motors and an IMU to provide haptic feedback for teaching ballroom dancing. However, extensive testing revealed significant challenges, including difficulties in localizing vibrations accurately and inaccuracies in the IMU data due to drift over time.
These issues hindered the effectiveness of our initial design, leading us to consult with our TA. Based on this feedback and the technical limitations encountered, we decided to pivot away from the ballroom dancing concept and refocus on leveraging pressure data from FSRs to generate music and facilitate interactive gameplay.
Milestone 3: Development and Integration
In the final month, we made substantial progress in developing the Musical Shoes. We finalized the design by integrating three FSRs into each insole to capture pressure data at the toes, heel, and side of the foot. To ensure seamless data transmission, we implemented wireless communication using WebSockets, allowing the ESP32 to communicate with the computer via Wi-Fi. This wireless setup eliminated the need for cumbersome data cables and enhanced the user experience.
For the One-Player Mode, we connected the FSR data to TouchDesigner to create real-time visualizations using an L-system, which simulates plant growth based on user input. Simultaneously, Ableton Live processes the pressure data to generate music, with different sounds triggered by varying pressure thresholds. This mode allows users to explore their creativity by manipulating pressure points to produce unique musical compositions.
In the Two-Player Game Mode, we developed a web interface that guides users on where to step and apply pressure. The system calibrates each user's weight to ensure fair gameplay and then generates random pressure points that players must press quickly. Points are awarded based on the speed and accuracy of each player's responses, fostering a competitive and engaging environment. This mode leverages WebSockets to handle real-time data transmission and scoring, ensuring a responsive and interactive gaming experience.
Additionally, we completed the physical fabrication of the insoles and the ESP32 casing. The casing was 3D-printed to securely house the ESP32 and battery, featuring a clip mechanism for easy attachment to the shoes. This design ensures that the system remains compact, comfortable, and unobtrusive, allowing users to wear the Musical Shoes without discomfort.
Custom parts and enclosures - Insole Design
Designing the insoles was a key part of making the Musical Shoes both comfortable and functional. Here's how we approached it:
- Choosing the Right Materials:We picked high-quality, durable insoles that are comfy and built to last. The materials we chose are perfect for long-term use, so users can enjoy the Musical Shoes without any discomfort or the insoles wearing out quickly.
- Integrating the FSRs:
We strategically placed three Force Sensitive Resistors (FSRs) to catch a variety of pressure points. One is near the toes to detect forward pressure, another near the heel for backward pressure, and the third is on the side of the foot to monitor sideways movements. This setup helps us gather detailed pressure data for both making music and playing games.
- Layering and Protecting the Sensors: To keep things comfortable and safe, we stuck the FSRs to one side of the insole and then added another insole layer on top. We didn’t stop there—we hand-stitched the two layers together. This makes the insole sturdy and keeps the sensors in place during use. Plus, this layering and stitching not only protects the sensors but also gives the insoles a sleek, low-profile look.
- Managing the Wires: We made sure the wires from the FSRs were neatly routed out of the shoe to avoid any tangling or discomfort. The wires are organized carefully to stay flexible and durable, so they don’t get in the way or wear out too soon.
- Designing the ESP32 Casing:
The ESP32 and battery are tucked inside a custom 3D-printed case. We designed this case to be small and lightweight, so it doesn’t bulk up the shoe. It has a clip mechanism that makes it easy to attach and remove from the shoe. Plus, the case has a lid that you can open, making it simple to charge or replace the battery without messing up the overall design.
- Keeping It Stylish:
We didn't just focus on how the insoles work—we also wanted them to look good. The design blends seamlessly with regular shoes, keeping their original style intact. This way, users can wear the Musical Shoes anywhere without drawing too much attention or sacrificing style.
By paying attention to these details, we created insoles that are not only tech-savvy but also comfortable and stylish. The Musical Shoes are easy to use and enjoyable to wear, making the whole experience seamless and fun.
Code
Find attached files.
Explanation
- Wifi Connection: The ESP32 connects to the specified Wi-Fi network using the provided SSID and password.
- WebSocket Setup: Initializes a WebSocket client that connects to the specified server IP and port. The webSocketEvent function handles connection events.
- Data Transmission: In the loop function, the ESP32 continuously reads analog values from the three FSRs and constructs a JSON payload containing these values. This payload is then sent to the WebSocket server, enabling real-time data transmission for further processing in TouchDesigner or the web interface.
Touch Designer
We used TouchDesigner, a versatile node-based visual programming tool, for the single-player game to design an immersive audiovisual experience that reacted dynamically to user input.
The shoe sensor data was transmitted via a WebSocket connection to TouchDesigner, where it was processed and utilized for two core purposes: reactive audio and visualizations.
Explanation
- Reactive Audio: The pressure data from the six FSRs was thresholded into binary signals, which were used to trigger MIDI notes. These signals were routed through TD Ableton, enabling TouchDesigner to interface seamlessly with Ableton Live using OSC data. This setup created an evolving soundscape that responded intuitively to the player's movements, adding a layer of sonic immersion.
- Dynamic Visuals: Using an L-system to create recursive particle structures, we crafted fractal tree visualizations that grew in response to the pressure data received from the shoes. The intensity of pressure directly influenced the growth patterns of the fractal trees, resulting in visuals that felt alive and organically connected to the user's actions.
Case Design (AutoDesk Fusion)
Demo
Challenges & Solutions
Building the Musical Shoes wasn't without its bumps along the way. Here are the main challenges we faced and how we tackled them:
Initial Conceptualization and Team Consensus
- Challenge: At the start, everyone on the team had different ideas. Some wanted to use flip dots for visual displays, others were excited about bio-knitting with sensory data, some thought haptic integration with TouchDesigner was the way to go, and a few were interested in building robotic arms for automated tasks. Getting everyone to agree on one direction was tough.
- Solution: We held structured brainstorming sessions where each person could share their ideas and talk about what they'd need and what the impact could be. By comparing each idea against our project goals and what was realistically doable, we all decided that Musical Shoes was the best fit. It combined wearable tech, music, and interactive gaming in a way that felt both unique and achievable.
Pivoting from the Initial Idea
- Challenge: We designed a compact and lightweight 3D-printed casing for the ESP32 and battery, so it doesn’t make the shoe bulky. The insoles are layered and hand-stitched to protect the sensors and wires while keeping the surface smooth and comfy. By focusing on a sleek and unobtrusive design, we made sure the Musical Shoes are comfortable to wear for long periods and look good with any style of shoe.
- Solution: We designed a compact and lightweight 3D-printed casing for the ESP32 and battery, so it doesn’t make the shoe bulky. The insoles are layered and hand-stitched to protect the sensors and wires while keeping the surface smooth and comfy. By focusing on a sleek and unobtrusive design, we made sure the Musical Shoes were comfortable to wear for long periods and looked good in any style of shoe.
Pressure Data Localization and Sensor Placement
- Challenge: We needed the FSRs to accurately capture pressure points for both making music and playing games. Our initial sensor placements were inconsistent, which messed up the system‘s performance and user experience.
- Solution: We did a bunch of experiments to find the best spots for the FSRs. By placing them near the toes, heel, and side of the foot, we got much more consistent and reliable pressure readings. We also improved the insole design to keep the sensors securely in place, so they wouldn’t move around and mess up the data.
Implementing Reliable Wireless Communication
- Challenge
: At first, we tried connecting the ESP32 directly to the computer with a data cable. This wasn’t practical for a wearable device because it limited how much users could move and made the setup bulky.- Solution: We switched to using WebSockets for wireless communication, allowing the ESP32 to send data over Wi-Fi. This made the setup much more mobile-friendly and got rid of the annoying cables. We made sure the wireless connection was stable and had low latency, which is crucial for real-time music creation and gameplay.
Developing Intuitive Game Mechanics
- Challenge: Creating a game that was both easy to understand and engaging was tricky. We wanted to balance the creative side of making music with the competitive aspect of gameplay to appeal to a wide range of users.- Solution: Through lots of trial and error, we developed a dual-mode system with clear, distinct experiences for one-player and two-player modes. For the one-player mode, we focused on letting users freely create music by changing their foot pressure, which boosted creativity. For the two-player mode, we designed a competitive game where players quickly and accurately press designated pressure points to earn points. This balance made both modes fun and functional, catering to different preferences.
Calibration and Handling User Variability
- Challenge: People have different weights and walking styles, which can mess with the accuracy and consistency of the pressure data.- Solution: We added a calibration step in the software that removes the effect of the user’s weight, so the pressure data is based solely on the force they apply. This makes the game fair for players with different weights. The calibration process is easy to use, allowing quick adjustments without needing any technical know-how.
Maintaining Comfort and Aesthetics
- Challenge: We wanted to integrate all the electronic components into the insoles without making them uncomfortable or ruining the look of the shoes.- Solution: We designed a compact and lightweight 3D-printed casing for the ESP32 and battery, so it doesn’t make the shoe bulky. The insoles are layered and hand-stitched to protect the sensors and wires while keeping the surface smooth and comfy. By focusing on a sleek and unobtrusive design, we made sure the Musical Shoes were comfortable to wear for long periods and looked good in any style of shoe.
SchematicsCircuit
Code
This article was first published on Hackster on December 15, 2024
cr: https://www.hackster.io/cmsc730fall24/musical-shoes-acf700
Author: Team CMSC730Fall24:Yu Chi Mei•Shruti Muralidas
