Sounair

Overview

Sounair is a futuristic iOS music concept app that lets users control music without touching the screen. Instead of relying on taps and buttons, the app uses the front camera, Vision hand pose detection, and a custom Core ML model to recognize hand gestures in real time. The project explores a more natural, touchless way to interact with music. Users can play, pause, and select tracks through hand gestures and hover-style movement, turning music control into a camera-driven interaction experience.

Problem

Most music apps still depend on direct screen interaction, which is not always convenient. If your hands are busy, dirty, or occupied while cooking, working, or moving around, even a simple action like pausing or changing a track becomes annoying. Sounair was built to explore whether music playback could feel more ambient and hands-free by using on-device gesture recognition instead of touch controls.

My Role

I designed and built the iOS app, including the SwiftUI interface, camera pipeline, gesture recognition flow, custom Core ML integration, audio playback logic, hover-based track selection zones, and visual feedback system. I also trained and integrated the custom hand gesture model using Create ML and connected it with Vision so the app could classify gestures from live camera frames.

What I Built

I built a SwiftUI music player with a 3x3 track grid, bundled audio files, custom typography, animated visual feedback, and a live camera-powered control mode. The app uses hand gestures such as palm, fist, peace, and pointing gestures to trigger playback actions. The music player supports play, pause, and track selection through gestures. Palm starts playback, fist pauses playback, and pointing/peace gestures work with finger-position zones to select tracks from the grid. I also built the live camera preview layer, connected it to Vision hand pose detection, passed detected hand keypoints into a custom Core ML model, and dispatched playback actions only after repeated gesture confidence checks to reduce accidental triggers.

Technical Highlights

Sounair uses AVCaptureSession to capture live frames from the front-facing camera and VNDetectHumanHandPoseRequest to detect hand pose data from each frame. The detected keypoints are converted into model input and passed into a custom Core ML hand pose classifier. The gesture pipeline includes confidence handling and repeated gesture confirmation before triggering actions, which helps prevent noisy or accidental playback controls. For track selection, the app maps the index fingertip position into normalized grid zones and plays the matching track when the finger hovers over a song tile. Audio playback is handled with AVFoundation using AVPlayer instances for bundled music files. The app also updates playback metadata through MediaPlayer/Now Playing APIs and uses Combine-published state to keep the SwiftUI interface reactive. The UI includes a camera-backed music control screen, glowing current-track feedback, a music visualizer animation, custom fonts, and a minimalist Apple-inspired layout.

What I Learned

Sounair taught me how to combine camera input, machine learning, gesture recognition, and audio playback into one real-time iOS experience. I learned how to work with Vision hand pose detection, Core ML model predictions, AVCaptureSession frame handling, AVFoundation playback, gesture debouncing, and SwiftUI state updates. More importantly, it helped me think about interaction design beyond normal buttons and taps.