Inspiration
In the Odyssey, no one could ignore the Sirens' voice. Many people live with the opposite: ALS, stroke, cerebral palsy, spinal cord injury, MS and brain injuries can take away speech and fine motor control. Traditional AAC devices are often expensive, rely on slow linear scanning, and speak in a generic synthetic voice. We wanted to build a low-cost, context-aware system that cuts the number of selections it takes to say something, and speaks in the user's own voice.
What it does
Sirenize is a head-tilt-controlled AAC system with AI-driven phrase prediction.
- Single-signal input: a GY-521 (MPU-6050) atturns head roll (left/right) into scan navigation and a forward nod into selection. It's a switch-scanning paradigm driven by one body movement.
- Vision-based prediction: frames from the device camera go to Gemini, which returns 4 context-ranked phrases (e.g. a breathing mask → "Adjust my mask"). They're placed at the front of the scan order, so the most likely phrase is one move away.
- Conversational replies (Listen mode): speech-to-text transcribes what someone says to the user; Gemini generates 3 reply candidates plus a "Something else" escape, conditioned on the user profile, the current scene and recent phrases.
- Personal voice (voice banking): ElevenLabs Instant Voice Cloning from a fresh recording or from old voicemails and videos, for users who have already lost their speech. All output is spoken in that voice.
- Multi-device, server-authoritative sync: iPhone and iPad clients mirror one shared state; an active-device lease decides which client owns audio output and the camera.
- Edge deployment: the whole stack runs on a battery-powered Arduino UNO Q. No laptop.
- Graceful degradation: phrase audio cached on the device, a fixed-grid fallback, and simple answer sets when the AI is unreachable. It never goes silent.
How we built it
Sensing and firmware
- A GY-521 (MPU-6050) accelerometer on a cap or glasses. The Arduino UNO Q's STM32U585 microcontroller reads it over I²C at 50 Hz (±2 g, with the chip's built-in ~44 Hz low-pass filter). We prototyped on an Elegoo Mega 2560.
- Roll and pitch are computed from the gravity vector with
atan2, smoothed with an exponential moving average (α = 0.2), and measured relative to a neutral pose captured in a one-second calibration, so the sensor doesn't need to sit perfectly straight. - Gesture detection: a gesture fires once the head passes 12° for 150 ms, and it can't fire again until the head returns within 7° of neutral (hysteresis). Each tilt fires exactly once, and talking or small head movements don't cause false triggers. The larger sideways movement wins left vs. right, and select fires only on a forward nod.
- Built to keep running: gesture events and a heartbeat every 2 s cross from the microcontroller to the Qualcomm Dragonwing Linux side (Debian) over the board's Bridge (RPC). The firmware detects a frozen sensor after a power dip and re-initializes it automatically.
Backend
- Python aiohttp (asyncio) server: REST endpoints plus a WebSocket pub/sub channel. Server-authoritative state (highlight index, suggestion row, active question, active device, last utterance), with full-state snapshots pushed on reconnect.
- systemd services for the server, the Bridge forwarder and a Cloudflare Tunnel (TLS, needed for camera access in iOS Safari), with auto-start on boot and restart-on-failure.
- Health telemetry: sensor heartbeat, client counts, and last-call status for each external API.
AI pipeline
- Gemini (
gemini-3.1-flash-lite, minimal thinking, 4 s timeout): camera frames go in as JPEG and come back as structured JSON, 4 short phrases ranked for the scene. The prompt includes the user's profile and recent phrases, and output is validated server-side before it reaches the screen. - Smart frame sending: frames are downscaled and only sent when the scene actually changes, one request at a time, with backoff on rate limits.
- ElevenLabs Instant Voice Cloning: a clone of the user's voice. Our test clone was built from about 4 minutes of audio in 77 seconds.
- ElevenLabs text-to-speech (
eleven_flash_v2_5): every phrase is spoken in the user's cloned voice. The phrase library is cached on the device for offline use, and new reply options are prefetched before the user selects them. - ElevenLabs speech-to-text (
scribe_v2): transcribes what someone says to the user in Listen mode (0.45–0.81 s per call), with the browser's Web Speech API as a fallback, so Gemini can turn it into reply options.
Built With
- aiohttp
- arduino
- arduino-mega
- arduino-uno-q
- c++
- cloudflare
- css3
- cursor
- debian
- elevenlabs
- gemini
- google-gemini-api
- gy-521
- html5
- ios
- javascript
- linux
- mpu-6050
- python
- react
- websockets