Offline voice command recognition on the Seeed Xiao ESP32-S3 Sense with the Round Display expansion. Say the wake word Jarvis, then speak a command — the recognized phrase appears as text on the 240×240 round LCD.

Built with ESP-IDF 5.5.x, ESP-SR (WakeNet + MultiNet6), and LVGL.

Before powering on:

1. Stack the Sense expansion board on the Xiao (provides the microphone).
2. Stack the Round Display on top.
3. Set the Round Display power switch to ON (backlight and power path).

• ESP-IDF v5.5.x (tested with 5.5.4)
• VS Code or Cursor with the Espressif ESP-IDF extension
• Windows: ESP-IDF Windows setup guide or use the extension’s guided installer

Pick one of these paths:

Option A — ESP-IDF Tools Installer (Windows, recommended for beginners)

1. Follow the Windows ESP-IDF installation guide.
2. Install ESP-IDF v5.5.x and the esp32s3 target when prompted.
3. Note the install path (e.g. C:\esp\v5.5.4\esp-idf).

Option B — Configure via VS Code / Cursor extension

1. Install the ESP-IDF extension (espressif.esp-idf).
2. Open the Command Palette (Ctrl+Shift+P).
3. Run ESP-IDF: Configure ESP-IDF extension.
4. Choose Express or Advanced install.
5. Select ESP-IDF v5.5.x and target esp32s3.

Extension install guide: vscode-esp-idf-extension tutorial

In VS Code or Cursor:

1. Open Extensions (Ctrl+Shift+X).
2. Search for Espressif IDF.
3. Install ESP-IDF by Espressif Systems.

If prompted, also install the recommended extensions from .vscode/extensions.json.

File → Open Folder → esp32-multinet-demo

1. Command Palette → ESP-IDF: Select ESP-IDF Path (or ESP-IDF: Configure ESP-IDF extension).
2. Set idf.currentSetup to your ESP-IDF directory, for example:
   • C:\esp\v5.5.4\esp-idf
3. Confirm the chip target is esp32s3:
   • Command Palette → ESP-IDF: Set Espressif Device Target → esp32s3

Workspace settings in .vscode/settings.json already set IDF_TARGET to esp32s3. Update idf.currentSetup to match your machine.

1. Plug in the Xiao via USB.
2. In the ESP-IDF status bar, click the port item and choose your board (e.g. COM4), or leave it on detect.

Important: Logs use USB Serial/JTAG, not the external UART pins. GPIO 43/44 are used by the round display. This is configured in sdkconfig.defaults.

1. Click Build (hammer icon) in the ESP-IDF status bar, or
2. Command Palette → ESP-IDF: Build your project

Or use Terminal → Run Task → ESP-IDF: Build.

.\scripts\build.ps1

Or manually:

. C:\esp\v5.5.4\esp-idf\export.ps1   # adjust to your ESP-IDF path
$env:PYTHONIOENCODING = "utf-8"      # required on Windows for esp-sr model packaging
cd path\to\esp32-multinet-demo
idf.py set-target esp32s3
idf.py build

• Downloads managed components (esp-sr, LVGL, GC9A01 driver) via the ESP Component Manager.
• Packages ~5 MB speech models into build/srmodels/srmodels.bin — the first build can take several minutes.
• Board defaults come from sdkconfig.defaults. If configuration looks wrong, delete sdkconfig and rebuild.

You do not need idf.py menuconfig for a standard run.

1. Select the correct COM port in the status bar.
2. Click Flash (lightning icon).
3. Click Monitor (plug icon), or use ESP-IDF: Monitor your device.

Or Terminal → Run Task → ESP-IDF: Build, Flash and Monitor.

.\scripts\flash-monitor.ps1 -Port COM4   # replace COM4 with your port

Or:

. C:\esp\v5.5.4\esp-idf\export.ps1
$env:PYTHONIOENCODING = "utf-8"
idf.py -p COM4 flash monitor

idf.py flash writes three images: bootloader, application, and the srmodels partition (required for speech recognition).

Press Ctrl+] to exit the serial monitor.

Boot          →  Display shows: Say "Jarvis"
Say "Jarvis"  →  Display shows: Jarvis!  then  Listening...
Say a command →  Display shows the recognized phrase
Silence       →  Returns to: Say "Jarvis"

Built-in commands (defined in main/speech_recognition.c):

Speak clearly, about 30 cm from the microphone. The USB serial log shows wake and command events for debugging.

Edit register_speech_commands() in main/speech_recognition.c, then rebuild and reflash.

main/
  main.c                 Application entry
  board_audio.c          PDM microphone (GPIO 41/42)
  speech_recognition.c   WakeNet + MultiNet pipeline
  display_ui.c           Round display text UI
  round_display.c        GC9A01 + LVGL init
  idf_component.yml      Managed dependencies
sdkconfig.defaults       Board and SR model defaults
partitions.csv           App + speech model partitions
dependencies.lock        Pinned component versions (keep in repo)
scripts/
  build.ps1              Windows build helper
  flash-monitor.ps1      Windows flash + monitor helper

• ESP-IDF Programming Guide (ESP32-S3)
• ESP-IDF Get Started
• ESP-SR / MultiNet documentation
• ESP-IDF VS Code extension docs
• Xiao ESP32-S3 Sense microphone
• Round Display hardware usage