Your iPhone sees your hands in mid-air. Grab, rotate and sculpt 3D objects - no touch, no controllers, no headset.
TrueDepth camera, Apple Vision, 120 Hz SceneKit rendering.
Quick start · Features · Gestures · How it works · Русский
![]() Object mode pinch or fist pulls the nearest object into your hand |
![]() Sculpt mode six brushes, colours, undo, export |
![]() Brushes clay, neon, silk, stardust, chrome, plasma |
Note
Requires a physical iPhone with Face ID (TrueDepth camera). The Simulator supports replay mode only (recorded or synthetic hands through the same pipeline). Developed and tested on an iPhone Air; other Face ID models are untested.
Hand tracking on a phone usually means a flat 2D skeleton from the back camera. HandGrab uses the front Face ID camera: Vision gives 21 hand joints, the TrueDepth map adds real depth, and the two are fused into a 3D hand with fixed bone lengths. The result is low latency (hand pose ready about 46 ms after exposure, measured on-device) and enough precision to actually draw in the air.
| 🖐 | Grab and throw four objects on a table with a pinch or a fist |
| 🎨 | Draw and sculpt in space with six brushes: clay, neon, silk, stardust, chrome, plasma |
| 🤲 | Two hands, independent: draw with both, rotate with one fist, scale with two |
| ⚡ | 120 fps rendering, hand pose ~46 ms after exposure (camera floor 33-38 ms) |
| ↩️ | Undo / redo, smoothing, X-mirror, export to Photos, USDZ and .hgsculpt |
| 🧪 | Test without a phone: record a session, replay it in the Simulator, or run synthetic hands |
Brush details: docs/sculpt.md (in Russian).
| Gesture | Object mode | Sculpt mode |
|---|---|---|
| Pinch (thumb + index) | grab the nearest object | draw a stroke; pinch near a stroke tip to extend it, mid-stroke to pull and stretch it |
| Fist | grab the nearest object | hold one fist for 150 ms to rotate the sculpture |
| Two fists | - | move, scale and turn the sculpture |
The grab "magnet" radius is 10 cm by default (adjustable 2-20 cm in the calibration screen).
brew install xcodegen
cp Config.example.xcconfig Config.xcconfig # set HANDGRAB_TEAM_ID and a unique HANDGRAB_BUNDLE_ID
make open # generates HandGrab.xcodeproj and opens XcodePick your iPhone in Xcode and press Run.
| Requirement | Notes |
|---|---|
| Mac with Xcode | iOS 18+ SDK |
| XcodeGen | brew install xcodegen |
| iPhone with Face ID | needed for live tracking |
| Apple ID | a free account is enough for your own device |
No phone? Simulator and other commands
make replay # build for Simulator, replay a synthetic hand (scripts/sim-replay.sh)
scripts/sculpt-sim.sh demo all 160 # render every brush without tracking
make fetch-mediapipe # optional second detector, used for cross-checks and the benchmarkLaunch arguments: -replay <synth|s-DATE.jsonl>, -sculptdemo <all|art|pull|rotate|brush>, -trackeval, -bench.
flowchart LR
A["TrueDepth camera<br/>video 640x480 @60 + depth @30"] --> B["Vision hand pose<br/>21 joints, 2D"]
A --> C["Palm depth<br/>matched to frame time"]
B --> D["HandFusion<br/>bone lengths, ray-sphere solve,<br/>1-euro filter, prediction"]
C --> D
M["MediaPipe (optional)"] -.-> D
D --> E["SlotAssigner<br/>two hands"]
E --> F["Pinch / fist detection"]
F --> G["GrabController"]
F --> H["SculptController"]
G --> R["SceneKit 120 Hz"]
H --> R
Capture/- separate video and depth outputs (a synchroniser would wait for the 30 Hz depth and halve the frame rate), format selection, device probe.Tracking/- detectors, palm-depth sampling, per-joint camera rays with fixed bone lengths, filtering, hand slots, pinch/fist state, session recorder/replayer.Scene/,Sculpt/- gesture-driven interaction, stroke meshing, brushes and materials, studio lighting.
Measurements and methodology: docs/measurements/ (the depth stream is capped at 30 fps on the tested device).
- Experimental research project; the UI is in Russian.
- Live tracking is tested on one device (iPhone Air). Not yet tested on other Face ID iPhones or iPads.
Issues and PRs are welcome. Keep changes small and describe how you tested them (device or replay).
MIT, see LICENSE. Third-party components: THIRD_PARTY_NOTICES.md.
Made by @meshoklv



