Skip to content
meshoklvPublic

About

Grab, rotate and sculpt 3D objects in mid-air with bare hands, using the iPhone TrueDepth camera at 120 Hz.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

HandGrab

HandGrab

Your iPhone sees your hands in mid-air. Grab, rotate and sculpt 3D objects - no touch, no controllers, no headset.
TrueDepth camera, Apple Vision, 120 Hz SceneKit rendering.

License iOS 18+ Swift 120 Hz PRs welcome

Quick start · Features · Gestures · How it works · Русский


Object mode
Object mode
pinch or fist pulls the nearest object into your hand
Sculpt mode
Sculpt mode
six brushes, colours, undo, export
Brush gallery
Brushes
clay, neon, silk, stardust, chrome, plasma

Note

Requires a physical iPhone with Face ID (TrueDepth camera). The Simulator supports replay mode only (recorded or synthetic hands through the same pipeline). Developed and tested on an iPhone Air; other Face ID models are untested.

💡 Why

Hand tracking on a phone usually means a flat 2D skeleton from the back camera. HandGrab uses the front Face ID camera: Vision gives 21 hand joints, the TrueDepth map adds real depth, and the two are fused into a 3D hand with fixed bone lengths. The result is low latency (hand pose ready about 46 ms after exposure, measured on-device) and enough precision to actually draw in the air.

✨ Features

🖐 Grab and throw four objects on a table with a pinch or a fist
🎨 Draw and sculpt in space with six brushes: clay, neon, silk, stardust, chrome, plasma
🤲 Two hands, independent: draw with both, rotate with one fist, scale with two
⚡ 120 fps rendering, hand pose ~46 ms after exposure (camera floor 33-38 ms)
↩️ Undo / redo, smoothing, X-mirror, export to Photos, USDZ and .hgsculpt
🧪 Test without a phone: record a session, replay it in the Simulator, or run synthetic hands

Brush details: docs/sculpt.md (in Russian).

👋 Gestures

Gesture Object mode Sculpt mode
Pinch (thumb + index) grab the nearest object draw a stroke; pinch near a stroke tip to extend it, mid-stroke to pull and stretch it
Fist grab the nearest object hold one fist for 150 ms to rotate the sculpture
Two fists - move, scale and turn the sculpture

The grab "magnet" radius is 10 cm by default (adjustable 2-20 cm in the calibration screen).

🚀 Quick start

brew install xcodegen
cp Config.example.xcconfig Config.xcconfig   # set HANDGRAB_TEAM_ID and a unique HANDGRAB_BUNDLE_ID
make open                                    # generates HandGrab.xcodeproj and opens Xcode

Pick your iPhone in Xcode and press Run.

Requirement Notes
Mac with Xcode iOS 18+ SDK
XcodeGen brew install xcodegen
iPhone with Face ID needed for live tracking
Apple ID a free account is enough for your own device
No phone? Simulator and other commands
make replay                        # build for Simulator, replay a synthetic hand (scripts/sim-replay.sh)
scripts/sculpt-sim.sh demo all 160 # render every brush without tracking
make fetch-mediapipe               # optional second detector, used for cross-checks and the benchmark

Launch arguments: -replay <synth|s-DATE.jsonl>, -sculptdemo <all|art|pull|rotate|brush>, -trackeval, -bench.

🔧 How it works

flowchart LR
  A["TrueDepth camera<br/>video 640x480 @60 + depth @30"] --> B["Vision hand pose<br/>21 joints, 2D"]
  A --> C["Palm depth<br/>matched to frame time"]
  B --> D["HandFusion<br/>bone lengths, ray-sphere solve,<br/>1-euro filter, prediction"]
  C --> D
  M["MediaPipe (optional)"] -.-> D
  D --> E["SlotAssigner<br/>two hands"]
  E --> F["Pinch / fist detection"]
  F --> G["GrabController"]
  F --> H["SculptController"]
  G --> R["SceneKit 120 Hz"]
  H --> R
Loading
  1. Capture/ - separate video and depth outputs (a synchroniser would wait for the 30 Hz depth and halve the frame rate), format selection, device probe.
  2. Tracking/ - detectors, palm-depth sampling, per-joint camera rays with fixed bone lengths, filtering, hand slots, pinch/fist state, session recorder/replayer.
  3. Scene/, Sculpt/ - gesture-driven interaction, stroke meshing, brushes and materials, studio lighting.

Measurements and methodology: docs/measurements/ (the depth stream is capped at 30 fps on the tested device).

⚠️ Limitations

  • Experimental research project; the UI is in Russian.
  • Live tracking is tested on one device (iPhone Air). Not yet tested on other Face ID iPhones or iPads.

🤝 Contributing

Issues and PRs are welcome. Keep changes small and describe how you tested them (device or replay).

📄 License

MIT, see LICENSE. Third-party components: THIRD_PARTY_NOTICES.md.

Made by @meshoklv

About

Grab, rotate and sculpt 3D objects in mid-air with bare hands, using the iPhone TrueDepth camera at 120 Hz.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages