Spatial Hand Gesture AR Engine

Use a webcam and hand gestures to interact with a small Three.js scene in the browser.

What this tool does

This browser AR experiment combines MediaPipe Hands hand-landmark tracking with a Three.js scene. It recognizes pinch, open-palm, fist, and pointing gestures, then uses them to move or trigger effects on the interactive 3D objects.

Who this is for

• Developers prototyping camera-based spatial interfaces
• Students and makers learning about hand landmarks, gestures, and 3D interaction
• Teams evaluating a browser-based AR interaction concept

How to use this tool

1. Allow camera access when the browser asks, then place one hand in view.
2. Point to hover over a scene object and bring your thumb and index finger together to pinch, grab, and translate it.
3. Open your palm to release the object; make a fist to trigger the holographic pulse effect.
4. Watch the tracking panel for frame rate, inference latency, confidence, gesture, pinch distance, and active target.

Inputs

The input is a live webcam stream. MediaPipe analyzes the visible hand landmarks in the browser; the gesture classifier uses the relative finger positions and a normalized thumb-to-index distance to identify the supported gestures.

Outputs

The page renders the webcam view, hand-tracking overlay, diagnostic KPIs, and a Three.js scene containing interactive objects. Pinching moves and rotates a grabbed target, while a fist produces a temporary scale-pulse effect. There is no file export.

Example use cases

• Prototype touch-free controls for a kiosk or interactive display
• Demonstrate the relationship between hand landmarks and gesture events
• Explore a spatial UI before investing in a larger AR or computer-vision build

Frequently asked questions