AR gestures

Overview

A browser-based augmented reality demo driven by the webcam. MediaPipe Hands tracks 21 landmarks on one hand in real time, a rule-based recogniser classifies the gesture from their geometry, and a particle effect is rendered on a canvas overlay that follows the hand. It is written in plain JavaScript with Canvas 2D rendering.

Pipeline

camera -> MediaPipe landmarks -> gesture rules -> smoothing
       -> active effect -> palm anchor and scale -> particles

Tracking and rendering are decoupled. MediaPipe processes 640 by 480 camera frames and delivers landmarks asynchronously whenever a frame is done, at which point the gesture is classified and the palm anchor updated. A separate animation loop redraws the canvas on every display frame from the most recent landmarks, so the effects animate at the display rate even when tracking runs slower.

Hand tracking

MediaPipe Hands is configured for a single hand, with a detection confidence threshold of 0.72 for finding a hand and a lower tracking threshold of 0.55 for following it once found. Each landmark comes in normalised image coordinates, with x and y in [0, 1] and y increasing downwards.

Users expect a webcam view to behave like a mirror, so the camera image is drawn flipped horizontally and every landmark is mapped to canvas coordinates as ((1 − x)·W, y·H), which keeps the effects aligned with the mirrored hand. When no hand is detected, the anchor is cleared and nothing is drawn.

Gesture recognition

Geometric rules

A finger counts as raised when its tip lies above its middle (PIP) joint in the image, which is a direct comparison of two y coordinates. Index, middle, ring and little finger are tested this way. The gestures then follow from which fingers are raised: all four for an open palm, index and middle for a peace sign, only the index for pointing and none for a fist. Any other combination counts as no gesture.

A pinch is detected from the Euclidean distance between the thumb tip and the index tip, with a threshold of 0.06 in normalised units. It is checked first because it is defined by the thumb and index alone, independent of the other fingers: during a pinch the remaining fingers may be raised or folded, and the finger-count rules, which ignore the thumb, would otherwise read the same hand as an open palm, a fist or no gesture.

Temporal smoothing

Per-frame labels flicker when a finger sits close to its threshold or a landmark jitters. Two filters stabilise them. First, the label is a majority vote over the last five frames, which removes isolated misclassifications. Second, a smoothed label must hold for 300 ms before its effect starts, and the hold timer restarts whenever the smoothed label changes. Once started, an effect persists until a different gesture is held, so a brief loss of the gesture does not switch it off.

The trade-off is latency against stability. The vote lags a real change by a few frames and the hold adds 300 ms, so a new effect starts a moment after the hand changes shape. In return, transitional hand shapes between two gestures do not trigger effects in passing.

Rendering

Palm anchoring and scale invariance

Effects are anchored to the palm centre, computed as the mean of the wrist and the four finger bases. Lightning alone is anchored to the index fingertip. The scale is the distance from the wrist to the base of the middle finger in canvas pixels, with a lower bound. Sizes, speeds, emission spread and orbit radii in every effect are expressed as multiples of this scale, so an effect looks the same whether the hand is close to the camera or far away.

Particle systems

Each element is its own particle system with an update step that advances the state by the elapsed time and a draw step that renders it around the current anchor.

• Fire. Particles are emitted at a rate proportional to the frame time, so emission is independent of the frame rate, with upward velocities and a small horizontal spread. Vertical velocity is damped every frame, each particle lives for about half a second to a second, with a hue between red and orange and lightness, size and opacity that fall as it ages.
• Wind. Eighteen particles orbit the palm on three elliptical rings, flattened vertically to suggest a ring seen at an angle. Each has its own angular speed and a sinusoidal modulation of its radius, and its opacity varies with its phase.
• Lightning. A new bolt is generated every 70 to 180 ms in a random direction around the vertical, by recursive midpoint displacement: the midpoint of a segment is offset randomly in proportion to the segment's length, and both halves are subdivided again, to a depth of four. Each bolt is drawn twice, a wide translucent stroke for the glow and a thin white core, and releases a burst of sparks whose velocity decays quickly.
• Earth. Six irregular rock fragments in earth tones orbit the palm on an ellipse while rotating about their own centres, and dust particles are spawned at random, drift upwards and fall back under a constant gravitational acceleration.
• Ice. Ten snowflakes orbit at three radii, each drawn procedurally as six arms with two side branches at 60 degrees, and rotate slowly. Two ripples expand from the palm half a cycle apart, fading as they grow.

Frame-time handling

All motion is integrated with the measured time between display frames, so speeds are the same on displays with different refresh rates. The time step is capped at 50 ms, so that a long pause, such as switching browser tabs, produces one bounded step instead of a large jump. Dead particles are culled every frame, and the five effect objects are created once and reused when a gesture returns, which keeps allocation low and the animation smooth.

Scope

The recogniser is rule-based and tracks one hand. The finger rules compare vertical positions, so they assume a roughly upright hand facing the camera.