Vision-based interfaces

The question

Can the body be the input? And if it can, how much of it should be read – and what should not be read at all?

The question came back on three projects: a moving car, a patrol control room, an exhibition floor. In all three the hands are busy or the screen is far away. So a head tilt, a gaze and a hand each took the place of a pointer, and each was pushed until it stopped working.

Evidence

• A head tilt – Glide – AirPods head-motion sensors and CoreMotion read the passenger's head posture, and a haptic arrives 2–7 seconds before the car sways (design value). On a curve the vibration flows top to bottom, so the rider tilts their head into the turn. The body is read, and the body is what gets spoken to.
• A gaze – Feasix AI – on the patrol-robot console, gaze is not an input device. It is evidence about the design. The eye-tracking round has not been run yet, and cognitive load and error rates are not measured. So this page does not claim reduced cognitive load.
• A hand – Distance of Thinking – the handle a visitor holds is the control: a tilt sensor (MPU6050) picks a branch of the probability field and a rotary dial sets temperature. No camera, which is why a dark gallery could be the design assumption. The Arduino controller was never tested on real hardware, though: the published recording and screens were steered with a mouse and a scroll wheel.
• The input I dropped – Distance of Thinking – shaking the sensor to trigger hallucination was removed. It would leave the cause of the hallucination in the visitor's hand, and hallucination had to be the result of an unlikely choice. Taking the body as input did not mean turning every movement into input.

What holds

Body data has a price. Glide dropped health-data access and stores no face video. The sickness metric is integrated on the phone with no cloud. Accuracy was the cost. Data never sent is also data never regulated. The lost accuracy is better recovered from vehicle-built-in sensors at production stage.

Gaze works as evidence, not only as input. Make people press with their eyes and a misread becomes a misclick. On a control-room screen, gaze is safer as the thing that judges the design.

A sensor is sometimes better than a camera. Use a camera where one tilt reading would do and lighting, privacy and computation come with it. Of Glide's four validation axes, the behavioural one detects head-down and hand-to-mouth gestures with CV – but that axis is for validation, not product input. All four are built; the sample is too small for statistical significance.

The more of the body you read, the more you must explain. When the system moves first, people ask why. That is why Glide always says what it noticed, and says nothing below the threshold.

Not verified yet

• Eye tracking – the Feasix AI round is planned, with no results.
• Real-vehicle test – Glide's effect figures are all simulation. Whether reading head posture and warning early actually reduces sickness needs a controlled run on the same stretch.
• The controller – Distance of Thinking's hardware steering still needs a real hardware test.