Teleoperation HMI for a patrol robot
Physix AI
Summary
I designed the screen an operator uses to drive a patrol robot through an underground car park – and merged two design teams' work into one token system: 20 screens, 30 components, 220 tokens.
- 20
- 30
- 220
Overview
The operator sits in a control room and drives the robot with a gamepad and a PC monitor, watching several camera feeds, driving state, warnings and licence-plate recognition for hours at a time. The screen is a fixed 1920×1080 with a single dark theme for a dark room.
The research runs in two phases: phase one (Jun–Jul 2026) set out the problems and design criteria; in phase two (Aug–Oct 2026) a base-layout team and a wide-layout team designed in parallel and then merged. I lead screen design for the base-layout track in phase two.
Problem
Phase one surfaced three core issues:
- The camera views had no clear roles or priority.
- Driving information and warnings had weak visual hierarchy.
- Plate-recognition results didn't lead to the next action.
All three add to the operator's cognitive load and decision burden.
Research
- Journey analysis – across onboarding → preparing → driving remotely → ending, remote driving was set as the critical journey and split into driving and situational awareness / plate checks / connectivity and exceptions, each broken into five steps. The key insight for exceptions: "Before the cause of a fault, operators first check whether the vehicle is safe and still controllable."
- Literature and cases – eight prior studies and four industry cases (Vay, Hyundai–Kia SVM, Ottopia, Fernride).
- A quick phase-one usability test – about ten people, with an RC car, several cameras, OBS and the Figma UI combined, checking field of view, sense of control, task feedback and emergency handling.
- Technical meetings and stakeholder feedback – development constraints taken as inputs before design, and six items of August feedback tracked to completion.
Key decisions
Fix the width of the top buttons at their ‘on’ state
While driving, operators press buttons by remembered position, not by looking. If a button's width changes with its state, its neighbours move too.
Across all eight state combinations, no button's x position moves by even 1px.
A red background means only ‘unregistered vehicle’
One colour should mean one thing. Based on the recognition performance confirmed with the developer, telling registered from unregistered mattered more to the operator than the rare recognition error.
Unrecognised or misread cards sit on a neutral surface with a coloured dot; a red card means one thing – a vehicle that isn't registered.
The front camera biggest, the minimap only when asked for
The first piece of feedback was “make the front camera the largest”. An always-on minimap was covering the view needed to drive.
The front feed grew to 780×616 – 2.4× the area – and the minimap moved into a popup behind a MAP button.
The alert bar always holds its 56px
Layout moving during control is the worst case; video mustn't jump every time an alert appears.
With no alerts it shows “Normal · patrolling” and keeps its place. The cost – video height down from 888px to 852px – was written down as a deliberate trade.
Design
- Six areas – top navigation, status, camera view, driving and control bar, map, plate-recognition panel.
- State by colour only – on and off differ only in colour; a button's position never changes.
- Three alert levels – general (toast), urgent (red gradients at the top and bottom edges), and log.
- One way in – the right panel went from eight states to four. Popovers grow out of their button, the camera layout rearranges itself when reversing, and ultrasonic distance shows in millimetres, turning yellow at 800mm and red at 180mm.
Design system
220 tokens, and how two teams' work was merged
Tokens are split into primitive 139 · semantic 44 · spacing 15 · radius 6 · text 16; adding amber and blue ramps took primitives from 128 to 139. The layers exist for the cost of change – change a value in one place and every screen follows.
When merging, the other team's design language was kept while its structure was rebuilt into our system: 431 hard-coded colours were bound to tokens, and an audit script confirmed zero direct primitive references, zero unstyled text and zero clipped text, with the remaining 274 unbound items listed as deliberate exceptions. Along the way, 11 lost screens and 20 labels were restored and 50 clipped text layers re-measured.
Results
- 0px
- 2.4×
- 6 / 6
Six phase-one concepts became 20 final screens, and both layouts – base and wide, each in light and dark – were handed to the developer on September 2, 2026.
Reflection
- Colour is allocated by frequency × need to act. Strong colour on common events buries the ones that matter.
- Development constraints are inputs before design, not obstacles after it.
- Comparing means fixing everything but one variable. To compare the two layouts fairly, only one thing changed at a time.
- Merging is translation, not copying.