Deterministic vision models score operator actions frame by frame, and the large model re-checks only when needed: wrong, missed, and anomalous moves are stopped at the station, every verdict carries traceable evidence, and images never leave the terminal.
Deterministic vision first · LLM as safety net · three-state, traceable verdicts
The target is occluded by the forearm for 6 consecutive frames, so the vision layer cannot confirm the angle band; semantic re-check judges whether the gesture stays in range.
Re-check window: 4.2s left · timeout counts as uncertainSpeed and explanation are split across two engines: the deterministic vision layer owns recomputable verdicts inside the line takt, while the semantic layer handles occlusions and long-tail cases — both converge into one three-state result.
Owns accuracy and speed
Detection, keypoint pose, ROI and rule thresholds all close within a single frame; identical input always yields an identical result, fitting the line takt and enabling offline regression testing.
Owns explanation and long-tail coverage
It steps in only when a step demands re-checking or the verdict is uncertain, confirms the frame semantically and returns a readable rationale; it stays off the high-frequency path and can be switched off entirely.
| Scenario | Judge | Why |
|---|---|---|
| Part presence, order, arrival posture | Deterministic vision layer | High-frequency and quantifiable — threshold rules judge it stably, frame by frame. |
| Occlusion, glare, rare postures | Semantic model layer | Long-tail cases are hard to enumerate; semantic reading returns a rationale humans can verify. |
| Neither layer can confirm | Human confirmation | Stop rather than slip through: uncertain never counts as a pass and escalates to the floor. |
| Recording and traceability | Structured record | Model version, mode, confidence and evidence time are logged per step and replayable per round. |
Station cameras stream into an in-browser Dedicated Worker for local inference — frames never leave the terminal.
Local · Per frameWorkpieces, tools and hands are detected; ROI and IoU tracking keep targets bound across frames.
Detect + trackKeypoint skeletons yield angles, movement direction and dwell time, checked against each SOP step.
Pose + timingWhen a step demands re-checking or the verdict is uncertain, the model confirms and explains.
On demandModel version, mode, confidence and evidence time are written per step into the record.
Full tracematched Pass Judgement and time-window checks both hold, so the flow may advance.
Move to the next station step→not_matched Fail Skipped steps, wrong order, timeout or off-spec posture are caught on the spot.
Alert on site and hold the flow→uncertain Uncertain Insufficient evidence or timed-out checking is never counted as a pass — it stops rather than slips through.
Escalate to human confirmation→The work instruction is broken into verifiable steps: who does it, how long it takes and what "done" looks like are checked against vision evidence. Wrong moves, skipped moves and anomalies are closed on the spot instead of surfacing at final inspection.
| Step | Check | Time limit | Status | System action |
|---|---|---|---|---|
| 01 Part verification | Harness colour code and bin label match the current work order | ≤ 3s | Pass | Advance to the next station step |
| 02 Housing insertion | Hand enters the zone, movement aligns with the port axis and dwells for the required frames | ≤ 6s | Pass | Record execution time and confidence |
| 03 Torque fastening | Wrist angle stays inside the standard band until the required frame count | ≤ 8s | Uncertain · occluded | Confirmed by model re-check |
| 04 Visual self-check | Head turns to the observation point and gaze covers the port for the required frames | ≤ 4s | In progress | Release once evidence qualifies |
| 05 Scan and release | Scan action completed and operator left the equipment safety zone | ≤ 3s | Not started | Waiting for all prior steps |
Deviations hold the flow immediately; defects never reach the next station.
Sound, light and on-screen cues point to the exact step to redo.
Semantic re-check or human confirmation completes the evidence; uncertain never passes.
Round, step, mode and evidence time are logged in structured form.
False rejects and misses are aggregated per step to drive SOP and threshold tuning.
Stations are authorized by vision account and SOP release version. Changes take effect immediately, while historical rounds stay traceable under the release version of their time.
Any job that breaks into verifiable steps can plug into the same decision engine — scenarios differ only in SOP configuration and thresholds, not in development effort.
Action error-proofing for harness insertion, fastening and clip pressing.
Pre-entry protective equipment checks — no compliance, no entry.
Checklists become actions: what to look at, what to press, which gauge to read.
Completeness re-check for packing, sealing, labelling and scanning.
Interception on the floor, confirmation by level, records built for replay: detection is not the end — turning deviations into a traceable, improvable loop is.
Sound, light and screen cues guide the operator to redo or complete the step; the round continues.
Anomalous rounds or repeated uncertain results are pushed to the line leader for an on-site call.
Deviation trends are aggregated per step and fed back into SOP revisions and threshold tuning.
Raw images stay on the terminal for inference only — never stored or uploaded. What leaves is structured verdicts and audit records; every verdict maps to model version, mode, confidence and evidence time, meeting compliance needs for sensitive scenes.
Book a demo with a dedicated consultant and get a One-MOM digital platform solution tailored to your industry.