Case Study · 2026 · Internal R&D · 8 engineering days

Reviewable AI Triage Router

Confidence-gated LLM classification for high-volume operational queues with human review by design.

Reference implementation for our reviewable-AI standard.

The problem

Most AI automation fails in operations for the same reason: it acts on every item with equal confidence, and nobody can audit why. The result is a system that is impressive in a demo and untrusted within a fortnight.

What we built

A classification service returning intent, urgency, and suggested owner with calibrated confidence, prompt version, and model version recorded per decision.
A gating layer that automates only above a measured threshold and routes everything else to a human queue, with the threshold derived from a labelled evaluation set rather than chosen by feel.
A regression harness that replays the labelled set on every prompt or model change and blocks deploys that degrade agreement or increase confident errors.
A full decision log making every automated action reversible and explainable months later.

The result

On a 2,000-item hand-labelled corpus, gated routing reached 94% agreement on automated items while holding 12% of volume for review, and the regression harness caught two prompt changes that would have raised confident errors.

Want this level of documentation on your project?

Every engagement ships with the same written record: what changed, why, and what it measured.

Start a conversation