How quality is controlled — and the failure that made it necessary.
If agents make your product, agent mistakes ARE your product — unless something stands in the way. This system learned that with a number: in one early batch, 62% of the generated pictures were missing the greeting text they were supposed to carry. Pretty, useless, and almost shipped. The answer was a dedicated QA agent with a simple creed: check a fair sample of everything, score each piece on explicit criteria, let a judge integrate the scores — but enforce the cultural and safety line in plain code, and keep a human able to overrule everything.
The failure, shown honestly.
Left: a picture that lost the greeting text it was ordered to carry — caught and flagged. Right: one that kept it. Same pipeline, one judge between them.
judge: missing forward text
text legible · culturally correct
The numbers
62% of shots in the first big batch dropped the overlay text — the Director wrote it, the renderer lost it. 3,144 shot texts were rewritten; 1,262 textless renders rejected. The fix was partly prompting and partly measuring: a text-legibility metric that compares what's on the image with what the brief intended.
source: memory/shot-text-review-2026-07 · Studio counts (live on /studio)
How — the engineering mirror.
- 1Population → candidate set. Every staged/scored/flagged greeting is the population; a run freezes a candidate set (all, or filtered by briefs/languages/q) so extend + publish act on exactly that set.
- 2Stratified · risk-weighted · Wilson-sized. Strata = briefs; never-scored first; n = z²p(1−p)/e² with finite-population correction (95%, ±10%, p=0.2 → n=31 of 61 on staging — the exhibit reproduces it). Modes: representative · full · custom-n; sequential extend (+N).
- 3Metric evaluators = versioned prompts. heimdall/metric-* ×7 (cultural-correctness, safety, text-legibility, aesthetic-quality, forward-appeal, brand-fit, prompt-adherence) + heimdall/judge, served from Langfuse per-env label with inline fallbacks; weights + veto flags in KV. Each runs against the image + its intended context (brief, overlay text).
- 4The judge integrates; code enforces the veto. Composite = weighted average over non-veto metrics; a veto metric failing makes the asset unreleasable regardless of the judge; flag = anything a human should see. (The actual guard is on /qa, act 3.)
- 5The run-scoped publish gate. Dry-run → publish: scored-and-passed + un-sampled-in-set ship; flagged/vetoed hold; unpublish = rollback. Overrides are audited (eval:override, admin-only).
- 6Feed-forward. Findings become Creator guardrails and Director prompt fixes; the QA insights page shows metric trends, flag/veto rate, brief×metric heatmap, flag-reason digest.
A real run, by the numbers.
eval-1783875159665 · population 61 · sampled 31 · ⌀ 86/100 · flagged 29 · vetoed 1 · model @cf/mistralai/mistral-small-3.1-24b-instruct
- cultural-correctness9929p · 2w · 0f
- safety9430p · 0w · 1f
- text-legibility9010p · 21w · 0f
- aesthetic-quality9429p · 2w · 0f
- forward-appeal8224p · 7w · 0f
- brand-fit7015p · 11w · 5f
- prompt-adherence8711p · 20w · 0f
Go deeper
Public places to see this part working.