27 Sept 🎟️ Fundamentals of AI · free, 50 seats Save my seat → For teams AI in Practice · dates being set now Join the waitlist →
Namaste Ji by Ayushman Dash

How quality is controlled — and the failure that made it necessary.

If agents make your product, agent mistakes ARE your product — unless something stands in the way. This system learned that with a number: in one early batch, 62% of the generated pictures were missing the greeting text they were supposed to carry. Pretty, useless, and almost shipped. The answer was a dedicated QA agent with a simple creed: check a fair sample of everything, score each piece on explicit criteria, let a judge integrate the scores — but enforce the cultural and safety line in plain code, and keep a human able to overrule everything.

The failure, shown honestly.

Left: a picture that lost the greeting text it was ordered to carry — caught and flagged. Right: one that kept it. Same pipeline, one judge between them.

Gold line-art of divine footsteps and an aarti plate on a twilight mountain temple path, with a Hindi Friday blessing.
flagged · शुभ शुक्रवार | माता रानी घर को सुरक्षित रखें
judge: missing forward text
A festive courtyard scene with Holi colors, petals, and a Radha-Krishna mural in the background.
shipped · नाचो, गाओ, रंग उड़ाओ
text legible · culturally correct

The numbers

62% of shots in the first big batch dropped the overlay text — the Director wrote it, the renderer lost it. 3,144 shot texts were rewritten; 1,262 textless renders rejected. The fix was partly prompting and partly measuring: a text-legibility metric that compares what's on the image with what the brief intended.

source: memory/shot-text-review-2026-07 · Studio counts (live on /studio)

How — the engineering mirror.

  1. 1
    Population → candidate set. Every staged/scored/flagged greeting is the population; a run freezes a candidate set (all, or filtered by briefs/languages/q) so extend + publish act on exactly that set.
  2. 2
    Stratified · risk-weighted · Wilson-sized. Strata = briefs; never-scored first; n = z²p(1−p)/e² with finite-population correction (95%, ±10%, p=0.2 → n=31 of 61 on staging — the exhibit reproduces it). Modes: representative · full · custom-n; sequential extend (+N).
  3. 3
    Metric evaluators = versioned prompts. heimdall/metric-* ×7 (cultural-correctness, safety, text-legibility, aesthetic-quality, forward-appeal, brand-fit, prompt-adherence) + heimdall/judge, served from Langfuse per-env label with inline fallbacks; weights + veto flags in KV. Each runs against the image + its intended context (brief, overlay text).
  4. 4
    The judge integrates; code enforces the veto. Composite = weighted average over non-veto metrics; a veto metric failing makes the asset unreleasable regardless of the judge; flag = anything a human should see. (The actual guard is on /qa, act 3.)
  5. 5
    The run-scoped publish gate. Dry-run → publish: scored-and-passed + un-sampled-in-set ship; flagged/vetoed hold; unpublish = rollback. Overrides are audited (eval:override, admin-only).
  6. 6
    Feed-forward. Findings become Creator guardrails and Director prompt fixes; the QA insights page shows metric trends, flag/veto rate, brief×metric heatmap, flag-reason digest.

A real run, by the numbers.

eval-1783875159665 · population 61 · sampled 31 · ⌀ 86/100 · flagged 29 · vetoed 1 · model @cf/mistralai/mistral-small-3.1-24b-instruct

  • cultural-correctness9929p · 2w · 0f
  • safety9430p · 0w · 1f
  • text-legibility9010p · 21w · 0f
  • aesthetic-quality9429p · 2w · 0f
  • forward-appeal8224p · 7w · 0f
  • brand-fit7015p · 11w · 5f
  • prompt-adherence8711p · 20w · 0f

Go deeper

Public places to see this part working.