Namaste Ji by Ayushman Dash

Why QA exists, and how the QA Agent is built.

The origin story is a number: 62% of the Director's shots dropped the overlay text — pretty pictures nobody would forward. 1,262 renders were sent back. That is when QA stopped being a nice-to-have and became the product. The QA Agent is the answer: sample honestly, measure explicitly, judge transparently, veto in code, keep a human able to overrule.

Why — the finding, as data.

Before: a render whose intended text is missing (flagged). After: one that carries it. Same pipeline, one judge between them.

Gold line-art of divine footsteps and an aarti plate on a twilight mountain temple path, with a Hindi Friday blessing.
flagged · शुभ शुक्रवार | माता रानी घर को सुरक्षित रखें
judge: missing forward text
A festive courtyard scene with Holi colors, petals, and a Radha-Krishna mural in the background.
shipped · नाचो, गाओ, रंग उड़ाओ
text legible · culturally correct

The numbers

62% of shots in the first big batch dropped the overlay text — the Director wrote it, the renderer lost it. 3,144 shot texts were rewritten; 1,262 textless renders rejected. The fix was partly prompting and partly measuring: a text-legibility metric that compares what's on the image with what the brief intended.

source: memory/shot-text-review-2026-07 · Studio counts (live on /studio)

How — the engineering mirror.

  1. 1
    Population → candidate set. Every staged/scored/flagged greeting is the population; a run freezes a candidate set (all, or filtered by briefs/languages/q) so extend + publish act on exactly that set.
  2. 2
    Stratified · risk-weighted · Wilson-sized. Strata = briefs; never-scored first; n = z²p(1−p)/e² with finite-population correction (95%, ±10%, p=0.2 → n=31 of 61 on staging — the exhibit reproduces it). Modes: representative · full · custom-n; sequential extend (+N).
  3. 3
    Metric evaluators = versioned prompts. heimdall/metric-* ×7 (cultural-correctness, safety, text-legibility, aesthetic-quality, forward-appeal, brand-fit, prompt-adherence) + heimdall/judge, served from Langfuse per-env label with inline fallbacks; weights + veto flags in KV. Each runs against the image + its intended context (brief, overlay text).
  4. 4
    The judge integrates; code enforces the veto. Composite = weighted average over non-veto metrics; a veto metric failing makes the asset unreleasable regardless of the judge; flag = anything a human should see. (The actual guard is on /qa, act 3.)
  5. 5
    The run-scoped publish gate. Dry-run → publish: scored-and-passed + un-sampled-in-set ship; flagged/vetoed hold; unpublish = rollback. Overrides are audited (eval:override, admin-only).
  6. 6
    Feed-forward. Findings become Creator guardrails and Director prompt fixes; the QA insights page shows metric trends, flag/veto rate, brief×metric heatmap, flag-reason digest.

A real run, by the numbers.

eval-1783875159665 · population 61 · sampled 31 · ⌀ 86/100 · flagged 29 · vetoed 1 · model @cf/mistralai/mistral-small-3.1-24b-instruct

  • cultural-correctness9929p · 2w · 0f
  • safety9430p · 0w · 1f
  • text-legibility9010p · 21w · 0f
  • aesthetic-quality9429p · 2w · 0f
  • forward-appeal8224p · 7w · 0f
  • brand-fit7015p · 11w · 5f
  • prompt-adherence8711p · 20w · 0f

Go deeper

In the repo

  • agents/heimdall/src/evaluate.ts (metrics → judge → veto in code)
  • agents/heimdall/src/prompts.ts (inline fallbacks; Langfuse heimdall/*)
  • packages/catalog/migrations/0004_heimdall_eval.sql · 0005 (candidate sets)
  • apps/console/src/pages/Heimdall*.tsx