Docs / content-production.md · mirrored from the repo

Namaste Ji — Content Production & QA Plane

Status: design / brainstorm. Nothing here is built yet. This is the agreed direction, captured so decisions aren’t lost. Treat it as living.

This is the execution half of the creative plane — what happens after the Creative Head emits briefs. It takes briefs → produces assets → evaluates them (QA / “Heimdall”) → stages → publishes. It extends CREATIVE-PLANE.md (the org-chart, lifecycle, rubric, calendar) and DISTRIBUTION.md (the embedding index, share-loop). Read those first; vocabulary here (stage modes, escalate-on-low-confidence, catalog-as-spine, amplify-the-recipe-not-the-dish) is reused, not redefined. The agents here are Agent Kernel instances where they reason, and plain Workflows where they don’t.

0. TL;DR — what this phase adds

The Creative Head answers “what to make.” This plane answers “make it, prove it’s safe, get it ready to publish.” Three new roles, one shared spine:

1. Content Director (agent) — pulls open briefs, explodes each into concrete single-asset work units (“shots”), enforces diversity + dedup before spending generation budget, fans them out over a Queue, and manages sample sets and batch execution.
2. Content Creator (async, batch-aware) — generates images + embeddings via the OpenAI Batch API (~24h/phase, 50% cheaper), tracked by a D1 batch-table + Cron poller; a sync path serves samples/spikes. On each shot’s arrival: optimize for WhatsApp-forward form factor → derivatives (thumbnails, placeholders) → caption → embed (batch) → metadata/SEO → stages it (quarantined, not live).
3. Heimdall (QA plane: evaluator workers + an LLM Judge) — runs smart-sampled evals over staged content against prompt-configurable metrics, with a hard cultural-veto gate, persisted scorecards, flagged-asset review, freshness/staleness tracking, and a feed-forward loop that turns its findings into the Creator’s guardrails.

Then a manual publish gate (a scheduler agent automates it later) flips staged → live. Everything is observable as a live funnel, queryable error logs, and an eval timeline — with single + bulk actions everywhere.

1. The frame — bind the vision to existing roles

This plane is the execution org-chart already sketched in CREATIVE-PLANE.md §1. We keep that vocabulary so we don’t grow a parallel one:

Principle (CP §6/§15): reasoning lives only where it earns its keep (Director + Heimdall’s Judge). The per-asset volume path is a Workflow over a Queue, not an agentic loop — at 200 assets/batch, per-asset reasoning is cost and latency we can’t afford. This is the load-bearing scalability decision of the whole plane.

2. The unit of work — the “shot”

A brief says count: 200. The thing that flows through the queue is not the brief — it is one shot: a fully-resolved single-asset spec. The Director’s first job is shot-list expansion.

Brief (count:200, variety_axes, metadata_spec, text_overlay, style_tokens, quality_rubric)
        │  Content Director explodes (LLM prompt-expansion → dedup → diversity/MMR)
        ▼
  shot {
    shot_id,            // deterministic = hash(brief_id, variety_cell, seq) → IDEMPOTENCY
    brief_id, batch_id,
    content_type: "image",          // the brief's `modality`; future: video, audio, image+audio
    prompt, negative_prompt,        // negative_prompt seeded from Heimdall's failure library (§12)
    language, variety_cell: { deity, festival, art_style, emotion, cohort },
    text_overlay, typography,       // from the brief
    style_tokens,                   // brandbook@vN
    guardrails_ref,                 // cultural-correctness checklist (§12 feed-forward)
    quality_rubric_ref,
    model_route,                    // which renderer/model + params
    intended_embedding?,            // for pre-spend dedup/MMR (§2.1)
    cost_budget_cents
  }
        │  one Queue message per shot (namaste-ji-jobs)
        ▼  many Content-Creator Workflows in parallel

2.1 Diversity + dedup enforced before spend (add-on, not in the raw vision)

Diversity is the north star (amplify the recipe, not the dish — DS-4, CP-15), so enforce it at creation, not only at serve time:

• During expansion the Director computes an intended embedding per candidate shot (cheap — embed the prompt text) and runs MMR across the candidate set + kNN against the existing catalog (the recommender index, §9). Candidates that are near-duplicates of what we already have, or of each other, are dropped or nudged. We don’t pay image-gen dollars to re-make content we already own.
• This is the production-time complement to serve-time diversity ranking (DISTRIBUTION §7): generate broadly (many distinct takes), let serve-time handle repetition.

2.2 Idempotency + cost control (add-on)

• shot_id is deterministic. Queues redeliver; the Creator checks “already produced this shot_id?” before calling the (expensive) image API. Without this we double-bill on every retry.
• Per-brief and per-run budget caps. Each shot carries cost_budget_cents; the Creator records cost_cents actually spent; the Director stops enqueuing a brief once its budget is hit. The Control DO kill switch drains the queue mid-batch.

3. Content-type abstraction — RendererPort

Images now; design so video / music / image+audio drop in tomorrow with zero changes to the spine (queue, lifecycle, eval, staging, publish). A new port impl in @namaste-ji/agent-kernel, sibling to the existing ModelPort / MemoryPort:

interface RendererPort {
  contentType: 'image' | 'video' | 'audio' | 'image+audio';
  // SYNC — one asset now (samples, spikes, dev). Fast feedback, full price.
  render(shot: Shot): Promise<RawAsset>;
  // BATCH — submit many shots to the OpenAI Batch API (§5.2). ~24h, 50% cheaper.
  // Returns a provider batch handle; results are collected later by the poller.
  submitBatch(shots: Shot[]): Promise<BatchHandle>;
  derivatives(raw: RawAsset): Promise<Variant[]>;   // type-specific recipe (thumbnails, posters, …)
}

• Two execution modes (CD-12). Production volume goes through the OpenAI Batch API (async, ~24h, 50% cheaper); samples / spikes / dev use the sync path for fast feedback. Same renderer, same downstream — only when the pixels arrive differs. This maps cleanly onto the product: sample sets (§11) and the Creative Head’s content-sampler are sync; full batch execution is batch.
• The brief already carries modality (image | image+greeting); extend that enum as content types arrive. The Director routes each shot to the registered renderer for its content_type.
• Images now: the GPT image model (gpt-image-1) invoked via the Responses API (/v1/responses image-generation tool — already our gateway’s openai_api=responses shape), which is Batch-API-eligible. Routed through the AI Gateway BYOK path for sync calls. ⚠️ Batch + Files are management endpoints that likely bypass the AI Gateway — see §5.2 and the open questions.
• Only the renderer and the derivative recipe are type-specific. Everything downstream is content-type-agnostic.

4. Content Director (agent)

A thin Kernel instance. New workspace: agents/content-director/. Proxied through console-api for RBAC, exactly like creative-head is today.

Responsibilities:

• Brief intake — pull approved briefs that are not yet in production or not fully processed (status/coverage query against the brief store). Convert each into a batch + its shot list. Briefs can be staged (claimed but not yet expanded) so multiple Director instances can pick up work without collision — claim via an atomic status flip (approved → expanding → queued).
• Shot-list expansion (§2) — LLM prompt-expansion guided by variety_axes + metadata_spec, then dedup/MMR (§2.1).
• Sample-set management (§11) and batch execution (§11) — including the include-samples vs generate-fresh toggle.
• Enqueue onto namaste-ji-jobs; does not generate anything itself.

Reasoning belongs here because expansion, dedup, diversity balancing and sampling are genuine judgment calls. Dispatch (enqueue, status flips) is plumbing.

4.1 Brief intake is a D1 work-table, NOT a queue (CD-10)

Briefs do NOT go through a Cloudflare Queue. A queue can’t be listed, has no queryable state, can’t be shown in the BO, and can’t re-claim a specific abandoned item. Briefs are low-volume, long-lived, stateful, and must be observable + re-claimable — so the right primitive is a claim-based work table in D1: the brief row is the queue. Only shots go through the Queue (namaste-ji-jobs) — high-volume, opaque, fan-out, with retry/DLQ handled for us. Two levels, two primitives, each fit to its job.

The existing brief_execution table (0002_briefs.sql) — already designed as the cross-agent write-back surface — becomes this work-list. Briefs stay in the creative-head D1; the Director binds it directly for claims (no HTTP-coupling). Graduate briefs to the shared catalog D1 only if that coupling ever bites.

Brief processing lifecycle (distinct from brief.status = proposed|approved, which is the governance/approval gate):

approved ──(claim)──> expanding ──(shots enqueued)──> dispatched
   ▲                      │                               │
   │ (lease expired →     │ (fatal: can't expand)         │ (all shots terminal)
   │  sweeper reclaims)   ▼                               ▼
   └──────── pending ── failed                     done | partial

• pending — approved + waiting to be picked up (the “staged” state).
• expanding / dispatched — in progress (claimed).
• done — every shot reached a terminal state (STAGED or REJECTED).
• partial — finished but some shots failed (surfaced as “182/200 staged, 18 failed”).
• failed — couldn’t even expand; carries an error reason.

4.2 The sweeper — claim, reclaim, reconcile (CD-11)

The Director’s continuous loop is a cron sweep (briefs aren’t latency-critical; generation takes minutes regardless). One idempotent sweep does three jobs:

1. Claim — atomic conditional update UPDATE brief_execution SET status='expanding', claimed_by=?, lease_expires_at=now+N WHERE brief_id=? AND status='pending'. D1 serializes statements, so only one Director wins (the loser sees changes=0) — safe with multiple Directors.
2. Reclaim abandoned work — a lease with expiry. A Director that crashes mid-run leaves a brief stuck expanding with an expired lease_expires_at; the sweeper resets it to pending and increments an attempts counter. After K attempts → failed (no infinite thrash). This is the “pick it up again while it’s not being worked on” guarantee.
3. Reconcile completion — for each dispatched brief, count its shots by terminal state; all terminal → done | partial. Counting is idempotent and self-heals from any missed event — preferred over a distributed decrement-counter. Shot-level crash recovery is the Queue’s job (redelivery + DLQ).

Migration adds to brief_execution: claimed_by, lease_expires_at, attempts, total_shots, error, and widens the status enum (+expanding, +dispatched, +partial).

Deferred (simplicity): no event-driven trigger and no second Director instance until volume demands them — the cron sweep covers “continuous” at this scale.

5. Content Creator (async, batch-aware)

New workspace: agents/content-creator/. Purely productive — it makes no quality judgments (those all live in Heimdall, §12). Generation is fundamentally asynchronous: images and embeddings are produced by the OpenAI Batch API (~24h per phase, §5.2), so the Creator is not a single synchronous Workflow — it is a batch lifecycle plus a per-shot post-processing Workflow that runs when each shot’s pixels arrive.

Per-shot post-processing (shared by sync + batch; runs once the raw image exists):

1. claim shot          → idempotency check; skip if shot_id already produced
2. optimize master     → WhatsApp-forward form factor (§8)         → R2 content/master/
3. derivatives         → thumbnails, LQIP/BlurHash, AVIF/WebP via CF Images
4. caption (sync VLM)  → caption + alt-text + attribute tags (§10) — also the embedding text (§9)
5. metadata + SEO      → write D1 row (§10)
6. stage image         → status = RENDERED  (image done; embedding pending)
   … then the embedding BATCH phase (§5.2) writes the vector and flips → STAGED.

Prompt + guardrails are baked at submit time (when the batch JSONL is built, §5.2): inject the cultural-correctness checklist + negative-prompt library (§12 feed-forward) and the brandbook@vN tokens (CP-9). A bounded sync self-check & regenerate (≤1–2×) is only available on the sync path; the batch path relies on Heimdall post-stage (you can’t mid-flight inspect a 24h batch).

Decision (CD-1): NO inline pre-stage safety/format gates. All evaluation — including safety/NSFW — is a Heimdall metric, run post-stage (§12). One place for all quality logic, all prompt-configurable; the Creator stays a clean producer. Trade-off accepted: storage + embedding spent on content that may later be rejected. Because staged = quarantined and publish is gated, nothing unsafe ever reaches users. Re-introducable as a config-flag pre-filter later without re-architecting.

5.1 Two acquisition modes (CD-12)

Samples need fast feedback (a human is waiting to approve); volume is latency-tolerant and cost-sensitive. Same renderer + same post-processing; only image acquisition differs.

5.2 Batch generation lifecycle — D1 batch-table + Cron poller (CD-12)

Generation runs as two sequential async Batch phases per content batch: (A) image-gen → then per-shot post-processing produces a caption → (B) embedding. Each is an OpenAI batch (~24h window, often faster). Orchestration reuses the work-table + sweeper pattern (§4.2) — no new agent, just durable plumbing:

ASSEMBLE  build JSONL (1 line/shot, custom_id = shot_id; prompt+guardrails baked in)
          → upload via Files API → create batch (completion_window 24h)
          → INSERT generation_batch { openai_batch_id, batch_id, shot_ids[], phase:image,
                                      status:submitted, submitted_at }
POLL      Cron tick (~hourly) lists non-terminal batches → GET /v1/batches/{id}
  ├─ in_progress / validating / finalizing → update status, keep waiting
  ├─ completed → download output file; per line, map custom_id → shot:
  │     success → enqueue per-shot post-processing (Queue → §5 steps 1–6) → RENDERED
  │     error   → that shot → FAILED + reason (DLQ)
  ├─ expired (didn't finish in 24h) / failed / cancelled → mark undelivered shots FAILED
  └─ when all image-shots are RENDERED + captioned → ASSEMBLE the EMBEDDING batch
        (text-embedding-3 over caption+tags, custom_id = content_id) → phase:embed
EMBED     poller completes the embedding batch → write vectors to Vectorize → flip → STAGED

Confirmed from OpenAI docs (2026-06): Batch supports image-gen (/v1/responses and /v1/images/generations|edits, gpt-image-1 family) and /v1/embeddings. Window is 24h only. Limits: ≤50,000 requests and ≤200 MB input file per batch. Statuses: validating → in_progress → finalizing → completed (+ failed, expired, cancelling, cancelled). Per-request failures and expiries land in a separate error_file_id.

• State lives in D1, not in a sleeping Workflow. A 24–48h wait spread over hundreds of shots is tracked by generation_batch / embedding_batch rows + a Cron poller — the same crash-safe, queryable, observable model as the brief sweeper (don’t park a Workflow asleep for two days per batch).
• Idempotency / partial results. The batch output file is keyed by custom_id; re-processing is safe (shot dedup, §2.2). Successes flow on; the error_file_id lines (per-request errors + expiries) → DLQ; a re-batch of just the failures is a fresh batch.
• Bound image batches by OUTPUT size, not the 50k cap. gpt-image-1 returns the image as base64 inline in each output line (~1.5–2.5 MB per 1024² image), so a 50k-image batch would be a multi-GB output file. Coalesce image batches to a practical size (~hundreds–low thousands); embeddings are tiny so they can pack densely. (Resolves the coalescing open Q.)
• Batch + Files go DIRECT to OpenAI, not via the AI Gateway (confirmed: the gateway docs cover only chat/completions + responses; /v1/batches + /v1/files are async management endpoints the inference proxy doesn’t handle). Use the BYOK key from Secrets Store. Batch spend/observability comes from the batch object’s request_counts + our structured logs, not the gateway. Only the sync sample path uses the gateway.
• Latency folds into lead time. Up to ~24h (image) + ~24h (embed) ≈ 48h end-to-end worst case. The Creative Head’s prebuild_by (CP §10 campaign window) must budget this batch latency — festival content has to be submitted days ahead, not just rendered ahead.

5.3 Failure handling

A shot that errors (batch per-request error, expiry, post-processing failure, exhausted retries) lands in a dead-letter with a structured reason. Per-shot post-processing is a durable Workflow (its own retries); batch-level failures are handled by the poller (§5.2). The DLQ is what the BO surfaces for single + bulk retry/reject (§14); retry re-batches.

6. Lifecycle state machine (extends CP §6)

flowchart LR
  D[DRAFTED] --> Q[QUEUED] --> GB["GEN_BATCHED<br/>(image batch, ~24h)"]
  GB -->|batch error/expired| FA[FAILED] -.re-batch.-> Q
  FA -.reject.-> REJ[REJECTED]
  GB -->|image arrives| RND["RENDERED<br/>(optimized+caption, embed pending)"]
  RND --> EB["EMBED_BATCHED<br/>(embedding batch, ~24h)"]
  EB -->|vector written| S["STAGED<br/>(quarantined)"]
  S --> EP[EVAL_PENDING] --> EV[EVALUATING]
  EV -->|clean| SC[SCORED]
  EV -->|Judge flags / veto| FL[FLAGGED] --> HR[HUMAN_REVIEW]
  HR --> SC
  HR --> REJ
  SC -->|publish gate| L[PUBLISHED]
  L -.rollback.-> U[UNPUBLISHED]

The two *_BATCHED waits are the OpenAI Batch phases (§5.2); a shot can sit there ~24h each. The sync path (samples) skips them — render lands straight at RENDERED and sample assets are quarantine-discarded, not embedded/staged (CH-3).

Every transition is an event on the existing run-event stream (emitRunEvent from @namaste-ji/agent-kernel), correlation-keyed by brief_id / batch_id / shot_id / content_id — the CP §13 “logs as a queryable dataset” contract. This is what powers every observability surface in §14 for free.

7. Where eval sits — async, post-stage, sampled

Because staging is continuous and async, evaluation must not block the volume path. Content is produced and staged independently; Heimdall runs as its own sampled, snapshot process over the staged population (§12). This is the reordering vs CREATIVE-PLANE §6 (which placed AUTO_EVALUATED before ingest): here, all evaluation is post-stage, which is precisely what makes “an eval runs while new content is still being staged” a tractable, trackable situation rather than a race.

8. Asset optimization & derivatives — the WhatsApp form factor

Product-critical for the next-billion / 2G–3G audience. A forward must feel instant.

• Master: render high-res (≥1024²), keep a lossless master in R2 (content/master/).
• Delivery variants via Cloudflare Images (CP-7):
  • Square 1080×1080 (feed) + portrait 1080×1920 (WhatsApp Status / phone scroll).
  • AVIF / WebP with a JPEG fallback, perceptually quantized to a file-size budget (~80–150 KB).
  • ⚠️ WhatsApp re-compresses on send — optimize for perceptual quality at low byte count (what survives its recompression), not pixel-perfection.
• Placeholders: BlurHash / ThumbHash string in D1 (tiny, inlined in feed JSON later) → zero layout shift on load.
• Watermark / brand lockup baked here from brandbook@vN tokens (CP-9); doubles as attribution/discovery.

9. Embeddings & the recommender index

The recommender needs to find similar content given an image, text, or both in one latent space (DISTRIBUTION DS-13). This stays behind an EmbeddingPort; the dimension is configurable and is NOT locked yet (CD-2). Vectorize fixes a dimension at index-creation, so we defer creating the production index (or recreate the provisioned namaste-ji-catalog, currently dim 1024) until the model is chosen.

interface EmbeddingPort {
  dim: number;                                   // config, not a constant
  embed(input: { image?: Blob; text?: string }): Promise<Float32Array>;
}

9.1 The real constraint and the options

OpenAI’s embedding API (text-embedding-3-*) is TEXT-ONLY — it cannot ingest an image. So raw OpenAI embeddings alone cannot satisfy “query by image or text in one space.” Two ways forward, both keepable free; pick at build time:

Decision (CD-13): embeddings run on the OpenAI Batch API ⇒ Option A (caption-bridge) is the chosen path. Since OpenAI embeddings are text-embedding-3 (text-only), batching them means we embed text — so we generate a caption/description of each rendered image (the sync VLM step, §5 step 4) and embed that text + tags. This is why the embedding Batch phase (§5.2, phase B) is downstream of image generation: it needs the caption, which needs the image. dimensions is set via the API param; the Vectorize index is created at that dim when we build (still not locked — CD-2 narrowed, not closed). Switching to true-multimodal (B/C/D) later = swap the port impl + recreate the index; nothing else changes. Trade-off: the text-mediated space loses pure-visual nuance (palette/composition) — acceptable for v1, revisit if recommender quality needs it.

Decision update (CD-16): implemented as Cohere embed-v4.0 (true multimodal + multilingual, 1024-dim) via the AI Gateway BYOK, DIRECT (sync) — not the caption-bridge, and batch deferred. Direct is easy to test on a few samples; the OpenAI/Cohere Batch variant slots in behind the same EmbeddingPort later (CD-12) with no flow change. Verified live (gpt-image-1 → Cohere embed-v4, dim 1024). Cohere’s image_url must be an object { url } (OpenAI-style).

9.3 Switching embedding models later (CD-16)

Models change (dim, space, provider). The design accommodates a swap without a rewrite:

• The seam: EmbeddingPort (model-agnostic embed()); the provider impl is the only model-specific code. Cohere now; Jina/OpenAI/Workers-AI are drop-in impls.
• Provenance: every asset + vector records embedding_model (+ dim). So we always know which space a vector is in, and can find what’s stale after a swap.
• Re-embeddable by construction: assets retain the R2 master + caption/tags, so any asset can be re-embedded into a new model at any time — no data loss blocks a migration.
• Migration = new index, not in-place (Vectorize fixes dim at creation): create a fresh index at the new dim → backfill (claim assets where embedding_model != active, re-embed into the new index) → flip the active embedding profile (model + dim + index) that both the writer and the recommender read → retire the old index. kNN never mixes spaces because each index is one model. The active profile is config, not hardcoded.

9.2 What we embed & store

• Embed a fusion of image ⊕ caption ⊕ tags (Option A embeds the caption+tags text; B/C/D embed image+caption jointly) → one vector per asset.
• Store in Vectorize with structured metadata facets (deity / festival / language / region / temporal_class / cohort) so the recommender does kNN ∩ structured filter.
• One index, many consumers (DS-13): dedup (§2.1), serve-time diversity (MMR), the Creative Head’s catalog gap analysis (CP-16), and the future recommender.

10. Metadata & SEO

Written at stage time (§5 step 7), stored as a JSON payload on the asset row:

• Caption (per language), alt-text, tags / attribute tags (style/motif/palette/ deity/composition/theme/language — the brief’s metadata_spec is the write-contract), temporal_class, occasion link, SEO (title, description, structured-data hints).
• The caption doubles as embedding input in the caption-bridge path (§9) and as Judge context (§12) — generate it once, use it three ways.

11. Sample sets — versioned, viewable, batch-executable

Your “generate brief samples, version them, view them, then execute the whole batch” flow, made first-class. Owned by the Content Director.

• sample_set is versioned per brief: samples@v1, samples@v2, … Each is a small N of real assets generated through the normal Creator path but tagged is_sample = true.
• Viewable in a dedicated BO page (§14): side-by-side, per-version, with their scorecards (samples are eval-scored so the signal is evidence, not vibes — mirrors the Creative Head’s sampler, CH-3).
• Execute batch action with the toggle you asked for:
  • Include approved samples — promote the approved sample assets into the batch and count them toward brief.count (generate only the remainder).
  • Generate fresh — samples were a probe only; ignore them in the count and generate the full count anew.
• Samples that are not promoted are quarantine-and-discarded (never silently leak into the live catalog).

12. Heimdall — the QA / evaluation plane

The largest piece, and the one whose nuances drive most of the design. Heimdall is the Evaluators + Judge of CREATIVE-PLANE §8, built for async, sampled, repeatable evals.

It may graduate into its own doc (docs/QA-HEIMDALL.md) as it firms up. Captured here for now so the phase is in one place.

12.1 Topology

Heimdall is not one worker. It is:

• an eval-run orchestrator (a Kernel agent — it decides what and how much to sample),
• a pool of stateless metric-evaluator workers (one per metric, fan-out like the Creator),
• an LLM Judge that integrates per-metric scores + applies the veto.

New workspace: agents/heimdall/. Proxied through console-api.

12.2 Metrics are prompts, not code (CD-3)

Each metric is a prompt in Langfuse (heimdall/metric-<name>@label), versioned. Add / remove / update a metric by editing a prompt — never a deploy. (Respects the project rule: sync prompts via Langfuse MCP; never auto-promote to production — see memory feedback-langfuse-sync.) Example metric set (you own the list): cultural-correctness, brand-fit, aesthetic quality, text legibility, prompt-adherence, forward-appeal, safety.

12.3 Scoring model — veto is a gate, not an average (CP-4)

• Cultural / religious correctness = HARD VETO. Wrong vahana/consort/attributes, mixed iconography across faiths, misspelt vernacular blessing, wrong festival date → UNRELEASABLE. Not averaged into a composite. One wrong devotional image can end the brand.
• All other metrics → a weighted composite score.
• Safety/NSFW is just another metric here (since there’s no inline gate — CD-1).
• Output: an asset_scorecard (per-metric scores + composite + veto flags + Judge rationale) and an eval_run roll-up (distribution, veto count, flagged ids).

12.4 Smart sampling — the async / staleness problem solved (CD-4)

An eval-run is a snapshot over the staged population at time T. Staging is continuous, so we make sampling, freshness, and provenance first-class:

eval_run {
  eval_run_id, trigger: manual | auto, prompt_versions,
  population_snapshot: { staged_count_at_start },
  sampling: {
    method: "stratified",
    strata: [variety_cell, model_route, brief_id],   // NOT pure-random
    risk_weighting: true,                             // oversample cold-start / high-risk cells
    target_confidence: 0.95, target_margin: 0.05,
    n_planned, n_evaluated
  },
  results: { composite_dist, per_metric, veto_count, flagged_ids },
  freshness: { staged_at_eval, staged_now, new_since, staleness_pct }
}
eval_sample { eval_run_id, content_id, was_sampled, selection_reason }  // ← "which were evaluated"

Design rules:

• Stratified, not random. Stratify by variety_cell × model_route × brief. A pure random sample hides one bad theme or a regressed model behind a good average.
• Risk-weighted. Oversample cold-start cells (new deity/festival, first batch from a new model) — that’s where cultural-veto risk concentrates.
• Statistically grounded n. Size the sample by a Wilson interval for the target confidence/margin on the pass/veto proportion (“n=84 → ±5% at 95%”). The orchestrator LLM narrates and adjusts; the floor is a real CI, not a vibe.
• Sequential / incremental sampling. “Increase sample size” = draw more into the same eval-run and recompute the CI — cheaper and statistically cleaner than re-running. CTA: Extend eval (+N) vs Full eval.
• Freshness / staleness explicit. staleness_pct = new_staged_since_eval / population. An eval auto-flags STALE past a threshold (e.g. >20% new). The BO badge reads, e.g., “Eval fresh · 84/1200 sampled · 95% CI ±5% · 320 staged since (27% stale).” This answers all four of your nuance questions: how many sampled · the sampled situation · how to grow the sample · which examples were evaluated.
• Provenance. eval_sample records exactly which assets were in the sample and why → the BO can always show “these are the N that were evaluated.”

12.5 Trigger — manual now, configurable to auto (CD-5)

Make the trigger a Control DO stage flag (stageMode['eval'] = manual | auto) — the existing mode mechanism. Default manual (your “keep a manual step for now”). Flip to auto (“fire immediately after content is created”) later by flipping the flag — no re-architecture.

12.6 The feed-forward loop — QA learnings become Creator guardrails (CD-6)

This is what turns a QA gate into a QA system and closes the loop (mirrors the Creative Head’s learning loop). Heimdall maintains two standing, versioned artifacts that the Creator injects at prompt-build time (§5 step 2):

1. a per-deity cultural-correctness checklist, and
2. a growing negative-prompt / known-failure library (iconography mistakes Heimdall caught, with examples).

So a mistake caught once prevents the same mistake across all future batches.

12.7 Eval timeline + flagged inbox (your observability asks)

• Eval timeline chart — overall composite + per-metric scores across eval-runs over time, veto-count, outliers / red-flag content thumbnails, with next-step CTAs (retry / reject / extend-eval / publish-clean). Backed by the eval_run roll-ups.
• Flagged inbox — only flagged/vetoed/sampled-low pieces reach a human (validation: submit); dead-simple one-task-at-a-time UI (CP §8). Humans review exceptions + samples, never the volume path — that’s what scales to millions of assets.

13. Staging → publish gate

• Staged = quarantined (the system of record holds it; users never see it).
• Publish is the gated step. Manual now (a button — single + bulk), governed by the Control DO. Instant unpublish / rollback is mandatory (CP-6). A scheduler / Publisher agent automates promotion against the events calendar later (out of scope here; DISTRIBUTION.md owns it).
• Publish should refuse vetoed assets and warn on stale/un-evaluated ones.

14. Back-office surfaces

IA + wireframes + per-screen API contracts: docs/console/CONTENT-SURFACES.md.

Reuse the existing run-event stream + console patterns (apps/console). New surfaces, all with single + bulk actions as a first-class affordance:

URL-state rule applies: tabs/filters/search must persist in the URL and survive refresh (memory ui-url-state-preservation).

15. RBAC additions

Extend the enum in packages/rbac/src/permissions.ts (and mirror in console-api migration 0001_rbac.sql):

content:create        # Director may explode briefs + enqueue shots
content:sample        # generate / approve sample sets
eval:read             # view scorecards, timelines, provenance
eval:run              # trigger a Heimdall eval-run / extend sampling
eval:override         # override a veto / resolve a flag (high-trust)
content:retry         # retry / reject failed shots (DLQ)

content:publish / unpublish / delete already exist. Role mapping: admin = all; creative-lead = create/sample/publish/eval:run/retry; reviewer = eval:read + flag resolution via validation:submit; engineer / viewer = read-only. MVP stays admin-only (CP §12).

16. Data model (D1 · R2 · Vectorize)

Same evolvability discipline as the brief table (0002_briefs.sql): store the whole object as a JSON payload, promote only query-critical fields to indexed columns — changing a shape needs no migration.

• content_assets — content_id, brief_id, batch_id, shot_id, content_type, status, is_sample, r2_master_key, variants(JSON), placeholder, embedding_id, model_route, cost_cents, payload(JSON: metadata/SEO/tags), created_at, updated_at. Index (status, batch_id), (brief_id), (is_sample).
• shots (work units) — the exploded specs + idempotency key + attempt count + DLQ reason + status.
• generation_batch / embedding_batch (§5.2) — openai_batch_id, batch_id, phase(image|embed), status, input_file_id, output_file_id, shot_ids(JSON), submitted_at, last_polled_at, completion_window, error. The poller’s work-table. Index (status).
• sample_sets / sample_items — versioned per brief (§11).
• eval_runs / eval_metrics / eval_samples / asset_scorecards (§12).
• guardrails / negative_prompt_library — Heimdall’s feed-forward artifacts (§12.6).
• R2 layout: content/master/…, content/variants/….
• Vectorize: one index, dimension chosen at build time (§9), structured-metadata filtered.

17. Observability / events

Extend the run-event stream (emitRunEvent) with content-level events keyed by brief_id / batch_id / shot_id / content_id / eval_run_id. Structured JSON, stable schema, correlation IDs (CP §13, CP-10) — so the Production board, DLQ inbox, and eval timeline are all queries over one event dataset, and a future log-monitoring agent can triage without bespoke wiring.

18. Cloudflare mapping

19. Decisions log (extends CREATIVE-PLANE CP-1…CP-16)

20. Open questions

• Batch image-return field (last live unknown, §5.2): confirm the exact JSON path of the base64 image in a /v1/responses image-gen batch output line vs /v1/images/generations — pick whichever endpoint gives the cleaner per-line custom_id→image mapping. (One live 1-request batch settles it — build step 0.) Everything else is confirmed.
• Embedding model (§9, CD-13 narrowed): caption-bridge OpenAI now; revisit true multimodal (Cohere/Jina/Voyage) if recommender quality needs pure-visual nuance — lock the Vectorize dim when chosen.
• Captioning model for the caption-bridge path (a VLM via the gateway) — sync per-shot; quality vs cost.
• Sampling thresholds (§12.4): default confidence/margin, staleness % that marks STALE, risk-weighting function.
• Sample-set size N and how a promoted sample’s existing scorecard carries into the batch eval population.
• Director claim semantics at scale (multiple instances) — atomic status flip vs a lease.

21. Build order (each step shippable)

0. Spike (small): confirm the batch image-return shape — capabilities + limits + the gateway-direct decision are already confirmed from docs (§5.2); a throwaway 1-request image-gen batch just settles the exact b64 field path (§20). De-risks before we build. Needs the OpenAI BYOK key. (Hours; not a PR.)
1. Content catalog data model (D1 + R2 + Vectorize contracts; defer the index dim) — the shared spine. (The ⬜ next item already in AGENTS.md.)
2. Content Director: brief-pull (claim/lease/sweeper, §4.2) + shot-list expansion + dedup/MMR. Visualize the shot list; no generation yet.
3. Content Creator — SYNC path first: one shot → /v1/responses image → optimize / derivatives / caption / metadata → RENDERED. De-risks rendering + R2 + CF Images fast, and is the sample/sampler path. DLQ on failure.
4. Batch path: assemble JSONL → Files upload → create batch → generation_batch row; Cron poller state machine → on completion fan out to the step-3 post-processing.
5. Embedding batch phase: caption → text-embedding-3 batch → Vectorize write → STAGED.
6. Production board + staging gallery + manual publish gate (with rollback) — shows the batch waits + freshness so the async pipeline is legible.
7. Sample sets: versioned (sync) samples + review UI + Execute-batch (include/fresh) toggle.
8. Heimdall v1: metric prompts in Langfuse + Judge + scorecards + flagged inbox (manual trigger).
9. Smart sampling + freshness/staleness + eval timeline + provenance; then the feed-forward guardrails loop.
10. Later: Publisher/scheduler agent (calendar-driven promotion); recommender over the index.

22. MVP cut line

In: admin only · Content Director (manual run, brief→shots via claim/sweeper, dedup/diversity) · Content Creator (sync sample path + OpenAI Batch volume path: image batch → caption → embedding batch → WhatsApp-optimized + thumbnails + placeholders → STAGED; Cron poller) · sample sets + execute-batch toggle · Heimdall v1 (prompt-configurable metrics + hard cultural veto + scorecards + manual trigger + smart sampling with freshness/provenance) · manual publish gate + rollback · Production board + staging gallery

• Heimdall console + DLQ inbox · structured content events.

Out (later): auto-trigger eval · human-validator consensus beyond a single reviewer · Publisher/scheduler agent + calendar-driven promotion · the live recommender · video / audio content types · per-user personalization · log-monitoring / auto-PR agents.

23. Relationship to other docs