Docs / console/feed-lab.md · mirrored from the repo

Feed Lab — the recommendation & feed-algorithm playground

Status: design v2 — direction approved, build in progress (PR ladder in §15). v1 captured the brainstorm; v2 folds in simulation research (RecSim / Agent4Rec / RecoWorld, click-model literature — §12.1) and codebase reality (Cohere multimodal embeddings shipped; FL-4 resolved — §4). Living doc.

Feed Lab is the interactive front-end for the design already committed in docs/DISTRIBUTION.md. It does not invent a new feed strategy — it makes the existing one (DS-1…DS-15) tunable, observable, and verifiable before a policy touches a real user. A BO surface, so it also follows docs/console/DESIGN-PATTERNS.md.

1. What it is / why now

A back-office playground to: visualize the catalog’s embeddings, see clusters, build and tweak recommender / feed algorithms, simulate a live scroll for a persona, test vector search, read a diversity score, and — once satisfied — save a versioned feed algorithm that the serving path picks up. It doubles as a catalog-gap tool: white space on the embedding map → a Creative Head brief (Loop B, DS-4).

Why build it early: the distribution objective is non-standard (amplify the recipe not the dish, novelty + rarity as first-class terms — DS-4/DS-5). You cannot eyeball whether a weighting honours that. The Lab is the forcing function that turns the strategy into knobs you can feel.

And one more first principle (v2): the feed is not just policy. User interaction with content must feed back into the feed. The Lab is therefore a closed loop — every scroll, dwell, skip and share (real or simulated) updates the user state that the very next batch is ranked against (§6).

2. Core architecture — one isomorphic ranker, policy-as-artifact

The single decision that makes the Lab trustworthy. Two rules:

1. The ranker is one pure function, shared by the Lab and the serving Worker: rank(candidates, userState, context, policy) → orderedFeed. Lives in a shared package (@namaste-ji/feed — packages/feed/, pure TS, no Cloudflare types; that absence is the isomorphism guarantee). The Lab is just different inputs (a persona, not a live user) and instrumented outputs (score breakdowns, metrics). No second implementation, ever — that’s what keeps sim ≡ serve.
2. A “feed algo” is a versioned FeedPolicy artifact, not saved code. A Zod contract in @namaste-ji/feed, stored in D1 (history) + KV (current per label), labeled like Langfuse prompts (draft → staging → production) — a pattern the repo already uses. The Lab edits a draft; the serving Worker reads whatever is production. “Save a version” = new policy version; “ship it” = move the production label. Free rollback + A/B (serve two labels to two cohorts) + an audit trail of why the feed changed.

flowchart LR
  L[Feed Lab<br/>edit draft] --> Pol[(FeedPolicy<br/>draft→staging→prod)]
  Pol --> W[Serving Worker<br/>rank of same code]
  L -. imports .-> R[[rank fn<br/>@namaste-ji/feed]]
  W -. imports .-> R

FeedPolicy shape (sketch):

{
  "version": 4, "label": "draft",
  "candidate_mix": { "context": 0.5, "behavioral_knn": 0.3, "fresh_unproven": 0.1, "rare": 0.1 },
  "weights": { "quality": 0.6, "share_propensity": 1.0, "novelty_to_user": 0.8, "event_priority": 0.6, "pref_affinity": 0.4, "taste_affinity": 0.5 },
  "circulation_penalty": 0.7,
  "diversity": { "mmr_lambda": 0.5, "attributes": ["deity","palette","motif","composition"] },
  "exploration_budget": 0.1,
  "seen_cooldown_hours": 72,
  "event_ramp": { "curve": "linear", "max_lead_days": 14 }
}

3. Page IA — a Lab section, six surfaces (IA v3)

v1 sketched one six-panel cockpit; v2 split it into a Lab section; v3 (shipped) dissolved the simulator cockpit too — it was doing four jobs at once (persona building + policy editing + population runs + the feed), and the founder wanted the underlying objects as separate places: who (personas), how it ranks (policies), what it feels like (feed). Routes under /lab/…, all state in the URL per DESIGN-PATTERNS §5:

The simulated feed is a mock consumer app (Reels-style): a phone frame running the real ranker as a full-screen vertical swipe feed with snap scrolling — one greeting per screen, endless. Category tabs (For you · Good morning · · Festival · Fresh) pick the candidate pool; swiping generates fresh batches on the fly (rank → vary → seen-cooldown). Each screen shows the real asset (via the console image route) + vernacular caption + action rail (share/WhatsApp · save · get) and its “why” + per-term score breakdown. A day-scrubber replays pacing + event ramps. The point is to feel the scroll, not read a score table.

Interactive mockup: mockups/feed-lab.html — a standalone, click-to-open prototype (onboarding cold-start sim, live re-rank, organic labelled embedding clusters with the prior anchor, diff metrics, and a mock app with category tabs + infinite-scroll batch generation). Design fidelity, not production code; the ranker is a toy stand-in for the real rank().

4. Embeddings & clusters — the lightweight way

Compute nothing heavy in the browser; precompute offline, ship coordinates.

• What exists (shipped): the content-creator embeds every asset with Cohere embed-v4.0 (caption + attribute/tag text ⊕ image → one 1024-dim multimodal vector) and upserts to Vectorize with facet metadata (deity, theme_tag, language, region, temporal_class, content_id).

• FL-4 resolved — raw-embedding copy in catalog D1:

  content_embeddings(content_id TEXT PK, model TEXT, dim INTEGER,
                     vector BLOB /* f32 LE */, created_at TEXT)

  written by the creator at embed time; existing vectors backfilled via Vectorize getByIds (admin endpoint on the creator, chunked). R2 parquet deferred until scale demands.

• Consequence (new, FL-15): the Lab needs no Vectorize at all. At current scale (low thousands of items), brute-force cosine over an in-memory f32 matrix (~20 MB @ 5k items, loaded lazily from content_embeddings) answers kNN in milliseconds inside the Worker. Retrieval sits behind a CandidateRetriever port; Vectorize becomes the optimization once the catalog outgrows the cache (~50k hard cap). Local dev ≡ staging.

• Projection pipeline = LabProjectionWorkflow (console-api Workflow, durable steps): page vectors from D1 → seeded PCA → 50d → seeded UMAP 2D (umap-js, epoch-capped; mode:'pca-only' fallback if CPU limits bite) → seeded k-means → cluster labels from facet majorities (top deity/theme/language — free, deterministic; LLM polish optional) → prior centroids (§6.1) → lab_projection / lab_clusters / lab_centroids + a lab_rebuilds status row. Triggered from the Atlas (lab:run), recompute on catalog delta.

• Browser draws points with a plain <canvas> scatter (fine to ~50k; sample or heatmap beyond — deck.gl only if that day comes).

• Clusters are the coverage unit: over-circulated vs starved (long-tail locales, DS-9) vs white space (→ brief). The map is a serving tool and a production-planning tool.

• Shipped as Atlas v2 — cluster-first: on-demand seeded k-means over the FL-15 store (no projection Workflow dependency; the UMAP point-scatter is a deferred zoom layer), three views — cluster gallery (medoid mosaics) · constellation (similarity-positioned bubbles) · coverage gap-finder. The reasoning lives in ATLAS.md — read it before changing the Atlas.

5. Diversity & vector-search testing

• Metrics exposed: intra-list distance (ILD, mean pairwise embedding distance in a served page), coverage/entropy across clusters & attributes, circulation Gini, novelty-to-user (distance from seen-set). MMR λ is the knob (DS-8/§8). DPP is a later upgrade for principled quality×diversity; MMR first.
• Vector-search bench validates “find the vein, never the same nugget”: run a taste vector → see the neighborhood → watch novelty + rarity + circulation-cap pick a distinct item within it. If that feels right, DS-12 is working.
• The bench shipped as a full retrieval playground — text ⊕ image(s) ⊕ item weighted queries, every retrieval/MMR knob, funnel + ILD diagnostics, A/B compare, and a hard Cohere-budget guard (per-source vector caching). The reasoning behind every knob lives in RETRIEVAL-BENCH.md — read that before changing the bench.

6. Simulating a feed — two-layer simulator, one event pipe

No behavioral data yet ⇒ “how would a user feel scrolling” is a counterfactual. The v2 answer (replacing v1’s Tier A/B framing, informed by §12.1): three event sources at three fidelities, all emitting the identical event stream into one closed loop.

• Layer 0 — the founder. You are the user: scroll the phone frame; the UI captures real dwell_ms (IntersectionObserver timestamps), fast-swipe → skip, share taps → share_intent. RecoWorld explicitly endorses humans as drop-in simulator replacements — this is the ground truth the synthetic layers get calibrated against.
• Layer 1 — statistical persona (default; free, instant, reproducible). Synthesize a user from coldStartVector(OnboardingSignals) (§6.1); per batch run a position-biased examine → engage choice model (cascade-model shape from the click-model literature): P(engage) = f(taste·item, novelty, quality, fatigue, position), seeded RNG throughout so the same seed reproduces the same session. This is the coverage/sanity harness — deterministic, catches gross errors immediately, cheap enough for population runs.
• Layer 2 — LLM persona agent (sparing; calibration + hypothesis tests). An LLM role-plays the persona over the same state, RecoWorld-style think → act → update mindset, and judges “would I forward this, and to whom?” per item. Via AI Gateway, budget-capped (Agent4Rec’s published cost: ~$16 per 1,000 simulated users — affordable for targeted runs, wasteful as the default). Validate against founder judgment, then trust for relative A/B only.
• Tier C — off-policy replay / IPS (needs logs; unchanged, future). Replay logged sessions, importance-weight by propensity, estimate counterfactual share-rate of a new policy without shipping. Slots in once DS-10 share-intent logging has volume — and the full-fidelity lab_events sessions (§6.2) are exactly the input shape it needs.

The closed loop (all layers): events → in-session re-rank of the remaining pool (applyEvents nudges the working taste state; skips repel, shares attract) → session aggregate nudges the persona’s taste vector at batch boundaries → next rank() call differs. This mirrors DISTRIBUTION §7’s loops 1–2 exactly; loop 3 (offline profile refresh) is out of Lab scope v1. Watching a deity’s taste_affinity term drop after you skip three of its cards is the acceptance test for the loop.

Topology (FL-14): the simulation steps server-side — POST /api/lab/sim/step takes the policy + an opaque user_state blob (taste vector base64, seen-set, fatigue; ~5 KB) + the latest events, and returns the next ranked batch with per-item term breakdowns + updated state. Layer-1 choices run server-side too (same @namaste-ji/feed code the tests exercise); the client is presentation + founder-event capture only.

Non-negotiable (FL-9): every session runs two arms with the same seed and persona — the candidate policy and current production (fallback: defaultPolicy(), flagged). Every metric is a diff. Compare policies, don’t vibe on one.

6.1 Cold-start & onboarding simulation — the realistic persona

The Lab must reproduce the actual onboarding funnel so a simulated feed feels like prod for a brand-new user. The real funnel is deliberately thin:

phone (SMS / WhatsApp OTP)  →  language (optional, skippable)  →  deity (optional, skippable)

A “skipped-everything” user is not a blank slate. Even with both optionals skipped, the app has always-on context — and the context floor (DS-12) is strong enough that cold-start is largely a non-problem:

⚠️ Don’t trust phone-circle → region: post-portability it’s unreliable. Lean on IP geo + device locale, treat circle as a weak tiebreak.

Cold-start taste vector = a blend of prior centroids (LLM-free, cheap). The projection workflow (§4) computes prior centroids in the multimodal space — per-deity, per-language, per-region, per-weekday and per-event (mean vectors of assets tagged accordingly) — into lab_centroids (D1 blobs; FL-17). Then a shared function

coldStartVector(onboarding, context) =
    w_deity·deity_centroid?     // present → sharp; skipped → dropped (flatter prior)
  ⊕ w_lang ·language_centroid    // explicit, else inferred from device locale
  ⊕ w_geo  ·region_centroid
  ⊕ w_time ·(weekday_centroid ⊕ active_event_centroids)   // blended at request time —
                                 // the festival/weekday floor; makes the day-scrubber free

is the taste vector a new user starts with — pure context, behavioral_knn weight ≈ 0. As signal accrues, DS-12 blends behavioral in. The Lab and the serving path call the same coldStartVector() and the same rank() (FL-1) — the only difference for a cold-start user is that shares are simulated (or founder-performed), not real. That’s what makes the sim honest. Each prior centroid + the blended start point renders on the Atlas (the mockup marks the deity prior with a ⊕ anchor).

Signal-availability matrix + population mode. Onboarding drop-off is real, so a single hand-picked persona lies about launch quality. The Lab models a cold-start population: sample N users from a configurable distribution over (geo, language-present?, deity-present?, maturity) — e.g. “~40% skip language, ~55% skip deity” (tune from the real funnel later) — run Layer-1 sessions for both arms, and report aggregate feed quality: mean/percentile diversity/coverage/predicted-share, and % of feeds that clear the quality bar. That answers the real next-billion question — is the cold-start experience good across the whole incoming population — not just for one tidy archetype.

Session warming. As the simulated user scrolls, the same in-session / session-aggregate loop (§7) nudges the taste vector from the prior toward behaviour; a cold → warmed dial (warmup_sessions) lets you feel day-1 vs week-2. Guarded by the exploration + coverage floor (DS-5) so warming can’t collapse into a filter bubble — the sim keeps the everyone-content (big festival) surfacing.

Realism contract (why it feels like prod): (1) one OnboardingSignals Zod schema in @namaste-ji/shared, consumed by both real onboarding and the Lab persona builder — every optional field has an explicit skipped state; (2) one coldStartVector() prior-assembly; (3) one rank(). Build the persona from the schema, never from ad-hoc Lab fields — same isomorphism principle as the ranker (FL-1), applied to the user.

6.2 Event taxonomy & sink — the “simple analytics pipe” (FL-12/FL-13)

The event stream is the spine of the whole Lab, and it is deliberately the same contract the consumer app will emit later (DS-10):

• FeedEvent — versioned Zod discriminated union in @namaste-ji/shared: impression | dwell | skip | open | share_intent, each with context {ts, session_id, seq, surface, locale, time_of_day, source: 'founder'|'persona'|'llm'|'app'} and FEED_EVENT_SCHEMA_VERSION.
• Sink = D1 (lab_events), full fidelity, behind an EventSink port. Not Analytics Engine yet, deliberately: AE is sampled/aggregate-oriented (great for Loop-B counters, wrong for per-session replay), lab volumes are tiny, and full-fidelity sessions are exactly what Tier-C off-policy replay and “why did my feed evolve” debugging need — plus free joins with scenarios/runs.
• When the consumer app exists, the AE sink slots in behind the same port for aggregate counters, and the D1/DO path handles per-user state (DS-15). Nothing built now is thrown away; the taxonomy is the durable asset.
• Layer-0/1/2 events are directly comparable (same schema, same table) — which is how the Layer-1 choice model gets calibrated against founder traces on identical batches.

7. Signals & real-time — three loops at three latencies

A daily-ritual product does not need (or want to pay for) a TikTok-style online-learning ranker. Split the “real-time” into three:

Event taxonomy (minimal, DS-10): see §6.2 — one shared FeedEvent contract. Tooling on Cloudflare (app-era): Analytics Engine for fire-and-forget aggregate counters (share-tap by content/locale/attribute) — sampled/aggregate, great for Loop B, not per-user history; Durable Object per user for session state; D1 (PII plane, DS-15) for the durable taste vector + profile; KV for the cached playlist; Queues + Cron for the offline refresh. The Lab’s lab_events doubles as the signal-replay substrate so “predicted share-rate” gets grounded in real aggregates once they exist.

8. Best factors — the feature taxonomy that moves this objective

quality × context-right × novel-to-you × not-over-circulated (DS-4). The load-bearing features:

• Share-propensity by attribute cluster (not by item — DS-4): segment-level share rate of a style/motif/deity/palette.
• Novelty-to-user (distance from seen-set) + rarity/circulation (global + locale send count) — the two inverted terms, first-class.
• Event-proximity (days-to-event × user affinity — §11).
• Time-of-day × calendar fit (morning GM, Monday deity, lunar).
• Locale/language match + a coverage floor (DS-9).
• Quality floor (eval/Judge) gating everything — novelty must never surface rare junk (§8 of DISTRIBUTION).

Each is a slider in the policy editor; the Lab shows marginal contribution per item (RankedItem.terms — the “why this, here” card).

9. Recommendation logic — the recommendation

Keep the committed funnel: context floor ⊕ behavioral kNN ⊕ fresh ⊕ rare → eligibility → score → MMR diversify → pace (DS-11/DS-12). Two refinements:

1. Evolve the multiply-heuristic into a value-model when ready: score = Σ wᵢ·P(actionᵢ) — a weighted sum of predicted actions (share, open, dwell), the IG/Twitter “heavy ranker” shape. The current heuristic is this with hand-set weights + crude P’s. Keep the score additive-in-log-space so the learned upgrade is a swap, not a rewrite. Don’t build the model now.
2. Exploration = segment-level Thompson sampling on share-rate (DS-5), never per-user bandits (no data, intractable at this cadence). The Lab sets + visualizes the budget.

10. LLMs for feed generation — where they earn their keep (all offline)

Five roles, all slow-brain, never in the per-request loop (DS-14):

1. Profile/persona inference — behavior → written taste profile + refreshed taste vector.
2. Cluster labeling — polish pass over the facet-majority labels (§4), optional.
3. Editorial composition — an LLM assembles a themed morning set for a cohort respecting diversity + events, then it’s distilled to embeddings/rules the fast path reads (Spotify-BaRT-style).
4. Policy copilot (killer Lab feature) — the LLM reads recent sim metric deltas and proposes weight changes (“novelty is starving festival coverage; drop λ 0.1, raise event_priority”). Writes the recipe, founder approves — meta-optimization with a human gate.
5. The LLM persona agent (§6 Layer 2).

Mantra: the LLM writes the policy; vector math serves the feed.

11. Events pre-positioning — the personalized runway

Model event lead-up as a personalized runway, not a flat campaign window:

• Lead time = f(user affinity). High-affinity (declared prefs + last-year engagement) users see preparation content 10–14 days out; low-affinity users only 2 days / day-of.
• Runway sub-types tagged on time_bound assets: anticipation/countdown → preparation/how-to → day-of greeting → afterglow. The ramp selects the stage, not just “Diwali content.”
• Ramp curve ties to the existing window (promote_from → peak → decay_until, DIST §6), but personalized: same window, different per-user entry point.
• Lab surface: the day-scrubber shows each persona’s runway building up — a high-affinity user’s feed warms toward Diwali two weeks out while a low-affinity user’s doesn’t. The “help them prepare and build up” experience, made visible and tunable.

12. How other apps do it (mapped to our choices)

• Pinterest — closest analog. Visual, save/send-oriented, PinSage graph embeddings. Borrow visual-first retrieval + “more like this vein”; their whole product is find-something- to-send.
• Twitter (open-sourced) — SimClusters. Community-based sparse cluster embeddings + multi-task heavy ranker. Our cluster map is SimClusters-lite.
• TikTok — content-first graph-free retrieval + exploration laddering (borrow, DS-11) + real-time online learning (skip — cost/cadence).
• Instagram — sends/saves as NSM + value-model = weighted sum of predicted actions (§9.1).
• YouTube — candidate-gen/ranking split; optimize the real goal not the proxy (good sends, not taps).
• Spotify — BaRT — bandits + editorial + embeddings (the LLM-as-offline-curator pattern, §10.3).

What’s most robust for us is exactly the committed inversion: content-first (no graph), share-intent objective, anti-virality, cheap vector serve + offline LLM brain. The Lab makes that inversion tunable and verifiable before it ships.

12.1 Simulation & tooling inspirations (v2 research)

• RecSim (Google, arXiv:1909.04847) — the canonical simulator decomposition: user latent state → choice model → response → state transition. Our Layer-1 persona is this shape.
• Agent4Rec (SIGIR’24, github) — LLM generative agents (profile/memory/actions) browsing page-by-page; ~$16 / 1,000 simulated users. Grounds Layer-2’s cost envelope and page-by-page session shape.
• RecoWorld (arXiv:2509.10397) — per-item think → act → update mindset loop; use real items, never synthetic content; session-level metrics over item rewards; humans can replace the simulator (→ our Layer 0); validate simulators against human judgments on shared batches (→ our calibration story).
• Click-model literature (cascade/examination models, position bias, dwell-time as quantified click; survey: arXiv:2306.08550) — Layer-1’s choice model is a standard position-biased cascade, not an invention.
• Apple Embedding Atlas (github) — interaction grammar for the Atlas page: density clustering, auto-labels, cross-filter, NN search.

13. Cloudflare mapping

14. Decisions log (extends DISTRIBUTION.md DS-*)

15. Build plan — the PR ladder

Each PR is small, atomic, and independently reviewable (repo convention). Sequence:

Phase-2 items (unchanged intent, post-ladder): cluster-gap → brief hand-off, policy copilot, signal-replay against real aggregates, DPP upgrade.

16. Open questions

• Layer-1 calibration: which choice-model parameters (position-bias curve, engage threshold, fatigue rate) best reproduce founder Layer-0 traces — collect traces first, fit second.
• LLM persona validation: how well Layer-2 tracks real share-intent once logs exist (same question as v1, now with a concrete comparison substrate — lab_events).
• Cold-start prior weights (w_deity / w_lang / w_geo / w_time) + realistic onboarding-skip distribution — seeded by hand, fit from the real funnel later.
• Persona library: hand-authored archetypes vs clustered real cohorts (once behavior exists).
• Embedding model: Cohere embed-v4.0 shipped (multimodal, 1024-dim); revisit dim/model swaps via the embedding_model provenance column when contenders (Voyage/Jina) warrant a bake-off — the FL-4 copy makes re-projection cheap.
• UMAP CPU inside a Workflow step at larger catalog sizes (the pca-only fallback is the hedge; revisit renderer + sampling beyond ~50k points).

17. Build sequence & prerequisites

The Lab is only as good as the catalog it explores. Status of the agreed order:

1. ✅ Pipeline deployed to staging (Content Director → Creator → embed → staged).
2. ✅ Model config per env (minis everywhere; KV-driven).
3. ✅ Content generated — a real, diverse catalog exists in staging.
4. ⬜ FL-4 backfill (PR-2) — persist + backfill raw embeddings so projection has vectors.
5. ⬜ Prereq package (PR-1): FeedPolicy + rank() + OnboardingSignals + coldStartVector() in @namaste-ji/feed / @namaste-ji/shared.
6. ⬜ Lab surfaces (PR-3…9) against the populated staging catalog.

This doc owns the Lab; the content-production plane is docs/CONTENT-PRODUCTION.md.