Docs / feed-serving.md · mirrored from the repo

Namaste Ji — Feed Serving & Personalization

Status: design. Researched + agreed direction (2026-07-18); the engine (packages/feed) is built and Lab-proven, the serving wiring is not. Living doc; decisions carry FS-* IDs.

This doc owns how the feed is actually served: the three-loop architecture, per-user slate precompute + buffering, live in-session adaptation, time-bound content injection, the user representation (multi-interest vectors + negatives), and the offline LLM “slow brain”. It operationalizes DISTRIBUTION.md (DS-* — the why and the objective) using the engine specified in console/FEED-LAB.md (FL-*), the client design in FRONTEND.md (FE-*), and the signal pipeline in EVENTS-AND-ANALYTICS.md (EV-*). Read those first; this doc does not restate them.

1. Research verdicts (what the literature says about our hypothesis)

The founding hypothesis — represent users as vectors over the shared multimodal embedding space, use negatives, and let offline LLMs prepare buffered per-user batches — is validated by 2023–2026 research and production art, with two corrections:

1. Multi-interest, not one vector (FS-1). A single averaged taste vector lands in a meaningless midpoint between distinct interests (PinnerSage, arXiv:2007.03634; off-manifold drift formalized in RecSys’23 10.1145/3604915.3608837). Fix: cluster engaged items over our frozen Cohere embeddings; represent each cluster by its medoid (a real item embedding — stays on-manifold); weight by importance = Σ w_action · ξ^Δdays (ξ≈0.9). Decay lives on importance weights, never baked into the vector (Okura et al., KDD’17: decayed vector averaging is worse than plain averaging). Our catalog’s low intrinsic dimensionality (deity × occasion × language × style) ⇒ 2–6 clusters per user, cap 5, merge below the ~0.85 within-vein similarity floor found by Atlas.
2. Negatives are penalties, never query arithmetic (FS-2). query = pos − α·neg retrieves noise: multimodal/CLIP-family spaces cannot represent negation (arXiv:2501.09425); Rocchio’s negative term has been known-weak for decades. What works (Qdrant best_score pattern): 1–3 negative centroids from confident negatives only (fast skip after a ≥50%-visible impression, explicit hide — never mere non-engagement), applied at rank time as score −= γ · max_j sim(item, neg_j) (γ≈0.3, policy-tunable) plus a hard filter at sim > θ≈0.9. Negative centroids expire after 30 days.

The strongest-evidence component is the cross-modal probe trick (FS-3): LLM-written affirmative text queries constrained to our attribute vocabulary, embedded as retrieval vectors — Amazon GPT4Rec (arXiv:2304.03879, +75% Recall@K) and Google’s deployed interest-exploration system (RecSys’24, arXiv:2405.16363, controlled-vocabulary generation). Because the space is multimodal, the same text probe retrieves images directly. Probes must be affirmative — never “Ganesha but not dark tones”.

The LLM-buffered-batches idea decomposes into four individually-validated offline patterns: NL user profiles (Meta EmbSum arXiv:2405.11441, Google interest journeys arXiv:2305.15498), incremental profile updates (PURE), constrained probe generation (above), and listwise slate re-ranking (RankZephyr-class: a small model matches GPT-4 at this). See §7.

2. The three-loop architecture

At our scale the industry’s four-stage funnel collapses: retrieval + pre-ranking exist to amortize model cost over billion-item corpora; with a small first-party catalog, precomputed embeddings, and a deterministic rank(), the online path is filter + deterministic re-rank, everything intelligent in precompute (FS-4). This is also the only shape that fits Workers CPU budgets and Vectorize’s eventual-consistency model — Vectorize never sits on the hot path (FS-5; extends FL-4/FL-15: the D1 embedding copy + in-Worker store serve retrieval until eligible pools exceed ~50k, then Vectorize becomes the offline/near-line pre-filter).

Structural gift: no follow graph ⇒ no celebrity fan-out problem ⇒ pure push (precomputed per-user slates) is safe, costing O(DAU), not O(publishes) (FS-6).

LOOP 1 · offline brain            nightly + festival-window triggers
  global pools per (language × festival-window × day-part):
  evergreen · festival/campaign (ramp→takeover→hard-cut) · exploration
  (impression-capped, Beta priors seeded from embedding neighbours)
  + ALL LLM work (§7)                     → KV pools, D1 stats/priors
LOOP 2 · per-user slate precompute pre-dawn (staggered) + on session end
  sample 2–3 interest clusters (importance × calendar relevance)
  → query per medoid + probe vectors → ∪ context/fresh/rare pools
  → seen-set filter → rank() with slot quotas
  → versioned slate (50–100 ids + light metadata)  → KV feed:{user}:{v}
LOOP 3 · online serving           Workers; zero model calls, zero Vectorize
  first paint from KV slate head (or overnight pre-positioned page)
  client: ~20–30-item metadata buffer, prefetch only next 2–3 images,
  refill at ≤5 unseen; in-session re-rank on-device (same rank()) +
  server re-rank at refill; slate versioning = freshness handoff

3. User representation — UserState v2

Extends packages/feed UserState (v1 fields remain for compatibility; taste stays as the cold-start/fallback vector). New fields (FS-7):

Rebuilt by the Loop-2 Workflow from the event stream (EV); in-session, v1’s applyEvents nudges remain the transient layer on top. Cluster sampling at retrieval: 2–3 clusters ∝ importance × calendar_relevance (festival proximity multiplies the matching cluster — the calendar is this product’s sequence signal).

4. Ranking & policy — FeedPolicy v2

rank()’s additive shape survives unchanged (FL-2’s value-model evolution path intact). New policy fields (FS-8):

• negative_penalty: { gamma: 0.3, hard_filter_sim: 0.9 } — the FS-2 term.
• slot_quotas (per 10-card page): { festival: 2 (ramp→takeover on the day), exploration: 1–2, personalized: rest } — quotas are explicit slots at assembly, the Netflix/Spotify slot-and-curated-pool pattern; a festival card competes for festival slots, not against evergreen scoring (FS-9). Existing event_ramp supplies the boost curve within the festival pool; time-bound decay past the window is a hard cut, not a half-life.
• exploration_priors: { neighbor_k, prior_strength } — new items inherit a Beta share-rate prior from their k nearest catalog neighbours with stats (IDProxy/Dynamic-Prior-TS pattern), then update per impression/share; exploration slots Thompson-sample from the under-exposed pool under a per-item daily impression cap (FS-10). This replaces any bandit infrastructure — the anti-virality rarity boost already doubles as an exploration bonus.
• Diversity stays MMR + sliding-window attribute caps (YouTube ran exactly this at scale for years pre-DPP); DPP is a deferred refinement (FS-11).

Seen-state: an exact per-user ID set, 90-day window (~40–80 KB, one D1 row/DO) — bloom filters (age-partitioned) only if it outgrows ~50 KB (FS-12).

5. Slates, buffering, and live adaptation

• Slate: 50–100 ranked content_ids + light metadata per item (cluster tag + truncated embedding + facets), versioned, in KV feed:{user_pseudo_id}:{version}. Serving maps it through the unchanged PlaylistPage contract.
• First paint: slate head from KV (single-digit ms). Morning ritual: the client pre-positions the first page including images overnight on unmetered Wi-Fi (DS-7/FE-1) — first paint needs no network.
• Client buffer (FS-13): hold ~20–30 items of metadata; prefetch only the next 2–3 images (measurement studies: >40% of aggressive prefetch is wasted bytes — decisive on metered Indian data plans); request refill when ≤5 unseen remain.
• In-session adaptation, two tiers (FS-14): (a) on-device — the same rank() (FE-1’s isomorphism is the point) re-orders the remaining buffer after every interaction: skip ⇒ demote same-cluster items, share/long-dwell ⇒ promote — pure arithmetic on shipped tags, zero round-trips (EdgeRec pattern: ~300 ms adaptation); (b) server-side — each refill carries the session’s FeedEvents; the Worker re-ranks the remaining slate + next pool chunk with session state (running seen-set, per-attribute fatigue).
• Freshness handoff (FS-15): a fresh ranking is a new slate version; the client finishes its current page then continues on the new version. The in-view page is never mutated. request_id (EV-5) = {slate_version}:{page} — the training-data join key for free.

6. Loop 1 — the offline brain (pools & priors)

Nightly + festival-window-triggered Workflow: assemble the day’s global pools per (language × geo-relevant festival window × day-part) from the catalog + events calendar; recompute circulation/rarity stats and Beta posteriors from the event lake (EV); refresh prior centroids (lab_centroids pattern). Day-parts collapse to simple IST slots (morning greeting ≈ 5–10am, evening devotional) — one timezone. All Vectorize/embedding-heavy work happens here, where eventual consistency is irrelevant.

7. The LLM slow brain — per-user curation artifacts

Nightly (or 2×/day pre-dawn) batch job per 7-day-active user; Heimdall-shaped (Workflow + OpenAI Batch API + Langfuse prompts + versioned artifact). Cost shape: one mini-model call/user/day, ~500–1500 tokens ⇒ fractions of a cent per user-day (FS-16).

Inputs: interest clusters (attribute profiles + top items), negative-centroid attributes, 7-day event summary, next-72h calendar, previous NL profile. Outputs — one JSON artifact (D1, KV-mirrored):

1. NL user profile — 3–5 named interests + sending persona, incrementally updated; human-readable in the BO console. Doubles as DPDP algorithmic-due-diligence evidence and gives Heimdall an auditable surface (FS-17).
2. 3–8 affirmative probe texts constrained to the attribute vocabulary, incl. 1–2 deliberate adjacent-interest exploration probes; embedded once in batch → §3 probe_vectors.
3. (optional, phase D) the curated buffer: listwise re-rank of ~100 pre-retrieved candidates into tomorrow’s 20–30-card batch, ~70/20/10 exploit/adjacent/explore, festival slots pinned.

Serving reads artifacts; live vector retrieval is the fallback when stale. The ritual framing falls out free: “your morning batch is ready” is an appointment mechanic and end-of-batch a designed stop point — aligned with time-to-a-good-send (DS-2), the ethical version of habit-forming (FS-18).

8. Compliance tie-in (EV)

The contextual fallback is the same code path: rank() with the cold-start context floor and behavioral pools off (candidate_mix.behavioral_knn = 0, no UserState v2) is the DPDP-compliant non-personalized feed for minors and consent-withdrawn users — one policy flag, not a second system (FS-19). Slates key on user_pseudo_id (EV-11); NL profiles live in the PII plane, region-pinnable.

9. What NOT to build (research-supported)

Trained two-tower / MIND / ComiRec (frozen-embedding clustering captures most value below ~10⁵ users) · deep sequence models (session-kNN-class baselines win at small scale; our sequence signal is the calendar) · DPP (MMR + window caps is the proven pre-DPP production state) · semantic IDs / HSTU / OneRec (research-scale; our catalog fits in a prompt) · per-request LLM calls (DS-8) · bandit infrastructure (fixed quota + seeded Beta priors, re-tuned offline) · query-vector subtraction (FS-2) · bloom filters before the exact seen-set outgrows ~50 KB (FS-12).

10. Phased build plan

Every phase is Lab-testable (FL-9 two-arm diffs) before touching production; each PR small and single-purpose.

11. Decisions log

12. Open questions

• OQ-1: cluster algorithm details — agglomerative vs k-medoids at our history lengths; validate merge threshold against Atlas’s vein floor in the Lab.
• OQ-2: slate refresh cadence vs KV write cost at scale — pre-dawn only vs +post-session; measure.
• OQ-3: on-device re-rank payload — truncated-embedding dims vs cluster-tag-only (bundle-size/adaptation-quality trade; FE-2 budget).
• OQ-4: curated-buffer (§7 output 3) value over probes+profile alone — Lab A/B before building phase D’s re-rank stage.
• OQ-5: when Loop 2 moves from cron-over-actives to event-triggered (session-end Queue) — DAU threshold.

13. Key references

PinnerSage arXiv:2007.03634 · kNN-Embed arXiv:2205.06205 · negation blindness arXiv:2501.09425 · Qdrant best_score article · GPT4Rec arXiv:2304.03879 · LLM interest exploration arXiv:2405.16363 · EmbSum arXiv:2405.11441 · interest journeys arXiv:2305.15498 · Instagram Explore Meta blog · Twitter Home Mixer repo · EdgeRec arXiv:2005.08416 · Netflix in-session RecSys’22 · Spotify algotorial blog · YouTube DPP CIKM’18 · exploration LTV arXiv:2305.07764 · Dynamic-Prior TS arXiv:2602.00943 · Okura KDD’17 paper · Vectorize limits docs