Docs / console/retrieval-bench.md · mirrored from the repo

The Retrieval Bench — why every knob exists

Surface: /lab/search · Design home: FEED-LAB.md §5 · Serving principle under test: DS-12 (DISTRIBUTION.md)

This is the reasoning document. Read it once top-to-bottom and the page should stop feeling like a wall of knobs — every control exists to answer one specific question about how the catalog’s embedding space behaves.

1. The question this surface answers

DS-12 says the feed must “find the vein, never the same nugget”: given a user’s taste, serving has to (a) land in the right neighborhood of the catalog — the vein — and then (b) pick distinct items within it, never the same-looking greeting twice.

Those are two separate failure modes, and the bench splits them deliberately:

• Relevance half — does a query land in the right vein at all? (text search, item pivot, filters, floor)
• Diversity half — once in the vein, can we pick distinct nuggets? (MMR, λ, ILD, result mix)

Everything else on the page is instrumentation for watching those two halves work.

2. The mental model: one space, exact math

Every asset has one vector: Cohere embed-v4.0, 1024-d, multimodal — caption ⊕ attribute/tag text ⊕ the image itself, embedded together (DISTRIBUTION: server-side personalization on one multimodal embedding index). There is no separate text index or image index to reconcile; “close in this space” already means “looks + reads + means alike”.

Retrieval is a brute-force cosine scan over an in-Worker copy of all vectors (FL-15). At ~700–50k items this is milliseconds, and it buys two properties worth naming:

• Exact — no ANN approximation artifacts. If a result looks wrong, the space is wrong (embedding, metadata), not the index. The bench never gaslights you.
• Identical local and staging — no Vectorize in the loop, so what you probe here is bit-for-bit what the simulator and the future serving path retrieve against.

Scores. Cosine similarity (higher = closer). The distance display is the same number renormalized to (1 − cos)/2 ∈ [0,1] (lower = closer) — a display toggle only, nothing re-runs. Cohere text→image scores land lower than text→text intuition suggests; on this catalog a strong text hit is ≈ 0.55–0.60, not 0.9. Use the histogram, not absolute expectations, to judge.

3. Query composition — why text ⊕ images ⊕ item, as a weighted blend

Three composable sides, because serving composes taste the same way:

• Text = “what I mean”. Embedded with Cohere’s search_query mode into the same space as the assets.
• Images = “what it should look like”. Drop up to 4 reference images; each embeds into the same multimodal space (that’s the point of one joint index — DS-13: query by image, text, or both). Several images average into one direction: “the common visual idea of these”.
• Item (↳ pivot) = “more like this one”. The asset’s stored vector is the query — free, exact, and the same move the feed’s behavioral-kNN pool makes with a user’s taste vector.

Any combination is a weighted blend: normalize(Σ wᵢ·vᵢ) — the exact same blend() helper the cold-start prior uses to mix onboarding signals (FL-10/FL-17). Only the ratios matter; weight 0 mutes a side. Dragging a weight slider is cold-start intuition in miniature: watch the neighborhood morph as intent (text) overrides look (image) or history (item).

Why blend-of-sides instead of one joint embedding? Cohere could embed text+image together in a single call — but then every re-weighting would be a new embed (budget), the sides couldn’t be cached independently, and “text 0.8 / image 0.2” wouldn’t exist at all. Per-side embedding + server-side blending keeps every composition knob free after the first embed of each side. (It’s also how serving will actually work: taste vectors and context vectors are maintained separately and mixed at rank time.)

Image mechanics (why it feels instant and stays cheap): images are downscaled client-side to ≤512px JPEG — embeddings don’t need more — then content-hashed (sha-256), so the same picture never embeds twice, even re-added after a reload (the vector is cached; the pixels aren’t needed again). One honest limitation, stated in the UI: query images are tab-local — a shared URL reproduces text/item/knobs but cannot carry binaries.

The embed as (query/doc) toggle exposes Cohere’s asymmetric embedding: queries and documents are embedded differently on purpose (input_type). Flip it on the same text and compare — the two points land in measurably different places. You want query for retrieval; the toggle exists so the asymmetry is something you’ve felt, not just read about, before you debug a “why is recall bad” mystery someday.

4. The budget guard — why caching is the design center

The Cohere key is a trial key with a monthly call budget, so the bench’s contract is: an embed call is spent only for a new source — a (text, input_type) pair or a unique image — never for a knob turn.

Mechanics: the server embeds a source once and the vector round-trips through the client’s sessionStorage cache (text keyed by model | input_type | text; images by model | content-hash), coming back as query_vector / image_vectors on every subsequent request. Every knob on the page — λ, pool, filters, top_k, floor, weights, compare — is post-embedding, so turning any of them re-queries without re-embedding. Compare runs A then B sequentially so B rides on A’s just-cached vector.

The ⚡/✓ chip next to the search bar makes the invisible visible: whether the last search spent a call or ran free off the cache (session total in its tooltip). Item-only queries never embed at all.

(Related hard-won rule baked into this path: agent workers must try/catch and return JSON errors — an uncaught throw crosses a service binding as a rejected fetch and hides the real error. Learned in #95, applied to /embed-query in #96.)

5. Retrieval knobs

• top_k — how deep into the neighborhood to look. A relative cut: you always get k items, however weak the tail is.
• floor (min cosine) — an absolute quality bar: hits below it are dropped even if that returns fewer than k. Use the histogram to find the vein’s natural cliff, set the floor there, and the funnel then shows you how big the vein actually is. The interplay (relative vs absolute cut) is exactly the eligibility-vs-ranking tension the serving policy has.
• Facet filters (deity, theme, motif, palette, language, region, temporal, status) — a prefilter, applied before the scan, mirroring rank()’s eligibility gates. Filter-first means the funnel and scores describe the constrained pool, not a post-hoc subset. Options carry live counts so you can see coverage before you filter.
• status — lifecycle gate. Today everything embedded is staged; once publishing lands this is how the bench probes “what would the live pool return”.

6. MMR — the diversity half, with rank() parity

Pure relevance ordering in a dense vein returns near-duplicates — the same nugget k times. That is precisely the DS-12 failure. MMR (DS-8) re-picks the top-k one slot at a time, trading relevance against similarity-to-already-picked:

value = (1−λ)·relevance − λ·max_sim_to_picked − λ·attr_repeat_penalty

• λ — 0 = pure relevance (MMR off in spirit), 1 = pure diversity. The interesting range is 0.3–0.7.
• pool — MMR can only diversify what was retrieved. With pool = k there is no room to trade; hence the default 3×k. Raising it lets diversity reach deeper into the tail (watch relevance cost in the badges).
• attr penalty — this is rank() parity, the reason it’s here. The production ranker doesn’t just penalize vector similarity; it penalizes facet runs (repeated deity/palette/motif) with a fixed per-repeat cost. The bench calls the same mmrOrder() that rank() shares its constant with (ATTR_REPEAT_PENALTY, @namaste-ji/feed), so with the box ticked you are watching the production diversify stage, not a textbook approximation.
• ↑/↓ badges — each hit shows its move vs pure-relevance order. Read “↑84” as: diversity paid 84 ranks of relevance to place this item here. That price is the λ conversation made visible.

How to read ILD against this: ILD (intra-list distance) is the mean pairwise cosine distance of the shown set — 0 means clones. Expect it to rise with λ. If λ goes up and ILD barely moves (observed live: 0.085 → 0.100 at λ=0.75 on the shiva vein), the pool itself is homogeneous — that’s a catalog gap, not a ranker bug, and it’s a production signal: the vein needs more visual/compositional variety → a brief. This is how the bench doubles as a production-planning tool.

7. Diagnostics — how to read each one

• Funnel (catalog → filters → ≥ floor → pool → shown): where candidates die tells you which constraint binds. Filters killing 95% = over-constrained metadata; floor killing everything = you’re outside the vein; nothing dying = the query isn’t selective.
• Histogram — the similarity spread of what’s shown. A tight cluster with a cliff = a real vein with a boundary (put the floor at the cliff). A flat smear = the query landed between veins.
• Result mix — facet composition of the shown set. The fastest “same nugget” smell test: 24/24 same deity + same temporal class at λ=0 is expected; at λ=0.7 it’s a coverage warning.
• ILD — see §6. Single number for the diversity half.

8. Compare (A/B) — why fork-materialize over one query

The point of compare is attribution: hold the query fixed, change one knob, see exactly what it did. Hence the two design rules:

• Shared query. Text/asset/blend stay global. If the queries could differ you’d be comparing two questions, not two configs. (Different-phrasings comparison is a possible later mode; it also costs a second embed.)
• B is materialized at fork, not inherited. Clicking Compare copies every knob into b_* URL params, so B is a complete standalone config. “Unset B’s floor” is then unambiguous — no dynamic-inheritance puzzles about whether an absent value means “same as A” or “off”.

The overlap strip is the diff: shared / only-A / only-B / Jaccard, ILD A→B, and the human-readable list of knobs B changed. In the grids, dashed outline = only in that panel; A#7 on a B card = that item sits at rank 7 in panel A.

And because all of it lives in the URL (DESIGN-PATTERNS §5), a compare is an experiment record: paste the link in a note and anyone reproduces it — at the cost of exactly one embed on first load.

9. When search “looks wrong” — the lakshmi case

Type “lakshmi” and the grid fills with shiva. That feels like broken retrieval; it’s the opposite — it’s the most important honest answer the bench gives.

What actually happened: the catalog has 2 lakshmi assets (vs shiva 120, hanuman 114, …). Retrieval finds both, ranks them on top — and then top_k = 24 keeps filling with the nearest remaining neighbors, which is the densest adjacent vein: shiva morning content. top_k is a relative cut; it always fills, however weak the tail.

The UI now says this out loud, three ways:

• Score dots, relative to the top hit (● strong ≥90% · ● close ≥75% · ● tail below) — the shiva filler wears tail-gray.
• The tail divider — in relevance order, a dashed line marks exactly where the vein ran out: “tail — top_k filling beyond the vein”.
• The detection chip — typing a value the catalog knows offers deity "lakshmi" — 2 in catalog · filter to it. One click and the funnel shows the truth: 694 → 2.

And the reframe that matters: this is not a ranker bug, it’s a coverage gap — the production-planning signal DS-9 cares about. A two-asset lakshmi vein means lakshmi seekers get filler; the fix is a content brief, and the bench is where you found it. (Same logic powers §6’s “λ up but ILD flat ⇒ homogeneous pool ⇒ brief”.)

10. Degraded modes — never a silent wrong answer

• no-vectors — content_embeddings is empty (fresh local mesh). Seed or backfill.
• no-embedder — no Cohere gateway configured (local default). Text queries are impossible; item-as-query still works fully — the bench points you there.
• dim-mismatch — the stored space isn’t the query space (local synthetic vectors are 64-d, synthetic-axes-v1). Text queries would be meaningless and the bench refuses to pretend otherwise — that refusal is the design principle.

11. Recipes

1. Find a vein. Text query → look at the histogram → floor at the cliff → the funnel’s ≥ floor count is the vein’s size under the current filters.
2. The same-nugget check (DS-12 core). Compare: A = MMR off, B = λ 0.5 with attr penalty. Watch ILD, result mix, and which items B pays for (↑ badges). If B can’t diversify, brief the gap.
3. Feel the asymmetry. Compare with only the embed as (query/doc) toggle differing (costs one extra embed, once — then cached).
4. Walk the manifold. Pivot ↳ from a hit, keep a whisper of text intent (text weight ≈ 0.3), pivot again. Free, unlimited.
5. Coverage probe. Filter to a thin locale (hi-Latn, a region) and search the big festivals — a missing vein here is next quarter’s content brief. (Or just type a deity and read the detection chip’s count.)
6. Visual vein. Drop a reference image (a look you want more of) — pure-image query shows what the catalog has in that visual direction, regardless of what the captions say. Caption-vs-image disagreements show up here first.
7. Style-anchored intent. Image (the look) + text (the occasion), weights ≈ 0.5/0.5, then slide: at which ratio does the occasion win over the aesthetic? That ratio is a taste-mixing intuition serving will need.

12. Verified live (2026-07-06, staging)

Total Cohere spend across both verification rounds: two calls.