Docs / console/atlas.md · mirrored from the repo

The Atlas — why it’s cluster-first

Surface: /lab/atlas · Design home: FEED-LAB.md §4 · Sibling why-doc: RETRIEVAL-BENCH.md

Read this once and the three views should feel inevitable rather than arbitrary.

1. The questions this surface answers

You are the creative director of a catalog that agents produced — you didn’t hand-pick these 694 assets, so you need instruments to see what you own:

1. What do I have? The catalog’s natural groupings (veins), browsable in seconds.
2. Where are the gaps? Which deity × language cells are thin or empty — the #1 production-planning signal (DS-9: starved veins → briefs). The lakshmi case (2 assets vs shiva’s 120) was found by accident in the search bench; the Atlas makes that discovery a first-class view.
3. Is each vein healthy? Coherent, or a pile of near-duplicates (the DS-12 “same nugget” risk)? Are the captions/attributes honest (mislabeled items)?
4. Later, once serving data exists: over-circulated vs starved overlays.

2. Why cluster-first, not a point scatter

The original design projected all items to 2D (PCA → UMAP) and drew a scatter. Two things argued for a rethink:

• Every question above is cluster-level, not point-level. A 694-point scatter makes you do the clustering with your eyeballs; cluster cards with real thumbnails answer “what do I have” directly.
• The UMAP projection Workflow is CPU-limited on staging (open bug). That was the occasion, not the reason — but it forced the useful question: what actually needs a projection? Answer: only the aesthetic layer.

So Atlas v2 makes the cluster the primary object, computed on demand in the Worker: seeded k-means over the same in-memory vector store the search bench uses (FL-15). No projection Workflow, no Vectorize — and, worth stating plainly:

Zero embedding computation. The Atlas only reads the vectors already sitting in catalog-D1 content_embeddings (the backfilled copy). No Cohere calls, no re-embedding, no backfill — clustering is pure math over stored vectors. The only caches involved are in-memory: the store itself, and cluster results memoized per (catalog-version, k, seed) so the k slider is instant after its first compute.

k-means runs on the raw unit vectors (squared Euclidean on unit vectors is monotone in cosine, so no PCA step is needed) and is deterministic under the seed: a (k, seed) URL reproduces the same clusters bit-for-bit until the catalog itself changes.

3. The three views

Clusters (gallery) — “what do I have?”

One card per cluster. The mosaic shows the medoids — the items closest to the centroid — because they are the honest representatives (random picks would misrepresent a loose cluster; edge picks would caricature it). The label is the facet majority (shiva · hi · timeless — free, deterministic, no LLM). Two chips carry the health check:

• Purity — the top deity/theme’s share. 95%+ = a clean identity; ~50% = the cluster straddles two ideas (raise k to split it).
• Tightness (⌀) — mean member→centroid cosine. Very high (> 0.85, amber) = near-duplicates: the vein exists but has no internal variety, which is exactly the DS-12 failure serving will hit (“never the same nugget” needs distinct items within the vein). Low = the cluster is incoherent (raise k, or the space is telling you the content is genuinely scattered).

Map (constellation) — “which veins sit near which?”

The spatial view — but of clusters, not points. Bubbles (the cluster’s medoid image, diameter ∝ √n) are arranged by a d3-force simulation over the centroid similarity matrix (springs = similarity, charge + collision keep it legible), with d3-zoom pan/wheel-zoom and draggable bubbles — drag one and its neighborhood follows, which is a surprisingly good way to feel which attachments are strong. Determinism survives interactivity: the simulation runs on a seeded random source and is pre-ticked to equilibrium before first paint, so the same (k, seed) URL settles into the same map. Hovering a bubble highlights its links and dims the rest. Near bubbles = adjacent veins — where “distinct item within the vein” can borrow from next door, and where a new brief would sit in the existing geography. All client-side, zero server CPU; the true point-level UMAP scatter remains a deferred layer (as a zoom inside a cluster) once the Workflow bug is fixed.

How to read it honestly (the in-map legend repeats this):

• Only relative closeness carries meaning. A stress layout has no coordinate system: left/right/up/down are artifacts of the seed, and the axes mean nothing. Two bubbles near each other = similar centroids; that is the entire vocabulary.
• Edges are each vein’s two most similar neighbors — always two. They form a relatedness skeleton, not a route map: an isolated cluster still gets two lines, so read line strength (opacity = actual centroid similarity, shown on hover), not mere presence. A vein whose two “nearest” links are faint is genuinely alone.
• The fuzzy halo is internal spread — bigger halo = more variety inside the vein. No halo + amber ring = a pile of near-clones.

What the map says about diversity — two different levels, one picture:

• Within a vein (the halo / the amber ring / ⌀): can serving show a user this vein twice without repeating itself? If ⌀ > 0.85, no — brief variations (new compositions, palettes, moods for the same subject), not new subjects. The header’s weighted within-vein ⌀ stat compresses this to one catalog-level number (today ≈ 0.91 — variety is the catalog’s scarcest resource).
• Between veins (positions + edge strength): where can the feed hop without jarring? Adjacent veins are the natural “distinct but related” moves DS-12 wants; a brief that lands between two strong veins extends the geography rather than duplicating it.

Coverage — “where are the gaps?”

A deity × language heat table (temporal split in the tooltip), computed from facet counts alone — no vectors at all. Cells with fewer than 5 assets wear an amber ring; empty cells show ·; the gaps chip row lists the thinnest real combos first (lakshmi × hi-Latn 0). — rows/columns mean absent metadata (theme-led content), deliberately distinguished from missing content. Every real cell and gap chip clicks through to the filtered search bench: spot the gap → see what little exists → brief it. (A one-click ”→ brief this gap” wired to the Creative Head is the natural follow-up once briefs accept seeded constraints.)

4. k and seed — the carving knobs

k-means doesn’t discover “the true number” of veins; it carves the space into k cells. That’s a feature here: k is an exploration knob. Low k = broad veins (the five-deity structure); high k = sub-veins (shiva-at-dawn vs shiva-temple vs shiva-abstract). Scrub it and watch which clusters split — a cluster that survives every k is a real vein; one that dissolves was an artifact. The seed picks among equally-valid carvings; keep it fixed for comparable links. (HDBSCAN — organic cluster counts — is the someday-upgrade noted in FEED-LAB §4.)

5. Cluster detail — center→edge as a QA instrument

Members are ordered by similarity to the centroid: archetypes first, oddballs last, with an “edge” divider on the final ~10%. The edge is where mislabeled captions, attribute mistakes, and embedding surprises concentrate — the cheapest QA pass the catalog has. The “next to” chips (nearest clusters by centroid) let you hop veins, and every thumbnail either opens the asset drawer or pivots (↳) into /lab/search as a free item-as-query — the Atlas finds, the bench interrogates.

6. Clustering lifecycle — how recomputation works, and the scale playbook

6.1 How clusters are (re)computed today

There is deliberately no pipeline — recomputation is derived from the data:

1. GET /api/lab/atlas/clusters?k&seed runs seeded k-means over the in-Worker vector store and memoizes the result in isolate memory under the key (store.version, k, seed), where store.version = count : latest-write : model comes from one cheap D1 aggregate (embeddingStats).
2. The catalog changing is the invalidation. Any new or re-written embedding changes store.version, so the very next Atlas request recomputes (~1s) and re-caches. There is no cron, no queue, no “refresh” button, and therefore no invalidation-bug class. Staleness is bounded by the store’s 5-second stats TTL.
3. Isolates are ephemeral and independent. A cold isolate recomputes on first hit; several isolates may each hold a copy (8-entry FIFO per isolate). At ~1s per compute this redundancy is free.
4. Deterministic: same (catalog, k, seed) → bit-identical clusters — that’s what makes an Atlas URL a shareable, reproducible artifact.
5. Clustering never embeds anything. Vectors are read-only input; no Cohere calls, no backfill, ever.

Cost model to keep in your head: k-means is O(N · d · k · iters). Today: 694 × 1024 × 19 × 25 ≈ 3.4×10⁸ ops ≈ well under a second. Memory: N × 4KB (the FL-15 store itself) — ~3MB today, and the ~128MB Worker isolate is the first hard wall, at roughly 25k items (the store’s 50k cap is optimistic at 1024-d).

6.2 The scale playbook (write-down so we don’t rediscover it)

Stage 1 — now → ~5k items: do nothing. On-demand stays under ~5s worst-case first-compute; the isolate cache absorbs repeats. Exact, pipeline-free, correct.

Stage 2 — ~5k → ~25k: precompute-on-change. Move the same k-means off the request path: a Queue consumer (or tiny scheduled Workflow) watches embeddingStats.version, recomputes the default carving when it changes, and writes (content_id, cluster_id, sim) + per-cluster summaries to D1 (the lab_clusters table family from the projection design is the natural home). The API becomes a D1 read; non-default k on demand remains available, just marked slow. Paginate the detail view’s members list (center top-N + an edge sample) — it is the only payload that grows with N.

Stage 3 — ~25k → millions: split assign from carve.

• Assign at embed time (streaming, O(k·d) per item ≈ microseconds). Centroids become a versioned artifact in D1; when the content-creator embeds an item it also assigns it to the nearest centroid and increments that cluster’s running stats (n, Σsim, facet counts). The Atlas then reads precomputed aggregates and stays instant at any N.
• Re-carve offline, chunked. Periodic mini-batch k-means over paged vectors: each Workflow step loads one page, updates centroids, checkpoints centroids to D1, and ends — the next step continues from the checkpoint. No single isolate does unbounded CPU (this is the exact lesson of the UMAP projection CPU-limit failure). Output = a new centroid version; assignments migrate in batches.
• Re-carve on drift, not (only) on a timer. Triggers worth wiring: (a) the share of new items whose best-centroid similarity falls below a floor rises (new veins are forming that no centroid represents); (b) the weighted within-vein ⌀ drifts; (c) catalog grew >25% since the last carve; (d) the embedding model changed — the version key includes the model, so a model swap forces a full re-carve after the re-embed backfill, by construction.
• Keep the k-slider interactive via hierarchy. One coarse global carve (k ≈ 32) plus sub-clustering within a vein on demand — each subproblem is only that vein’s members, so exploratory clustering stays exact and instant at any N.
• Members & neighbors at scale: member pages and within-cluster nearest-neighbor queries go to Vectorize; coverage stays the SQL GROUP BY it already essentially is.

6.3 What never changes, at any stage

• Clustering never re-embeds — stored vectors are the immutable input.
• The API contract (clusters, sim, facet dists) and URL semantics survive every stage — only the computation behind GET /atlas/clusters is swapped, exactly like the CandidateRetriever port on the serving side (FL-15’s strategy applied to clustering).
• Determinism stays a requirement: seeded carves, versioned centroid artifacts.
• Every rendered object stays a k-sized aggregate (k cards, k×k sim matrix, facet tables) — independent of N, which is why the gallery, map, and coverage views look identical at 694 items and at 40 million.

And the standing rule: we refuse to build Stage-2/3 machinery at 694 items. The current approach is exact and has no pipeline to break; each stage gets built when its threshold is actually crossed, not before.

7. What’s deliberately deferred

• Point-level UMAP scatter — as zoom inside a cluster, once the projection Workflow’s CPU limit is fixed. Nothing in Atlas v2 blocks on it.
• Circulation overlay (over-served vs starved per cluster) — needs serving logs (DS-10); the cluster payload is already the right aggregation unit for it.
• Filtered clustering (cluster only the hi-Latn subspace, etc.) — useful, cheap, waiting for a concrete need; keeps today’s cache key simple.

8. Verified live (2026-07-06, staging — zero embed calls)