Docs / back-office.md · mirrored from the repo

Namaste Ji — Back Office: Platform & Systems Design

Status: design / brainstorm. Nothing here is built yet. This is the agreed direction, captured so decisions aren’t lost. Treat it as living.

This is the platform/systems design of the back office (BO): how it runs, how we develop it locally, how it deploys, how config/secrets/flags/RBAC work, and how the model layer stays provider-agnostic. It extends ARCHITECTURE.md (two-plane model, Control DO, observability, auth) and CREATIVE-PLANE.md (the BO product surfaces — agent registry, run console, validator inbox, review & approve, audit viewer — see CP §12). This doc is about the substrate under those surfaces, not the surfaces themselves.

1. Scope & the one real tension

The BO is the control plane app (apps/console + a thin control API + the authoritative Control DO). The user asks for: local Docker dev, a best-in-class agent framework, staging-now/prod-later, Statsig, first-class RBAC, provider-agnostic multimodal embeddings, solid config + secret management, and auto-deploy — all buildable solo with Claude Code.

The one architectural tension to resolve up front: “everything Cloudflare” vs “run and test everything locally in Docker Compose.” Cloudflare Workers don’t run in a generic container — they run on workerd. The resolution (decision BO-1):

Cloudflare is the runtime and deploy target. Docker Compose is the local dependency mesh and parity harness, not the production runtime. The Workers/Agents/Workflows run under wrangler dev (Miniflare + workerd, the same runtime as production); Compose orchestrates the non-CF dependencies they call (Ollama, the embedding server, a local vector DB) and ties them together into one docker compose up.

This keeps the everything-CF principle intact while giving a fully local, offline, cheap dev loop. Local dev runs against open-source models via Ollama + a CLIP server; nothing hits a paid API unless you opt in.

2. Guiding principles (BO platform)

1. Same runtime everywhere. Dev uses workerd via wrangler dev; we never simulate Workers with a Node shim. Behaviour parity comes from the runtime, not from mocks.
2. Provider-agnostic by construction. Every model call (LLM and embedding) goes through a narrow interface; no provider SDK leaks into business logic. Swapping Ollama → Workers AI → Jina is a config change, not a refactor.
3. Config is typed, validated, and layered. A worker that boots with bad/missing config fails loudly at startup, not at first request.
4. Secrets never touch the repo or vars. Centralized in Secrets Store; .dev.vars locally; CI injects, never commits.
5. RBAC is designed in full on day one, shipped narrow. The permission model is complete from the start; MVP only populates admin. Adding roles later is data, not re-architecture.
6. Two environments are designed together. Staging exists now; prod is the same topology with different resource IDs and stricter gates. Promotion is a pipeline step, not a rebuild.
7. One person + Claude Code must be able to operate it. Favour IaC-as-TypeScript, one-command bring-up, and push-to-deploy over bespoke ops.

3. Agent framework — research & decision

The user asked for a real survey. Summary of the 2026 landscape for a TypeScript, serverless-first, Cloudflare-resident system:

Decision (BO-2): the durable substrate is Cloudflare Agents SDK + Workflows; the model layer is the Vercel AI SDK behind our own interface; all production calls route through AI Gateway. This confirms ARCHITECTURE.md D3 and pins how model-agnosticism is achieved. Two complementary layers of indirection:

• AI SDK = code-level abstraction (swap provider in TS, one interface).
• AI Gateway = ops-level abstraction (routing, fallback, caching, cost/latency logs, rate limits) — used in staging/prod; bypassed or pointed at Ollama locally.

flowchart LR
  Agent["Agent / Workflow step"] --> Port["ModelPort / EmbedderPort<br/>(our interface)"]
  Port --> AISDK["Vercel AI SDK adapter"]
  AISDK -->|local| Ollama["Ollama / Infinity<br/>(open-source, free)"]
  AISDK -->|staging·prod| AIG["AI Gateway"]
  AIG --> WAI["Workers AI"] & Ext["Jina / Cohere / Voyage / Anthropic …"]

4. Model & embedding abstraction (provider-agnostic, multimodal)

Two ports, both dimension- and provider-agnostic, defined in a shared package (packages/model):

interface ModelPort {            // text/vision LLM calls
  generate(req: GenRequest): Promise<GenResult>;
  stream(req: GenRequest): AsyncIterable<GenChunk>;
}

interface EmbedderPort {         // ONE multimodal embedding space (text + image)
  readonly id: string;           // e.g. "jina-clip-v2", "infinity:jina-clip-v2"
  readonly dim: number;          // pinned per environment
  embedText(texts: string[]): Promise<Vector[]>;
  embedImage(images: ImageRef[]): Promise<Vector[]>;   // R2 key | bytes | URL
}

Multimodal embedding — the provider matrix

The catalog uses one shared multimodal index (DISTRIBUTION DS-13, CP §16): text and image land in the same vector space so we can do cross-modal retrieval, gap maps, and dedup. Provider per environment:

Two hard constraints this abstraction must enforce:

1. Dimension pinning. A Vectorize index is created at a fixed dimension; you can’t mix vectors of different sizes. Pick a canonical dim (e.g. 1024) and use Matryoshka truncation so every provider (local CLIP, Jina, Cohere) emits that size. The local model and the managed model don’t have to be the same model, but they must agree on dim so a locally-embedded row is index-compatible.
2. Re-embed migrations. Store embed_model_id + embed_version on every catalog row. Switching providers = a backfill job that re-embeds into a new index, then an atomic cutover — never an in-place mix. This is a first-class operation, not an afterthought.

The Vectorize local-dev gap (important)

Unlike D1/R2/KV/Queues/Workflows/Browser Rendering — all of which now run fully local under wrangler dev — Vectorize has no local simulation; in wrangler dev it only works via a remote binding to a real index. That breaks “everything local/offline.”

Decision (BO-3): wrap the vector index in a VectorIndex port too, with two implementations — Vectorize (staging/prod) and a containerized Qdrant (local offline). Same interface (upsert, query, deleteByFilter), so the catalog code is identical; only the binding differs by environment. (Alternative for low-volume local: sqlite-vec inside the worker’s local D1 — but Qdrant in Compose matches Vectorize’s filter+ANN semantics more closely.)

5. Local development — Docker Compose

One command (docker compose up) brings up the whole BO against free local models. Compose orchestrates the dependency mesh; the Workers run under wrangler dev (real workerd), with D1/R2/KV/Queues/DO/Workflows persisted locally.

# docker/compose.yaml  (sketch)
services:
  ollama:          # LLMs + text gen for dev (free, local)
    image: ollama/ollama
    volumes: [ollama:/root/.ollama]
    ports: ["11434:11434"]

  embeddings:      # multimodal CLIP embeddings — Ollama can't do image embeds
    image: michaelfeil/infinity
    command: v2 --model-id jinaai/jina-clip-v2 --port 7997
    ports: ["7997:7997"]

  vectordb:        # local stand-in for Vectorize (no local sim exists)
    image: qdrant/qdrant
    volumes: [qdrant:/qdrant/storage]
    ports: ["6333:6333"]

  bo-api:          # control API + agents/workflows via wrangler dev (workerd)
    build: ./services/console-api
    command: wrangler dev --env local --ip 0.0.0.0 --port 8787
    env_file: [.dev.vars.local]
    depends_on: [ollama, embeddings, vectordb]
    ports: ["8787:8787"]

  bo-web:          # the console UI (Vite/Next dev server)
    build: ./apps/console
    command: npm run dev
    depends_on: [bo-api]
    ports: ["3000:3000"]

volumes: { ollama: {}, qdrant: {} }

Local parity matrix — know what’s real vs a stand-in:

Things that only exist remotely (Vectorize internals, Access, AI Gateway analytics, real Images transforms) get a documented stand-in locally and a --remote escape hatch (wrangler dev --remote) for the rare time you must test against the genuine article.

6. Configuration management

Four layers, each with a clear owner — never overlap them:

For the BO these three layers are enough now — Statsig is deferred (§8). The BO has ~tens of users, so flags/experiments/segment-targeting buy little; small runtime knobs live in KV and anything safety-shaped lives in the Control DO. Statsig becomes a config layer when the consumer frontend exists, where segment targeting and experiment-grade analytics actually pay off.

The sharp line for the BO, because these can blur: Control DO owns governance/safety state (autonomy modes, kill switch, publish gates) — strongly consistent, audited, part of the safety model. KV holds the rest of the BO’s read-fast operational config. Never put a safety control or a secret in KV.

Typed, validated loader (BO-4). A shared packages/config exposes a zod-validated Env parsed once at worker start; missing/invalid config fails the boot, not the first request. vars are non-inheritable per environment in Wrangler — declare them under each [env.*] explicitly; the loader makes “you forgot one in prod” a startup error.

wrangler.jsonc is our IaC-lite. Environments (local, staging, production) are first-class; each names its own D1/R2/KV/Vectorize/Queue resources (separate IDs, suffixed names) so blast radius is contained. Bindings + vars are declared per env; none are inherited (a Cloudflare footgun we make explicit).

7. Secret management (Cloudflare)

Decision (BO-5): Cloudflare Secrets Store is the system of record for secrets; per-worker wrangler secret only for worker-private values; .dev.vars.<env> for local (gitignored, already enforced by CI).

• Secrets Store (account-level, audited, RBAC’d; 100 secrets/account) holds shared secrets — model provider keys, Statsig server secret, OAuth client secret, signing keys. Bound into a worker declaratively:

  // wrangler.jsonc
  "secrets_store_secrets": [
    { "binding": "JINA_API_KEY", "store_id": "namaste-ji", "secret_name": "jina-api-key" }
  ]

• Local: .dev.vars.local / .dev.vars.staging (dotenv). Use the secrets.required declaration so a missing local secret warns loudly. The Google OAuth client secret already in secrets/ migrates into Secrets Store for staging/prod.

• Sourcing & rotation: one upstream source of truth (GCP Secret Manager or 1Password) → synced into Secrets Store via a CI step, so rotation is a single update. Never echo secrets to logs; the structured-log schema (CP §13) must have no secret-bearing fields.

• Guardrail: CI already fails if .env*/.dev.vars/secrets/ are tracked; extend the secret-scan to the BO packages. (The guard allows *.example templates.)

Status (live). Cloudflare allows one Secrets Store per account, so Alchemy adopts the auto-created default_secrets_store rather than making a new one. It holds langfuse-secret-key, langfuse-public-key, and google-oauth-client-secret; values are sourced from the environment (CI → GitHub secrets, local → infra/.env, Google falls back to the on-disk file). console-api reads the Langfuse keys via secrets_store_secrets (async .get()) — no per-worker secret injection. Deploying a Worker with such a binding needs an API token with Secrets Store scope.

8. Statsig — frontend-plane experimentation (deferred for the BO)

Decision (BO-6): Statsig is the flag + experiment + dynamic-config + product-analytics layer for the consumer frontend plane — not the BO, for now. The BO is internal, ~tens of users; flags/experiments/segment-targeting earn little there, so it uses KV + Control DO + vars (§6) instead. Statsig comes in with the frontend/consumer projects, where segment targeting and experiment-grade analytics at scale pay off. The design below is the direction for that plane, captured so it’s ready when the surface exists.

When adopted (frontend plane), with the Cloudflare KV edge adapter: the server SDK reads config specs from KV (Statsig syncs them there) for sub-50ms local evaluation at the edge — no per-check round-trip. (Statsig × CF KV)

What it will do for the consumer product:

• Feature gates — dark-launch consumer features; gradual rollout by environment and segment.
• Experiments / A/B — ranking/feed experiments (DISTRIBUTION), onboarding, surface variants. (Creative-plane strategy A/B at the brief level can stay in the brief’s own versioning, CP §5 — it doesn’t need Statsig.)
• Dynamic Config — remotely-managed, segment-targetable JSON config objects changed without a deploy: feed/ranking parameters, festival/theme switches, vernacular copy variants, per-locale rollout values, which model/prompt a given surface uses. Targeting (by locale, language, app version, cohort) is the reason to reach for it over vars.
• Product analytics + session replay — consumer usage; feeds the Creative Head’s share-intent loop (CP §2) — but mind the PII boundary (§10): consumer analytics is the point where DPDP/GDPR and region-pinning bite.

If/when Statsig Dynamic Config meets the Control DO — the boundary that keeps it safe (BO-11). Even on the frontend plane, anything that reaches back into pipeline governance must respect this split. Both are “runtime config you change without deploying”:

Rule: never put a safety control or a secret in Dynamic Config; never put a segment-targeted product knob in the Control DO. When unsure, ask “if this were stale or Statsig were down, could it ship bad/unsafe content?” — if yes, it’s Control DO.

Flags × RBAC compose (don’t conflate them) — a principle that holds whichever plane gates a feature. A feature renders iff flagEnabled(feature) && permits(user.role, action). A flag answers “is this feature on / which variant / what config”; RBAC answers “is this user allowed.” A flag or Dynamic Config value is never an authz control. (In the BO today, “flag” is just a vars/KV toggle, not Statsig.)

Read path & availability. Dynamic Config (like gates) evaluates against the KV-synced specs at the edge, so reads are fast and survive a Statsig outage on the last-good snapshot. Anything that must be authoritative (not just fast and usually- right) does not belong here — it belongs in Control DO.

Local (when adopted on the frontend plane): Statsig offline/bootstrap mode (a committed dev spec file, including Dynamic Config defaults) so local dev needs no network and no Statsig account; staging/prod use real projects (separate keys per env via Secrets Store). The BO local loop has no Statsig dependency at all.

9. RBAC — defined in full on day one

Per ARCHITECTURE.md §5: coarse at the perimeter (Cloudflare Access), fine in the app (D1). Two enforcement points, one identity core.

flowchart LR
  U[User] --> CA["Cloudflare Access<br/>SSO · MFA · group policy"]
  CA -->|signed JWT identity| MW["Console API middleware"]
  MW --> DIR["users · roles · role_permissions (D1)"]
  MW --> DEC{permits action on resource?}
  DEC -->|yes| H[handler] --> AUD[(audit log)]
  DEC -->|no| R[403] --> AUD

The model (complete now, populated narrow):

• Identities: human users (via Access) and service accounts for agents. Agents act under scoped service identities — an agent gets only the capabilities its manifest declares (CP §12 agent registry), so an over-eager agent can’t exceed its remit. This matters for an agent-operated product.

• Resources × actions (the permission vocabulary), e.g.: brief:{read,create,approve}, content:{read,publish,unpublish,delete}, agent:{view,run,schedule,configure}, control:{set_mode,kill_switch}, validation:{submit}, user:{invite,assign_role}, audit:{read}, config:{edit}.

• Roles → permissions (seed matrix; rows are roles, ✓ = granted):

  Permission	admin	creative-lead	engineer	reviewer/validator	viewer
  audit:read	✓	✓	✓	–	✓
  agent:view	✓	✓	✓	–	✓
  agent:run / schedule	✓	✓	–	–	–
  brief:create / approve	✓	✓	–	–	–
  content:publish / unpublish	✓	✓	–	–	–
  validation:submit	✓	✓	–	✓	–
  control:set_mode	✓	✓¹	–	–	–
  control:kill_switch	✓	–	–	–	–
  user:invite / assign_role	✓	–	–	–	–
  config:edit (secrets/flags)	✓	–	–	–	–

  ¹ per-stage mode only; global mode + kill switch stay admin-only.

• Schema (D1): users(id, access_subject, email, status), roles(id, name), role_permissions(role_id, permission), user_roles(user_id, role_id), service_accounts(id, agent_id, scopes). Permission is a typed enum in packages/rbac, shared by middleware and UI (so the UI hides what the API would forbid — defence in depth, not the only line).

• Enforcement: one middleware resolves Access identity → D1 user → effective permissions, attaches them to the request, and every mutating handler asserts a permission; every decision (allow/deny) writes to the append-only audit log (CP §13). Deny-by-default.

• MVP: only admin is populated (the founder) — CP §12/§19. The matrix, schema, enum, and middleware all ship now; adding creative-lead/validator/… later is inserting rows, not changing code.

• Future: stay role-based; move to attribute/policy-based (e.g. per-locale or per-deity scoping for reviewers) only if a real need appears (ARCH §5).

10. PII boundary (carried from ARCH §6 / CP §14)

The BO/content/audit plane is broadly geo-portable and low-PII. Statsig product analytics on consumers is where PII enters — keep the consumer-PII plane separate (region-pinned D1/DO/R2, consent ledger, DPDP+GDPR bar). BO user data (a few staff accounts) is low-risk but still lives behind Access + audited.

11. Environments & promotion (staging now, prod later)

Decision (BO-7): design two environments together; provision staging now, prod when the surface is real. Three Wrangler environments — local, staging, production — each with its own isolated CF resources (D1/R2/KV/Vectorize/Queues/DO namespaces, distinct names + IDs), its own Statsig project, its own Secrets Store scoping.

Promotion = the same artifact + migrations applied to the prod env with a human approval gate; no code divergence between envs (only config/resource IDs differ).

Status (live). Staging runs at bo-staging.namasteji.org behind Cloudflare Access — same-origin (Pages UI at /, Worker /api/*), workers_dev disabled. Auto-deploys on merge to main (CI-CD.md); prod (bo.namasteji.org) is manual + gated. Free-plan SSL footgun: Universal SSL covers only the root + first-level subdomains, so BO hosts must stay first-level (bo-staging, bo) — a second-level host (bo.staging.*) can’t get a free edge cert and leaves the domain stuck. Paid ACM/Total TLS would be needed for deeper nesting.

12. Auto-deploy / CI-CD

Decision (BO-8): GitHub Actions orchestrates deploys; provisioning is Infrastructure-as-TypeScript via Alchemy; wrangler deploy --env does the publish. Rationale for the solo-with-Claude-Code constraint:

• Alchemy (pure-ESM TypeScript IaC) provisions D1/R2/KV/Vectorize/Queues/secrets as code — reproducible across staging/prod, diffable, and Claude-Code-native (it reads and writes plain TS, no separate HCL/Terraform toolchain). (repo)
• GitHub Actions runs the gates that push-to-deploy alone can’t: typecheck/lint/test, D1 migrations, the secrets sync (§7), then wrangler deploy. You already have CI + CODEOWNERS + protected main by convention — this slots in.
• Flow: PR → CI (build, test, migration dry-run, preview deploy) → merge to main → auto-deploy to staging → manual/tag + GitHub Environment approval → deploy to production (migrations first, then workers, then cutover). Instant rollback = redeploy previous version (Workers keep version history) + control:kill_switch for behaviour (ARCH §3).
• Minimal alternative considered: Workers Builds (CF-native push-to-deploy) — less setup, but weaker control over migration ordering and multi-resource provisioning, so it’s the fallback, not the default. (Workers Builds)

13. Repo layout (when scaffolded — not yet)

Per CLAUDE.md, don’t scaffold ahead of need. When BO work starts, the shape:

namaste-ji/
├── apps/console/            # BO web UI (auth via Access, RBAC-aware)
├── services/
│   ├── console-api/         # thin control API + Control DO
│   ├── creative-*/          # creative-plane workers/agents (CP)
│   └── …
├── packages/
│   ├── config/              # zod-validated Env loader (§6)
│   ├── rbac/                # permission enum + matrix + middleware (§9)
│   ├── model/               # ModelPort / EmbedderPort / VectorIndex (§4)
│   └── contracts/           # brief schema, log schema, shared types
│                            # (packages/flags = Statsig wrapper — frontend plane, later §8)
├── infra/                   # Alchemy IaC (§12)
└── docker/                  # compose.yaml + local model setup (§5)

Each service keeps its own deps (monorepo convention, AGENTS.md).

14. Observability

No new mechanism — reuse ARCH §4 (Workflows dashboard, AI Gateway dashboard, agents/observability event stream, Mission Control canvas) and CP §13 (structured, schema-stable, correlation-ID logs as a dataset for agents → Workers Logs + Logpush + Analytics Engine). The BO is the renderer of these, not a second logging system.

15. Decisions log (extends ARCH D1–D8, CP CP-1–CP-16)

16. Open questions

• ✅ Console UI stack — resolved: a Vite SPA (apps/console) + the Hono control-API worker (services/console-api), served same-origin (UI at /, API at /api/*).
• Canonical embedding dim pinned to 1024 (provisional); the launch managed embedder (Jina vs Cohere vs Voyage vs Workers AI) still open — decided by a small quality+cost bake-off on devotional imagery.
• Qdrant vs sqlite-vec for the local VectorIndex impl (fidelity vs simplicity).
• Secret upstream — GCP Secret Manager (we have manaste-ji) vs 1Password as the source that syncs into Secrets Store. (Secrets Store itself is live — see §7.)
• ✅ Alchemy vs Workers Builds — resolved: Alchemy (infra/). Workers Builds was disconnected (it auto-created cruft and violates CI-is-only-deployer).
• Statsig data residency for consumer analytics under DPDP (§10).

17. Suggested build order (BO platform)

1. packages/config (typed Env) + wrangler.jsonc with local/staging envs.
2. packages/model — ModelPort / EmbedderPort / VectorIndex with local impls (Ollama, Infinity, Qdrant) — unblocks the catalog spine (ARCH §9 step 1).
3. docker/compose.yaml — one-command local bring-up (§5).
4. packages/rbac — permission enum + matrix + middleware + D1 schema; seed admin.
5. Secrets Store wiring + .dev.vars.* + CI secret sync (§7).
6. CI/CD — GitHub Actions (test → migrate → deploy staging) + Alchemy infra (§12).
7. Then the BO product surfaces per CREATIVE-PLANE §12/§18 land on this substrate.
8. (Frontend plane, later) Statsig wrapper + KV adapter + offline bootstrap (§8) — when the consumer surface exists, not part of the BO build.