Docs / events-and-analytics.md · mirrored from the repo

Namaste Ji — Interaction Events & Analytics

Status: design. Researched + agreed direction (2026-07-18); nothing consumer-side is built yet beyond the Lab pipe. Living doc; decisions carry EV-* IDs.

This doc owns how user interactions (likes, saves, shares, impressions, dwell) are registered, streamed, stored, and consumed — by the feed recommender and by product analytics — under India’s DPDP regime. It sits between DISTRIBUTION.md (what the signals are for), FRONTEND.md (who emits them, FE-6/FE-10), and console/FEED-LAB.md (where the taxonomy was born, FL-13). Legal grounding: DPDP Act 2023 + DPDP Rules 2025 (notified 14 Nov 2025; substantive obligations in force ~14 May 2027).

1. The core principle — one stream, sibling consumers

The recommender does not read signals “through analytics.” There is one first-party, append-only event stream; product analytics and the feed/recsys are sibling consumers of it. Analytics is never in the recsys data path, and no third-party analytics SDK ever sees the firehose. This is the standard shape at recsys-heavy feed companies (Pinterest, ShareChat, TikTok) and the only shape that keeps DPDP consent semantics tractable — purpose separation falls out of the architecture instead of being bolted on. (EV-1)

Three layers with different guarantees:

Safety property: features are always recomputable from the stream; state writes emit events, never the reverse. (EV-2)

2. Interaction state — likes & saves

• D1 (consumer DB): interactions(user_id, content_id, kind, created_at), PK (user_id, content_id, kind) — the natural key makes writes idempotent.
• API is set-state, not toggle (EV-3): PUT /api/interactions/:content_id/:kind / DELETE …. Toggle endpoints + mobile retries = flapping likes. Every mutation carries a client-generated event_id UUID as idempotency key.
• Client (Expo, web + native): optimistic UI + a disk-persisted outbox (SQLite/AsyncStorage) replayed in order on reconnect; queued toggle sequences collapse to final state before sending; backoff to avoid retry storms.
• The state handler also enqueues the corresponding stream event (same request; upgrade to a same-transaction D1 outbox row if dropped events ever matter).
• Counters (like/save counts per item): never a hot row. Queue consumer batch-increments item_stats (a 100-message batch = 1 write); displayed counts derive from the OLTP table (periodic COUNT(*) reconcile), served via short-TTL KV. WAE counts are for trending/ops only (sampled, 92-day). (EV-4)

3. Event taxonomy v2 — extend FeedEvent, don’t replace it

packages/shared/src/feed-events.ts (v1, proven in the Lab) is the durable asset; the app era extends it as schema_version 2:

• New variants: like, unlike, save, unsave (today save/download is folded into open and like doesn’t exist — the recommender and DPDP purpose-tagging both want them explicit). download stays an open sub-kind or gets its own variant at build time.
• Envelope additions (Segment/Snowplow canonical fields):
  • event_id — client UUID, the dedup key (at-least-once transport + idempotent ingest = effectively exactly-once).
  • request_id — the feed-serve token, minted by the feed endpoint per response page and echoed on every impression/engagement from that page. This one field makes serve-time-feature logging joins and training-data construction trivial (the ShareChat/TikTok “online joiner” pattern). Add it from day one. (EV-5)
  • user_pseudo_id — opaque per-user UUID (see §7). Never the phone number, never the Better Auth id.
  • purpose — analytics | personalization | ops (see §7).
  • Timestamps: client occurred_at + sent_at, server received_at; server computes the skew-corrected canonical timestamp (received_at − (sent_at − occurred_at)).
• Zod schemas in @namaste-ji/shared are the schema registry; ingest validates and routes invalid events to a bad-rows side-channel instead of dropping them.
• Dwell mechanics: IntersectionObserver (web) / viewability callbacks (native); accumulate visibility client-side and emit one terminal impression/dwell event with dwell_ms — send the aggregate, not raw scroll telemetry (less collected = the strongest privacy control, and ~10× less volume).

4. Ingest & fan-out

Expo app ── batch (disk queue, flush ~10s/25 events, fetch keepalive, retry+backoff)
   ▼
consumer-api  POST /api/events   (session auth · Zod validate · dedup · received_at ·
   │                              consent check · pseudonymize · bad-rows channel)
   ├─ FIREHOSE  → Pipelines stream → SQL transform → Iceberg (R2 Data Catalog),
   │             partitioned by day            [source of record]
   ├─ OPS       → Workers Analytics Engine writeDataPoint()  [dashboards; sampled, 92d]
   └─ SIGNALS   → Queue (like/save/share_intent only)  → feature updates (§6)

• Pipelines → Iceberg on R2 Data Catalog is Cloudflare’s own “Data Platform” story (exactly-once into sinks; 5 MB/s per stream ≫ our horizon). Both are beta — accepted (EV-6) because Iceberg’s open format is the exit hatch (queryable by R2 SQL, DuckDB, Spark, Snowflake with zero egress) and pricing lands at GA, not retroactively. Fallback if beta bites: plain Parquet on R2 behind the same sink port.
• The existing EventSink port stays the seam: D1EventSink (Lab) · PipelinesEventSink + WAEEventSink (app). Nothing in the Lab is thrown away (FL-13).
• WAE limits to respect: 25 writes/invocation headroom (max 250), 1 index per point, sampling above ~100 events/s/index, hard 92-day retention — a dashboard, not a record.

5. Analytics strategy

• First-party by default. Deep product analytics (cohorts, retention, funnels) run on the Iceberg lake via R2 SQL / DuckDB, surfaced in the BO console; real-time ops dashboards run on WAE’s SQL API. (EV-7)
• Statsig (behind packages/flags, PostHog plan-B — FE-6) receives only experiment exposures + key conversions (share_intent, session_open, push_received). Never the firehose (≈$15K/mo and a needless DPDP surface), never direct identifiers.
• No third-party analytics SDK (no GA/Firebase Analytics). Sentry for crashes only.
• Sessionization, DAU/retention, and share-funnel metrics are computed offline from the lake — the client’s session_id (existing envelope field) is the join key.

6. Features for the feed

• Online path (seconds-fresh, low volume): Queue consumer updates user_profile in D1 — last-N actions, per-deity/language/occasion affinity counters with exponential decay, recency/share-weighted embedding mean (the DS-8 taste vector) — plus a compact KV read-mirror for serve-time access. A Durable-Object-per-user upgrade is available later if within-session responsiveness becomes a ranking lever; not the starting point. (EV-8)
• Impressions/dwell never touch the per-user online path — too voluminous. They flow to Iceberg; scheduled Workflows compute the heavy features offline: embedding centroids, seen-sets/novelty, circulation counters, and impression-based negatives (shown-but-not-engaged — the highest-value ranking signal in the Pinterest playbook, free once impressions carry request_id).
• Training data = serve-time feature log ⋈ impressions ⋈ engagements on request_id.
• This operationalizes DISTRIBUTION DS-8/DS-12/DS-14 unchanged: offline is the slow brain, serving stays rank() from @namaste-ji/feed.

7. Privacy & DPDP architecture

Timeline (DPDP Rules 2025, notified 14 Nov 2025): machinery in force now · Consent Managers + Board penalty powers 14 Nov 2026 · all substantive obligations ~14 May 2027 (consent/notice, security floor, breach, retention/erasure, children, rights). Nothing substantive is enforceable today; we build it into v1 rather than retrofit.

1. Children (the structural risk). Child = under 18; DPDP §9(3) bans behavioural monitoring of children absolutely (parental consent cannot override; ₹200 crore exposure). Consensus reading: a personalized feed for a known minor = prohibited. Stance (EV-9, industry-standard — ShareChat/Moj route): 18+ terms + age declaration at signup, risk-based escalation, and a non-personalized contextual feed (festival-calendar/context-pool ranking — our cold-start floor already is this) as the automatic fallback for any account flagged possibly-minor. Personalization is never built on a known-child account; the age-assurance step itself is expressly exempt processing (Fourth Schedule Part B).
2. Consent (EV-10): standalone, itemized, unbundled — separate items for (i) account/auth, (ii) analytics, (iii) personalization — served in the user’s chosen language (hi/en/hi-Latn now; the Act gives users the option of English or any Eighth Schedule language, so the notice is translatable i18n content, not hardcoded copy). Withdrawal is one tap and actually works: event collection stops, feed degrades to contextual mode. Consent receipts (who/when/what/version) stored — burden of proof is ours. Consent-Manager interop is a post-Nov-2026 integration, not a launch blocker.
3. Pseudonymity (EV-11): events carry user_pseudo_id only; the pseudonym↔identity mapping is a single guarded D1 table. Phone numbers (the consumer identity, E.164 in Better Auth) never enter the event lake, WAE, Statsig, or Langfuse.
4. Purpose tagging: every event stamped analytics | personalization | ops at capture — cheap now, brutal to retrofit, and it’s what makes “withdraw personalization but keep ops telemetry” implementable.
5. Retention tiers (EV-12): raw events 90–180 days in Iceberg (day-partitioned, snapshot expiry on schedule) → pseudonymous aggregates 1–2 years → anonymized aggregates indefinitely. WAE’s forced 92-day expiry is the ops-tier TTL. Wrinkle: Rule 6 requires processing logs kept ≥1 year even as data is erased.
6. Erasure (≤90-day rights SLA): delete OLTP rows + the pseudonym mapping immediately (user-visible effect instant); enqueue the pseudo-id for a periodic Iceberg delete + snapshot-expiry Workflow (physical erasure). No crypto-shredding needed because no direct identifiers enter the lake.
7. Breach readiness: Rule 7 is stricter than GDPR — notify affected users “without delay” (in-app/SMS) with no materiality threshold, plus DPBI intimation immediately and a detailed report within 72h. Runbook + processor flow-down terms (Cloudflare, OpenAI) needed before May 2027.
8. Cross-border: green light today (blacklist model, list empty; no localization for non-SDFs). SDF designation is a when, not if at our ambition — its algorithmic due-diligence duty is pre-covered by keeping versioned FeedPolicy artifacts, Heimdall eval runs, and diversity audits as the evidence trail; keep an India-region (CF jurisdiction) fallback plan for the PII plane.
9. GDPR: irrelevant while India-targeted (accessibility to diaspora ≠ targeting, EDPB 3/2018). Revisit only on deliberate EU marketing.

Existing principles P-8 (stop at the share-sheet), P-28 (region-pinned, minimized, retention-bounded PII plane), and DS-10 (aggregate wherever per-user history isn’t needed) are load-bearing here and unchanged.

8. Build order (PR ladder)

Each is small and independently shippable; 1–3 unblock real signal collection on staging.

9. Decisions log

10. Open questions

• OQ-1: download — own event variant or open sub-kind? Decide at PR 1.
• OQ-2: Pipelines GA pricing — revisit EV-6 economics when announced.
• OQ-3: Consent-Manager interop mechanics (regime starts Nov 2026; DPBI registrations and MeitY guidance pending) — track, integrate post-launch.
• OQ-4: Age-assurance escalation signals (what flags “possibly minor”) — await MeitY guidance; industry has no battle-tested pattern yet.
• OQ-5: When does user_profile move from D1 row to DO-per-user — trigger = within-session responsiveness measurably lifting share-intent in the Lab.