Namaste Ji by Ayushman Dash

Docs / local-llm.md · mirrored from the repo

Local LLM mesh (llama.cpp → Tunnel → AI Gateway)

Just want a fast local dev loop? Run native llama.cpp on the Mac (Metal GPU) and let the agent hit it directly — see RUNNING-LOCALLY.md. This doc is the gateway-parity path (routes through CF AI Gateway, matches staging/prod).

Goal: run a local LLM that the agents reach through Cloudflare AI Gateway, so local routing is identical to staging/prod — same ModelPort, same gateway observability, same config-driven model switching. Only the gateway, custom-provider slug, and model id differ per environment, and all three live in KV (editable in the BO Settings page).

events-calendar agent (Worker)
  └─ OpenAI SDK → ${ai_gateway_base_url}/custom-local-llama/v1
       └─ CF AI Gateway  (gateway: namaste-ji-gateway-local · custom provider: local-llama)
            └─ https://llm-local-development.namasteji.org      (named Cloudflare Tunnel, stable)
                 └─ cloudflared (compose) → http://llm:8080   (llama.cpp, CPU, Gemma 3 4B)

macOS note: Docker can’t use the Mac’s Metal GPU, so the in-compose llama.cpp is CPU-only. We default to a small model (Gemma 3 4B Q4) to keep it usable. Switch models via .env.

One-time setup (account-bound — you do these)

1. Create the named tunnel

Zero Trust → Networks → Tunnels → Create a tunnel (Cloudflared):

  • Name it e.g. namaste-ji-local-llm.
  • Public hostname: llm-local-development.namasteji.org → Service HTTP http://llm:8080 (the compose service name resolves inside the docker network).
  • Copy the connector token → paste into root .env as CF_TUNNEL_TOKEN.

2. Create the gateway + custom provider

Both are one-shot API calls. Get a token with AI Gateway: Edit (the one in infra/.env already has it). Then:

ACCOUNT_ID=b00772422a2e2a143f0b2058b85155bc
TOKEN=<your CF token with AI Gateway:Edit — from infra/.env>

# NOTE: paste the curls WITHOUT these comment lines — interactive zsh tries to glob "(a)".

# (a) Create the local gateway (skip if it already exists). All these fields are required.
curl -X POST "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai-gateway/gateways" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"id":"namaste-ji-gateway-local","cache_ttl":0,"cache_invalidate_on_update":false,"collect_logs":true,"rate_limiting_interval":0,"rate_limiting_limit":0,"rate_limiting_technique":"fixed"}'

# (b) Register llama.cpp (via the tunnel) as a custom provider, slug "local-llama".
# Required: name, slug, base_url. enable defaults to false, so set it true. base_url is the
# ROOT DOMAIN only — no /v1 (that comes from the agent's SDK path).
curl -X POST "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai-gateway/custom-providers" \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"name":"Local Llama (dev)","slug":"local-llama","base_url":"https://llm-local-development.namasteji.org","enable":true}'

The agent then calls …/namaste-ji-gateway-local/custom-local-llama/v1/chat/completions, which the gateway forwards to https://llm-local-development.namasteji.org/v1/chat/completions.

Daily run

cp .env.example .env          # first time: fill CF_TUNNEL_TOKEN
docker compose up             # boots bo-api, bo-web, events-calendar, llm, cloudflared
./scripts/seed-local-config.sh   # first time (or after wiping state): point agent at local gateway
  • First boot downloads the GGUF (cached in the llm-models volume) — watch docker compose logs llm.
  • Direct LLM sanity check: curl localhost:8081/v1/models (8081 is the host port for llama.cpp).
  • Trigger the agent: curl -X POST localhost:8788/run (agent is on 8788; BO API on 8787, UI on 3000).

Switching models / providers — no redeploy

  • Local model: edit LLM_HF_REPO + LLM_ALIAS in .env, docker compose up -d llm, then set model_id in the BO Settings page (agent:events-calendar) to the new alias.
  • Local → cloud: in BO Settings, change agent:events-calendar.provider to anthropic (and model_id), or point global:defaults.ai_gateway_base_url at the staging gateway. The agent picks it up on its next run.

Observability

Every call — local or cloud — shows up in the matching AI Gateway dashboard (namaste-ji-gateway-local locally, namaste-ji-gateway-staging in staging). Prompt traces go to Langfuse with the local label.