Docs / running-locally.md · mirrored from the repo

Running everything locally

A start-to-finish runbook for booting the whole back-office mesh and the local LLM, then triggering an agent run. Two LLM paths are supported:

• Fast path (recommended for dev): native llama.cpp on the Mac host (Metal GPU, direct). The agent calls it directly via LOCAL_LLM_URL — no gateway, no tunnel. Fast, simple.
• Parity path: llama.cpp → Cloudflare Tunnel → AI Gateway (CPU, inside Docker). Mirrors staging/prod routing. See LOCAL-LLM.md. Use it when you need gateway parity.

This doc covers the fast path.

0. Prerequisites (one time)

# Docker Desktop (running)
# llama.cpp with Metal (Apple Silicon):
brew install llama.cpp

The default model is Qwen2.5-7B-Instruct (Q4_K_M) — it has a native tool-call template (llama.cpp returns structured tool_calls), which the orchestrator/subagent loop requires. It auto-downloads from Hugging Face on first -hf use and caches under ~/.cache/huggingface/hub/.

1. Start the back-office services

docker compose up -d          # bo-api (8787), bo-web (3000), events-calendar (8788), creative-head (8789)

The agent container reaches the Mac host via host.docker.internal (compose sets extra_hosts: host.docker.internal:host-gateway).

First run (or after wiping .wrangler-state), seed the config KV:

./scripts/seed-local-config.sh

1b. Consumer app (web) — Metro on the HOST, API in compose

The consumer app (apps/consumer, universal Expo) runs its dev server on the host — never in compose (file-watching over the bind mount is slow, and nothing needs it in-network):

cp apps/consumer/.env.example apps/consumer/.env.development  # once
npm --workspace=consumer-api run seed:test-user               # once: test / Namaste@123
npm --workspace=consumer-app run dev                          # Metro web on :8081

Log in at http://localhost:8081 as test / Namaste@123 (or create users from the BO console → Users → App Users). The app talks straight to app-api on :8793 (same-site cookies; the Worker’s local-only CORS allows :8081).

2. Start the LLM server (native llama.cpp, Metal)

The agent’s config (agent:events-calendar) uses provider: "local", and .dev.vars points LOCAL_LLM_URL=http://host.docker.internal:8080/v1. So run llama.cpp on the host at port 8080, bound to 0.0.0.0 so the container can reach it:

MODEL=$(find ~/.cache/huggingface -iname 'qwen2.5-7b-instruct-q4_k_m*00001-of-00002.gguf' | head -1)

nohup llama-server -m "$MODEL" -a qwen2.5-7b-instruct \
  --host 0.0.0.0 --port 8080 --jinja \
  -ngl 99 -fa on -c 8192 -b 2048 -ub 768 -np 1 -t 8 --mlock --no-webui \
  > /tmp/llama-server.log 2>&1 &
disown

If you don’t have the GGUF yet, let llama.cpp fetch it (first run downloads ~4.7 GB):

nohup llama-server -hf Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M -a qwen2.5-7b-instruct \
  --host 0.0.0.0 --port 8080 --jinja -ngl 99 -fa on -c 8192 -np 1 --mlock \
  > /tmp/llama-server.log 2>&1 &
disown

Why these flags (tuned for fastest response on Apple Silicon):

The built-in prompt cache is on by default, so the identical system-prompt+tools prefix is reused every turn — real per-turn latency is far below a cold prefill.

Speculative decoding (a 0.5B draft) was tested and gave no speedup on base M-series (the draft competes for the same GPU) — don’t bother.

Verify it’s up and the agent can reach it:

curl -s http://localhost:8080/v1/models -o /dev/null -w '%{http_code}\n'   # 200 when loaded (503 = still loading)

3. Trigger a run

• UI: open http://localhost:3000 → Agent Hub → events-calendar → Trigger Run.
• API (mock identity locally):

EMAIL=dash.ayushman.99@gmail.com
curl -s -X POST http://localhost:8787/api/agents/events-calendar/runs \
  -H "x-mock-user-email: $EMAIL" -H "Content-Type: application/json" -d '{}'

Watch it in the UI run timeline (List or Timeline/waterfall view), or poll GET /api/agents/events-calendar/runs/<run_id>.

4. Tuning the agent (no redeploy — KV config)

Edit agent:events-calendar in the BO Settings page, or via API:

curl -s -X PUT http://localhost:8787/api/admin/config/agent:events-calendar \
  -H "x-mock-user-email: $EMAIL" -H "Content-Type: application/json" \
  -d '{"provider":"local","model_id":"qwen2.5-7b-instruct","max_steps":16,"confidence_min":0.6,"calendar_months_ahead":12,"research_depth":"standard"}'

TAVILY_API_KEY (in .dev.vars) powers the research subagent’s web search. A TLS-inspecting VPN can block Tavily — turn it off if research calls error.

Troubleshooting