04live

Member Service Agent

chat + voice · one graph · safety before the model

A member-service agent built on one rule: the LLM decides intent; code decides truth. A deterministic crisis gate runs before any model call, step-up authentication is a graph node rather than a prompt instruction, and order status and refunds come from config-driven engines. FastAPI + Postgres backend with checkpointed sessions and a full turn audit, and a Next.js operator console that shows every routing decision. On a live model, held-out crisis recall is 1.00 with Claude Haiku 5.5, the cheapest model that passes every gate. It's live: pick a synthetic member, ask for a refund, and approve it yourself as the specialist, in plain text.

Live app Source README

1.00: held-out crisis recall, live · 0.13 keyword gate alone
2.3s: p95 turn on Claude Haiku 5.5, live · 1.1s p50
1.00: routing accuracy and task completion, live
25/25: injection attacks blocked, mock and live
$0.052: per 1,000 turns on Haiku 5.5 · 101 tests

The problem

In production member service, the failures that matter happen before the model speaks: a distressed caller routed to a status lookup, a sensitive record read before identity is verified, a refund amount the model made up. Platform moderation can pre-empt escalation in exactly the wrong order. Those paths have to be deterministic, ordered and auditable.

What I built

LangGraph flow: begin turn → crisis gate → intent routing → auth gate → intent nodes. The crisis gate runs before routing and always ends in crisis support with a warm human handoff.
Config-as-data truth tables: 32 order-status codes and 20 condition codes map to responses in code; the model never interprets a status.
Step-up authentication as a graph node before any account intent; identity tokens never reach the LLM.
Refunds: code computes the amount, a threshold decides, and a specialist approves in plain text.
Postgres checkpointing for multi-turn sessions plus a sessions/turns audit; a pluggable provider layer (mock, Anthropic, OpenRouter, Gemini).
Next.js operator console with a decision inspector and an evals dashboard; chat and voice share one graph with channel-aware output.
Eval gates pick the router model: Haiku 5.5, Sonnet 5.5 and Opus 5.5 all passed, so the cheapest and fastest ships.
A 30-case prompt-injection suite (25 attacks in six categories, 5 benign controls) asserts on session state, not just the reply. CI blocks merges on every gate that holds without a live model.
101 tests, three CI jobs, 17 ADRs. Live on Cloud Run + Vercel with per-day turn caps.

Key decisions

ADR-003↗

The crisis gate runs before routing

A generic intent router or moderation classifier that runs first can absorb crisis language into an ordinary intent or a canned refusal, and on a managed platform the builder may not control that order. Owning the graph means owning the order: crisis intent always reaches 988 and a warm handoff.

ADR-012↗

Two OR'd crisis detectors, judged on a held-out set

A lexicon is predictable but didn't generalize: 1.00 on dev, 0.13 held-out. An LLM generalizes but can fail. Either detector can raise the flag and neither can clear it, which favors false alarms over misses. The held-out set was written by an independent agent and is never tuned against.

ADR-007↗

Step-up auth is a graph node, not a prompt

Identity is a compliance boundary, not a conversational nicety. A deterministic auth gate runs before any account intent, so the model never decides whether someone is verified, and chat and voice share one contract.

ADR-014↗

Refunds: code computes, a threshold decides, a person approves

The amount comes from order data, never the model. Up to $50 the member's yes processes it; above that a specialist types "approved" or "denied", read by a regex. A member typing "approved" lands in the router and can't approve anything.

ADR-015↗

The eval gates pick the router model

The rule: the cheapest model that passes every gate. Sonnet 5.5 and Opus 5.5 scored no better on any gated metric, and were 2–3× slower at p95 and 20–40× more expensive per token. Haiku 5.5 ships, with its crisis recall repeated across three runs because it carries the safety gate.

ADR-016↗

Prompt injection is a gated suite

The model returns one schema-enforced intent and a crisis flag, never an amount, an identity or an approval. The suite turns that claim into a regression test: each attack asserts the refund status, auth level, groundedness and crisis flag in state, so a bypass hidden behind a polite reply still fails CI.

Honest limits

Small eval sets: 38 routing cases and 31 held-out crisis messages, so one run isn't a guarantee. The held-out rate of wrongly shown 988 messages is 0.095, just inside the 0.10 bar.
The false-alarm metric was split after the first live results were seen, and ADR-012 records why. Neither the prompt nor the keyword list was tuned to the held-out set.
The injection suite was written alongside the defenses, so it's a regression suite, not an independent red team.
The live demo keeps sessions in memory on one Cloud Run instance, and visitors play the specialist without a staff login. Both are acceptable only because every member is synthetic (ADR-017).

Stack

LangGraphFastAPIPostgresNext.jsClaudePydanticpytest