03live

Tenant-Safe Support Agent

multi-tenant agent · human-in-the-loop · eval-driven architecture

A customer-support agent with six tools, human approval on every refund, and tenant isolation enforced at the server before any conversation state loads. I pre-registered pass bars, then benchmarked a flat agent against a supervisor: both resolved 30/30 with zero security failures, flat was 1.7× faster with fewer tool calls — so flat shipped. It's live: sign in as one of two customers and try to read the other's account. 163 deterministic tests in CI, 8 code evaluators plus an LLM judge, 25 ADRs.

Live app Source README

30/30: resolved, both architectures
0: security failures across the suite
3.8s: p50 full turn, flat · 6.6s supervisor
147/150: resolved, flat repeated 5× at the current agent
163: deterministic tests in CI · 25 ADRs

Live app, signed in as Helena: a prompt-injection request for another customer's invoices is refused with no tools called

Flat vs supervisor scoreboard: both 30/30 resolved; flat 3.8s p50 and 45 tool calls vs supervisor 6.6s and 61

The problem

A support agent that can read invoices and issue refunds is a data-leak and money-movement risk wearing a chat UI. It has to answer only for the signed-in customer, never move money without a person approving, and earn any added architectural complexity with measured results rather than intuition.

What I built

Six fixed, parameterized tools over the Chinook store database — no text-to-SQL, and no tool takes a customer ID, so the model can't choose whose data to read.
Human-in-the-loop on refunds: approve, edit or reject, with the approval card read from the database rather than from the model's description of it.
Tenant isolation in layers: scoped data access as the boundary, threads bound to their tenant, a result guard that asserts on the data layer's audit log, and native Agent Server authorization that derives identity from the credential.
PII middleware redacts card numbers; tool-call limits stop runaway loops.
Eval suite in LangSmith: 30-example dataset, 8 code evaluators (3 release-blocking) and a calibrated LLM tone judge, plus a grader that catches actions the agent only claimed to take.
Pre-registered flat-vs-supervisor experiment, frozen at one commit, dataset and evaluator version so the comparison is reproducible.
Public front door: FastAPI on Cloud Run behind a Next.js client on Vercel. Identity comes from a signed session token, never the request body; the visitor plays the refund reviewer; spend is capped per session and per day.

Key decisions

ADR-023↗

A real cross-tenant read, closed at the auth boundary

With one customer's valid token and their own thread, a caller could pass another customer's ID in run context and read that customer's spend. Every layer below was correct — scoped to the identity it was handed. Identity now comes only from the authenticated principal; a mismatch gets a 403.

ADR-025↗

A public URL needs a front door, not Studio

Studio takes identity from the selected assistant, which is fine on a laptop and unacceptable on the internet. The public API mints an HMAC-signed token for one of two personas and derives every call's identity from it, so nothing a visitor sends can name a tenant.

ADR-002↗

Flat agent is the baseline; the supervisor must earn its place

Pass bars written before the challenger existed: visible quality gain, no per-workflow regression, at most 2.0s latency penalty. The supervisor tied on quality and failed on latency. Ties go to the simpler topology.

ADR-003↗

Fixed parameterized tools, not text-to-SQL

If the model writes the WHERE clause, any successful prompt injection is an exfiltration path. With fixed tools, get_my_invoices has no customer_id parameter — the model isn't forbidden from requesting another customer's data, it's structurally unable to express the request.

ADR-013↗

Threads are tenant-bound, not just queries

Scoped queries don't help if one customer's thread, with their invoices already in the history, is resumed under another customer's context.

Honest limits

The public app uses two demo personas, not real accounts, and keeps conversations in memory, so they reset when the service restarts. Postgres is the scaling path.
The five-times repeat missed 3 of 150. Two were the exact-fact check reading $27.84 when the agent answered Helena in Czech as 27,84 USD; one re-proposed a refund after rejection and stopped at a second approval card, with nothing written.
30 examples is a decision set, not a statistical claim. The first run showed the supervisor ahead by one example; it vanished on a controlled rerun, which is why the bars were pre-registered.

Stack

LangChainLangGraphLangSmithOpenAIFastAPINext.jsSQLitepytest