FinSight — Financial Evidence Engine
hybrid RAG · cited answers · refuses when evidence is thin
Answers analyst questions over 15K earnings-call chunks with inline citations, an evidence panel and a pipeline trace, and refuses when the corpus can't support an answer. Hybrid retrieval + reranking beats dense-only by 27% NDCG@10. It's live: a React app on Vercel streaming from a FastAPI service on Cloud Run, with telemetry in Postgres and a behavioral eval you can run from the browser.
- 0.743
- NDCG@10 hybrid + rerank, vs 0.584 dense-only (n=40)
- 0.806
- RAGAS faithfulness
- $0.0139
- mean cost per query, measured live
- 3.6s
- time to first token, streaming (live)
- 340ms
- hybrid retrieval · 146ms rerank
![FinSight Ask page: a grounded answer about Apple's 2020 iPhone supply chain with inline [N] citations and a ranked evidence panel](/_next/image?url=%2Fwork%2Ffinsight-ask.png&w=3840&q=75)
The problem
An analyst asking what a company said about margins or guidance needs the quote, the call and the date — not a fluent paragraph. A generic chatbot answers confidently from training data, can't show its sources, and won't tell you when the transcripts simply don't cover the question. The bar here: every claim cited, and an honest refusal when evidence is thin.
What I built
- Corpus of 15,023 earnings-call chunks across 76 tickers, 2019–2023, indexed in Qdrant with both BM25 sparse and voyage-finance-2 dense vectors, fused with reciprocal rank fusion.
- Six-node LangGraph pipeline: Haiku query understanding → hybrid retrieval → Cohere rerank-v3.5 (local ms-marco cross-encoder fallback) with a recency boost → conflict check on comparison questions → Sonnet generation with inline [N] citations.
- Refuses out-of-scope questions — pre-2019, live prices, buy/sell advice — and flags stale evidence instead of guessing.
- React + TypeScript frontend: streaming answers, evidence panel ordered to match citation indices, pipeline trace, and an Evaluation page that runs a 9-case behavioral suite against the live API.
- Telemetry and feedback flow through FastAPI into Neon Postgres with CHECK constraints and a per-IP rate limit — no database credential in the browser.
- Demo-mode fallback: if the API health check fails, the app replays captured responses, clearly labeled, so the product stays inspectable during an outage.
- CI runs ruff, backend tests and the frontend typecheck, test suite and build on every push; 39 backend and 27 frontend tests.
Key decisions
Browser calls the Cloud Run API directly, not through a proxy
A same-origin proxy would remove CORS but hide every client IP behind Vercel's edge and break the per-IP rate limit. CORS is locked to the production origin instead. Verifying it, a VPN's DNS resolved the API to a private address and Chrome blocked it — the outage fallback caught a network failure nobody planned for.
Demo mode when the API is unreachable — labeled, never silent
Silently serving canned answers would misrepresent a stored response as a live system, the exact failure the project exists to avoid. Demo mode is badged, offers a retry, and is excluded from telemetry.
Telemetry through FastAPI into Postgres
The browser wrote telemetry straight to a BaaS with a public key, and feedback lived on the container's local disk, lost on every deploy. Routing both through FastAPI into Postgres bounded the volume with a rate limit, kept credentials out of the bundle and stopped the data loss.
Inline [N] citations instead of forced tool-use
Forced tool-use gave clean structured citations but buffered the whole answer: 11.6s to first token versus 1.16s for plain streaming. Inline [N] markers, parsed after the fact, kept both the structure and the streaming.
Always-on Cloud Run, capped at two instances
On the free tier, a hiring manager clicking the link cold waited ~51s for the UI alone, with no logs to diagnose restarts. One warm instance per service removes the cold start; a max of two caps the spend.
Honest limits
- Latency misses its own target: non-streaming /query median is 10.4s, almost all of it generation (7.5s). Streaming is what makes it usable.
- Most of the NDCG lift comes from hybrid retrieval alone; reranking adds less on a 40-query set. The set is a decision aid, not a statistical claim.
- RAGAS answer relevancy is low (0.20) — answers are faithful but often broader than the question.
- A fine-tuned embedding improved Recall@5 from 0.825 to 0.870 on 48 pairs but isn't deployed; the gain didn't justify a second index.