▍ Amey Parmarthi
← All work
02live

Churn Decision Intelligence

production ML · decision policy · ROI

Churn prediction on KKBox's ~33 GB subscription dataset, turned into a decision: who to contact, through which channel, and what it nets. The naive "call the top 10K" campaign loses $6.8K per cycle; the ROI-optimized policy nets +$17.7K and a tiered policy +$24.3K. Served from FastAPI on Cloud Run with per-score SHAP, behind a live decision app, with evals that gate every deploy.

Live app Source README
0.966
ROC-AUC, time-based holdout
0.539
PR-AUC — 43× the 1.24% base rate
+$17.7K
ROI threshold: 1,478 contacts, 1,044 churners caught
+$24.3K
tiered policy v2, per cycle (simulated)
52.7ms
p95 /predict latency, local (gate ≤ 500ms)
Churn Decision Intelligence overview: the 0.76% of subscribers who drive 43% of churn, with KPI tiles
Churn Decision Intelligence overview: the 0.76% of subscribers who drive 43% of churn, with KPI tiles
Model evidence page: time-based validation (ROC-AUC 0.966, PR-AUC 0.539) and the model-family benchmark
Model evidence page: time-based validation (ROC-AUC 0.966, PR-AUC 0.539) and the model-family benchmark
Policy simulator: choose a threshold and see contacts, true positives and net value
Policy simulator: choose a threshold and see contacts, true positives and net value

The problem

A subscription business loses 1.24% of subscribers per cycle and has a fixed retention budget. A model score alone doesn't help the retention team; they need to know who to contact, at what cost, and whether the campaign makes money. The obvious campaign — contact the top 10,000 — loses money, because below the top ~1,500 precision is too low to cover a $5 contact.

What I built

  • Data pipeline: 32.7 GB of raw CSV (392M listening-log rows) to Parquet to a 992,931 × 38 model table, all in DuckDB on one machine.
  • Twelve experiments across baselines, gradient boosting, deep tabular models and AutoML, tracked in MLflow. Champion: FLAML-tuned LightGBM, validated on a time-based holdout.
  • Decision policy as a versioned artifact the API serves: contact_now (≥ 0.68, $5), nudge (0.2058–0.68, $0.50 email or in-app), monitor. Changing a cutoff is a reviewed artifact change, not a code change.
  • FastAPI /predict returns the score, the policy action and TreeSHAP attributions per customer; batch scoring reports PSI drift; inputs with more than half the features missing get a warning.
  • Eval harness (26 checks: economics, lift, score parity, input contract, p95 latency) gates CI against the in-process API and runs again against Cloud Run after every deploy.
  • Keyless deploys to Cloud Run through GitHub Actions workload identity federation; 41 backend and 62 frontend tests.
  • React decision app for the retention team: executive overview, policy simulator, model evidence, live scoring, MLOps and experiment plan.

Key decisions

D4↗

ROI threshold as the default, not top-K

Contacting the top 10K loses $6,776 per cycle at the documented assumptions. The ROI-max threshold contacts 1,478 at 70.6% precision for +$17,666. Top-K stays available for teams with a fixed call list.

D5↗

Tiered policy as a versioned artifact the API serves

The v1 policy contradicted the default and the API never read it. v2 defines three tiers, loads at startup, returns an action with every score, and is recomputed by tests and evals.

D13↗

Evals gate CI and every deploy

Docs, UI and artifacts once disagreed on headline figures. Now the harness recomputes the business numbers from committed artifacts, so they can't drift without failing CI.

D7↗

Scores are rankings, not probabilities

Trained at 6.4% prevalence, evaluated at 1.24%: the nudge tier churned at 8.9%, well below its scores. All economics use observed precision per tier, never the raw score; calibration comes before any per-subscriber expected-value logic.

D8↗

Warn on missing data, don't reject

An empty record scores 0.99. Rejecting it would break batch jobs; silently scoring it would trigger a $5 contact on garbage. The API scores and warns.

Honest limits

  • Economics are simulated from stated assumptions ($120 per save, $5 contact, 20% save rate). The nudge tier's 12% save rate is a placeholder until a pilot measures it.
  • Scores rank customers; they aren't calibrated probabilities, and the API says so.
  • The ROI threshold trades recall (75% → 43%) for profit; the remaining churners are reached only through the cheap channel.
  • The post-deploy eval runs after traffic shifts. A staging revision gated by the same eval is the next step.

Stack

DuckDBLightGBMFLAMLMLflowSHAPFastAPIReactCloud RunGitHub Actions

Read the source