The thesis from day one was that the model is maybe 20% of an ML system. Six weekends later, I can put a number on it. The logistic regression took one afternoon. Everything that made it worth anything took the other five weekends: data versioning, leakage-safe evaluation, a serving contract shared with training, secretless deployment, monitoring that catches a real shift, a retrain loop with a human in it, and a dashboard that reads the same log the monitor reads. Every interesting number in the repo came out of those five weekends.
This is the last of six posts. The repo is public. The live dashboard is up. It scales to zero, so the first load waits out a 32.7-second cold start. That's the price of $0 idle, and it's a fair trade.
One Container, One URL
I didn't build a second app. The React + Recharts dashboard is built before the image and served by the same FastAPI container at /dashboard, with a small metrics router at /api behind it. Adding a second Container App would have meant a second deploy pipeline, CORS, and another managed identity for nothing the first container wasn't already doing.
The metrics router reads the same blob-stored JSONL the drift monitor reads, through the same code, plus three feeds the workflows now publish next to it: drift verdicts, champion-vs-challenger results, and deployment records. It's cached for a minute, because the log is a few thousand records and the dashboard polls.
What it shows: tiles for the champion model, the latest drift verdict (status color with a label, never color alone), the champion's ROC-AUC on the replayed regime, prediction volume, and the latest deployment. Below that: predictions per hour stacked by regime; the failure-probability distribution against the operating threshold; the share of drifted input columns per regime each time the detector ran, against the 30% line; and the champion's ROC-AUC on labeled traffic rising from 0.50 to 0.99 across a vertical "v2 promoted" line. Tables sit behind every chart so no value is reachable only by hover.
A few design decisions worth naming, because charts are a place where "make it look good" hides mistakes. I did form before color. Two categorical slots, blue for FD001 and orange for the replayed FD002 regime, validated with a script for color-vision-deficiency separation in both light and dark (worst-case delta E 24.7 and 26.8, well above the 8 target) rather than eyeballed. Thin 2px lines, 8px markers with a surface ring, bars capped at 24px. Straight segments between detector runs, because each point is one discrete run and nothing was measured in between. The first draft used smooth curves, which implied a trend between two points that doesn't exist. I rendered it headless in Chrome, light and dark, and looked at it before shipping. Those renders are in the evidence folder.
Every Number, Measured
The README's rule was: never check a box unless the feature worked end to end, and never print a number that wasn't measured. Here's what that produced.
Data. FD001, 100 engines, 20,631 cycles. N = 30 chosen from measured sensor separation (2 to 4 SD inside the last 30 cycles). 7 of 21 sensors dropped by a distinct-values rule. Window 20. Split 80/20 by engine unit. Serving parity with the training table to 6e-15.
Models. Logistic regression held-out ROC-AUC 0.9923 beat XGBoost defaults (0.9899) and 50 Optuna trials (0.9899). Registered as version 1. One afternoon of work.
Serving. Image builds in 22 s, healthy 2 s after start, p50 6.7 ms / p95 8.8 ms locally. On Azure: 12 ms server-side, p50 93 ms / p95 101 ms from a laptop, cold start 32.7 s, idle $0. The container refuses to answer if the prediction cannot be logged.
CI/CD. Merge to live in 4m 46s. Zero password credentials on the app registration. Four resource-scoped roles. The managed online endpoint demonstrated on the fourth attempt and torn down under always() every time.
Drift. 24 held-out FD002 engines replayed, 889 requests. Champion v1 ROC-AUC 0.5007 on that traffic vs 0.9921 on FD001 traffic. A coin flip. 17 of 17 raw columns and all three operating settings drifted; setting_1 at 10,911 reference SDs. FD001 control traffic: 0 of 17.
The retrain loop. Drift verdict to registered challenger: about 9 minutes. To an approved promotion: about 16. Version 2 (logistic regression on FD001 + FD002) bench 0.9875 vs 0.5463. After promotion, the same 24 engines score 0.9933 through the live endpoint, and the monitor, now comparing per regime against the champion's training engines, reports no drift.
What I didn't measure, labeled as such. The managed endpoint's server-side latency (the demo timed only the CLI round trip). RUL RMSE is absent because the regression stretch didn't ship. I said so in the README rather than leaving the gap unmarked.
Eight Lessons, Each Tied to Something That Actually Happened
1. The model was 20% of the work, and it showed. A logistic regression beat tuned XGBoost, went to a coin flip on a regime it had never seen, and recovered by being retrained on it. The pipeline produced every interesting number. The model produced one afternoon.
2. Enforce data rules in code, not prose. The FD002 quarantine was a guard in the ingest script and a DAG that named its input files. That's why it could be lifted deliberately, with a flag and a warning, on the day it was needed, and never by accident before that day.
3. Prove parity numerically. One feature function shared by training and serving, and a test that the served probability equals the training table's to floating-point noise. That's the single check that makes the drift monitor's inputs trustworthy. Without it you're comparing apples to whatever your serving layer decided to do.
4. A green run is not a deployment. The promotion run reported success while the old model kept serving, because the image tag was the commit SHA and a promotion doesn't change the commit. The fix was a unique tag and a smoke test that asserts what was supposed to change. I kept the screenshot of the run that lied, under an honest filename.
5. Gate the promotion, automate everything else. Drift, retrain, challenge, registration, and the promotion request need no human. The one step that changes production has a reviewer, a comment, and a record in the run. GitHub Environments gave me that for free on a public repo.
6. Calibrate the detector against real in-distribution traffic before believing it. Defaults produced a false alarm on 25 records and a near miss on 781. A minimum-sample gate and a per-column cut set from held-out FD001 engines fixed both. Then compare per regime, or the retrained model gets accused of the composition of its own traffic.
7. Secretless means fewer things to get wrong, not zero. Federated credentials removed every stored cloud secret. The OIDC subject still changed twice, first to ID-qualified names and then to environment names, and both times the failing run's own token details named the fix. Worth it, but don't treat "no passwords" as "no debugging."
8. Cost guardrails are features. Scale-to-zero at $0 idle, a budget alert before the first deploy, and a managed endpoint that tears itself down, tested by three failures before it was tested by success. Month-to-date Azure spend stayed in the low single digits.
On Building in Public
Six posts, each written from a factual brief the moment a phase's checkbox flipped, with an explicit list of what wasn't done yet. An evidence folder with a screenshot per checkpoint, indexed by what each one proves, including the ones that show failures. That discipline is harder than it sounds. The temptation is to write the post after you've cleaned everything up and made it look inevitable. Writing it from the checkpoint brief means the failures are in there, and you have to name what's still missing.
The README and the dashboard are the same rule expressed two ways: don't check a box until it works end to end, and don't print a number you didn't measure.
What Comes Next, If Anything
These are options, and none of them is a promise. A pytest suite around the parity and drift rules, so they break loudly instead of silently. Canary traffic and automated rollback, so promotion could safely go unattended. A replay of FD004, the other quarantined regime and still untouched, to see if the retrained model generalizes or just memorized FD002. And the Phase 0-to-3 portal screenshots I deferred.
If you're building something similar and want the short version: serve the dashboard from the container you already have. Read the same log your monitor reads. Validate chart colors with a script. Give every chart a table twin. Measure everything you claim and label what you didn't measure. Keep the evidence of failures. And build the pipeline first. The model is the small part.