Parts 1 through 4 of DriftWatch were setup: data quarantine, training, serving, and CI/CD. This is the phase the whole thing was built for. The claim I wanted to be able to make was: my monitor caught a real distribution shift, the system retrained, a human approved exactly one step, and a better model served. I can make it now, with numbers. I can also show the two bugs the first run exposed, because the evidence folder has a screenshot of a green promotion that hadn't deployed anything.

The repo is at github.com/vondraysanford/DriftWatch. This is Phase 5 of 6. The dashboard does not exist yet.

The Shift Was Real, Not Synthetic

NASA C-MAPSS has four subsets. FD001, with one operating condition and 100 training engines, trained the champion model in Phase 2. FD002 has six operating conditions and 260 training engines. It had been sitting quarantined in the repo since Phase 1, untouched by any code, until now.

In Phase 5 the quarantine is lifted deliberately. An explicit --replay flag on the ingest script logs a warning and offsets FD002 unit numbers by 1000, so units 1–100 are always FD001 and units 1001–1260 are always FD002, in every table and in the prediction log. A replay script then sent 24 held-out FD002 engines through the live Container Apps endpoint as ordinary traffic: the last 20 raw cycles of an engine, every fifth cycle, 889 requests, zero failures. Every FD002 engine runs to failure, so the true label at every replayed cycle is derivable afterwards. A control replay sent 20 held-out FD001 engines through the same endpoint (781 requests).

Champion v1, the Phase 2 logistic regression trained on FD001 only, scored ROC-AUC 0.9921 on the FD001 control traffic and 0.5007 on FD002 traffic. A coin flip. It flagged every single FD002 window as failing: recall 1.00, precision 0.16. The sensor values under six operating conditions sit so far outside its training range that its output probabilities saturate.

The drift detector confirmed it. monitoring/drift.py reads the prediction log back from Blob Storage (raw inputs, computed features, outputs, timestamps, the record shape I locked in Phase 3) and runs Evidently's DataDriftPreset against the champion's training data. FD001 traffic: 0 of 17 raw input columns drifted, 0 of 99 features. FD002 traffic: 17 of 17 raw columns, 98 of 99 features. setting_1 sits 10,911 reference standard deviations from its FD001 baseline. Six sensors that are constant in FD001 started varying. The verdict rule fires when 30% of raw input columns drift or any operating setting drifts. It fired. The same report table showed 0.9921 next to 0.5007, side by side.

Three Things the Detector Got Wrong Before It Got Right

I want to record these because they're easy to skip over, and they're where the time went.

Minimum sample gate. My first negative control was 25 smoke-test predictions, really the same two windows repeated. The detector called drift. A two-point sample compared with 16,000 reference rows will drift on anything. Fix: no traffic below 200 predictions from at least 5 engines can drive a verdict. It reports as skipped instead. This is a standard gate in production ML and it's easy to forget.

Per-column threshold calibration. Evidently's default per-column cut is 0.1 reference standard deviations of normed Wasserstein distance. Twenty held-out FD001 engines, as in-distribution as traffic gets, scored up to 0.15 on single sensors, and the raw drifted share came out at 0.29 against a 0.30 threshold. That's not a detector; that's a coin. I calibrated the cut to 0.2: in-regime FD001 traffic now scores 0 of 17 drifted columns. FD002 traffic scores in the hundreds to thousands of standard deviations. Now I trust the threshold.

Reference set composition. After a promotion, the reference has to follow the new champion, or the monitor cries wolf about the regime the loop just resolved. My first attempt compared pure FD002 traffic against a mixed FD001+FD002 reference and flagged 15 of 17 columns. That was a composition artifact (27% FD001 in the reference, 0% in the window), not a new regime change. Fix: compare per regime, each window against the training engines for that regime, with the reference set following the champion's registry tag. A regime the champion never trained on gets compared against everything it did train on, which is exactly when drift should fire. I wrote the wrong claim into the guide before I checked it. The check caught it. The guide now records both.

From Verdict to Challenger in 9 Minutes

drift.yml runs every 6 hours (and on demand), authenticates via OIDC, pulls reference tables with DVC, reads the champion's tags from the model registry, publishes Evidently HTML reports as run artifacts, and on a drift verdict fires a repository_dispatch event. That needs a fine-grained GitHub PAT stored as an Actions secret, because the built-in GITHUB_TOKEN can't start workflows. It's a GitHub credential, not a cloud one, so the "no stored cloud secrets" claim still holds.

retrain.yml catches the event, trains a logistic-regression baseline and a 20-trial Optuna XGBoost search on FD001 plus the FD002 training split, and evaluates the best of them against the champion on a mixed held-out bench: the 20 FD001 and 52 FD002 engines neither model has seen, split by engine unit. Champion v1 scored 0.5463 overall (0.9923 on FD001, 0.5003 on FD002). The best challenger, a logistic regression retrained on both regimes, scored 0.9875 overall (0.9846 / 0.9887). The regime recovered at a cost of 0.008 on the original condition. Gain of +0.44 against a required margin of 0.005. Registered as version 2, tagged stage=challenger.

Registering a challenger changes nothing in production. What serves is the single version tagged champion, and the build-time model fetch follows that tag and nothing else. The retrain then dispatches a "model-registered" event to deploy.yml, whose deploy job runs in a GitHub Environment named production with a required reviewer. The run pauses, GitHub emails the reviewer, and on approval the job tags version 2 as champion, builds, deploys, and smoke-tests. The approver name and comment ("Promoting to champion") are recorded in the run. GitHub Environments with required reviewers are free for public repos. About 9 minutes from drift verdict to registered challenger; about 16 to a green promotion run, with the approval wait taking the largest share.

One OIDC detail that bit me: a job that declares a GitHub Environment presents a different OIDC subject, ...:environment:production instead of the branch. Each environment needed its own federated credential in Azure. The failing run's "Federated token details" block names the exact missing subject, which makes it easy to fix once you know to look there.

The Bug the Screenshot Caught

The promotion run finished green. "Model version: 2, Promotion: 2." The live endpoint kept answering version 1.

The image tag was the commit SHA. A promotion changes the model but not the commit. The build overwrote that tag in the registry with a version 2 image, but the Bicep template then received an identical image string, so Container Apps saw no change and kept the old revision. The smoke test passed because it only asserted the label, not which model version served it.

A deploy that reports success while the old model serves is exactly the failure this project argues against. Fix: image tags are now commit SHA plus model version, so a promotion produces a new revision. The smoke test now asserts that the live endpoint reports the model version the run deployed.

Then the same mutable tag showed its other face. After the app scaled to zero, the next cold start re-pulled the tag, which by then resolved to the version 2 image, and production changed with no deploy event at all. Unique tags close both doors.

I kept the green-but-wrong screenshot in the evidence folder under a filename that says what it is. It's the most useful artifact in the repo.

What the Numbers Looked Like After the Loop Closed

The same 24 FD002 engines, replayed through the live endpoint after version 2 became champion:

ROC-AUC:   0.9933  (from 0.5007)
Precision: 0.918
Recall:    0.849
Latency:   11 ms server-side, p50 97 ms round trip

The drift monitor afterwards, with its reference following the new champion (FD002 traffic compared with FD002 training engines), drifted on 0 of 17 raw columns and 1 of 99 features. Verdict: no drift. The loop answered the shift and then stopped talking about it.

Month-to-date Azure spend is under a few dollars against the $30 budget. Training ran on GitHub-hosted runners, which are free for public repos.

Why a Human Still Approves the Promotion

Auto-promotion needs canary traffic, automated rollback, and post-deploy monitoring to be safe. None of those exist in DriftWatch yet, and the gate is one metric on a held-out set the retrain job itself assembled. A human catches "too good to be true." The recorded approval is also how regulated teams operate in practice: there's an artifact that says who said yes and when.

If you're building something similar, the practical list I'd take from this phase: replay real held-out data as production traffic so the detector has something real to catch. Log raw inputs and computed features at serving time or you can't monitor anything later. Gate drift verdicts on a minimum sample count. Calibrate per-column thresholds against traffic you know is in-distribution before you trust them, and don't use the defaults blindly. Compare per regime and let the reference follow the champion. Separate "registered" from "deployed" with an explicit pointer. Put a human at the promotion step and record the decision. Tag images by everything that changes what runs, not just the code. Make the smoke test assert what was supposed to change.

And keep the screenshot of the run that lied.

Phase 6 is a React + Recharts dashboard over a small FastAPI metrics API: predictions, drift verdicts, and performance over time. That's the final post.