# DriftWatch, measured: what a logistic regression, a coin flip, and a 32-second cold start taught me about ML systems

Published: 2026-09-08 · Tag: mlops
Canonical: https://vondraysanford.com/writing/2026-09-08-driftwatch-measured-what-a-logistic-regression-a-coin-flip-a.html
Author: Vondray Sanford (https://vondraysanford.com)

> Six phases, 21 days, and every number measured. The wrap-up on DriftWatch: what it produced, what it cost, what broke, and what's still not done.

The thesis going into DriftWatch was simple: the model is about 20% of an ML system. Versioning, deployment, monitoring, and retraining are the point. Six phases and 21 days later, I have numbers that confirm that harder than I expected. This is the wrap-up post. Everything below was measured, not estimated.

## The three numbers that tell the story

The task: predict turbofan engine failure within 30 cycles from NASA C-MAPSS sensor data. I trained on 80 engines, evaluated on 20 held-out engines split by engine unit (never by row, because rows inside one engine are correlated and a row split makes the metrics fiction), and also scored against NASA's official test holdout of 100 unseen engines.

The first number is **0.9923**. That's the held-out ROC-AUC for the logistic regression baseline. XGBoost at defaults got 0.9899. XGBoost with 50 Optuna trials got 0.9899. The baseline won. On 20-cycle rolling features, FD001's degradation is close to linear, and fifty tuned trials could not beat a scaled logistic regression on engines it had never seen. Registry version 1 is the baseline.

The second number is **0.5007**. That's what happens when you replay a regime the model has never seen. FD002 has six operating conditions where FD001 has one. I quarantined it in the ingest script from day one, then lifted the guard deliberately in Phase 5 and replayed 24 FD002 engines through the live endpoint as production traffic. The champion became a coin flip. Recall 1.00, precision 0.16. It flagged every window as failing. Six sensors that are constant in FD001 started varying, and 17 of 17 raw columns drifted. Setting 1 moved 10,911 standard deviations from baseline.

The third number is **0.9933**. That's the ROC-AUC on those same 24 FD002 engines after the loop closed: drift detected, retrain triggered, challenger registered, human approved, version 2 promoted. The regime is recovered at a cost of 0.008 on the original one.

## What the loop actually is, and where a human sits

The drift detector runs on a schedule every six hours as a GitHub Actions workflow. It uses Evidently with a per-column normed Wasserstein distance cut at 0.2 reference standard deviations, and it declares drift at 30% of raw input columns drifting or any operating setting drifting. It refuses to issue a verdict on fewer than 200 predictions from 5 engines, which matters because the calibration story is less clean than the final numbers suggest. Detector defaults produced a false alarm on 25 records and a near miss on 781 in-distribution records before I set the minimum-sample gate and the per-column cut from held-out FD001 engines.

When the detector returned `DRIFT`, it fired a `repository_dispatch` event via a fine-grained GitHub PAT scoped to this repo. `GITHUB_TOKEN` can't start workflows, so a PAT is the right tool there. That's a GitHub credential, not a cloud secret, so the "no stored cloud secrets" claim still holds. The retrain workflow trained a baseline and a 20-trial XGBoost search on the combined data, judged them against the champion on a mixed held-out bench of 20 FD001 plus 52 FD002 engines, and registered the winner as version 2 tagged challenger.

Then everything paused. A GitHub "production" environment gate held the promotion workflow until a reviewer approved it with the comment "Promoting to champion." Registering never changes what's served. Only the approved promotion does. Drift verdict to registered challenger: about 9 minutes. Drift verdict to a green promotion run: about 16 minutes, with the approval wait being the largest piece.

## The bug that made a green run lie

This is the most transferable lesson in the whole project. Promotion run #9 finished green, reported "Model version: 2," and the live endpoint kept answering version 1.

The image tag was the commit SHA. A promotion doesn't produce a new commit, so the tag didn't change. Azure Container Apps saw no template change and kept the old revision. The smoke test passed because it only checked that the response returned a valid label, not which model version produced it.

Two fixes: tag images with commit plus model version, and make the smoke test assert the served model version. The screenshot of that green run that lied is in the repo. I kept it there on purpose.

The related issue is that a mutable image tag means a cold start after scale-to-zero can silently change production with no deployment event. Same fix covers both.

## Cost and the cold start as an honest trade

Month-to-date Azure spend stayed in the low single digits of dollars. The budget alert, deployed by Bicep before the first deployment, was set at $30/month with notifications at 50, 80, and 100 percent. The persistent demo idle cost is $0 because the Container App scales to zero with min replicas set to 0.

The price of that is a 32.7-second cold start. I'm not hiding that number; it's on the [live dashboard](https://driftwatch.vondraysanford.com/) and in the README. A minimum of one replica would remove it for roughly $30/month. For a portfolio demo, the trade is obvious. For a production system with an SLA, it's a different conversation.

Server-side per request once the instance is warm is 12 ms (features plus inference plus log write). The endpoint takes the last 20 raw cycles for one engine and computes features server-side with the same feature function used in training. Parity between served and training features is verified to 6e-15. Every prediction is written to the log sink before the response returns; with the sink down, `/predict` returns 500 rather than an unlogged result.

The Azure ML managed online endpoint, which bills per instance-hour with no scale-to-zero, is a manual-only workflow: stand it up, prove it, tear it down in one run. Run #4 went green in 18m 29s after three earlier failures from a missing `azureml-ai-monitoring` package in Azure's own no-code MLflow path and a quota validation issue that required `Standard_DS2_v2`. The teardown step runs under `always()` and was tested by three failures before it was tested by success. Nothing left billing.

## What's still not done

The managed endpoint's server-side latency was not measured, only the CLI round trip. FD004 was never replayed; the drift story uses FD002 only. There's no remaining-useful-life regression, no LSTM, and no pytest suite. CI runs a single feature-contract sanity check. Training stages are not in `dvc.yaml`; the DVC DAG covers data only. Evidence screenshots exist for phases 4, 5, and 6; phases 0 through 3 have not been backfilled. The Container App was not load-tested.

None of that changes what was measured. It just means the project is a real project, not a cleaned-up demo where everything worked the first time.

## The repo and the dashboard

The code is at [github.com/vondraysanford/DriftWatch](https://github.com/vondraysanford/DriftWatch). The dashboard is at [driftwatch.vondraysanford.com](https://driftwatch.vondraysanford.com/). The API behind it scales to zero, so the first numbers take about half a minute. That's the trade. Every chart on the dashboard has a table twin, light and dark modes follow the portfolio theme, and the chart blue was validated for color-vision-deficiency separation in both modes.

The same scale-to-zero pattern, OIDC setup, and Cloudflare Pages front end run in [DocQuery](https://docquery.vondraysanford.com/) and [AgentReview](https://agentreview.vondraysanford.com/). Once you've wired it once, it reuses cleanly. Enterprise DevOps patterns transfer directly to ML systems. The model was 20% of the work. The thesis held.