# Raw Cycles In, Probability Out: DriftWatch, Part 3

Published: 2026-09-03 · Tag: MLOps
Canonical: https://vondraysanford.com/writing/2026-09-03-raw-cycles-in-probability-out-driftwatch-part-3.html
Author: Vondray Sanford (https://vondraysanford.com)

> Phase 3 of DriftWatch is a FastAPI endpoint that accepts raw sensor cycles, computes features from the same code training used, and refuses to answer if it cannot write the prediction down. One decision drives all of it.

The most common silent failure I know of in production ML goes like this: the feature code in your training pipeline and the feature code in your API diverge, and nobody notices until the model has been wrong for a month. Two files, same logic, different authors, different moments in time. They start identical and drift apart. The model was trained on one version of "rolling mean of sensor 11 over 20 cycles." The API has been computing a slightly different one since the last refactor. You can't see it in the output, because the probability still looks reasonable. You see it when you finally compare a batch of API predictions against what the training code would have produced for the same inputs.

DriftWatch Phase 3 is my answer to that problem. The API takes raw sensor cycles, the 26-column rows straight out of the NASA C-MAPSS file, computes the 99 features server-side with the same `build_features` function the training pipeline imports, and scores the result. One function. Training uses it. The API uses it. The container ships it. This post covers what that looks like in practice, what the container bakes in at build time, and why a failed database write fails the whole request.

If you're just landing here: [the repo is public](https://github.com/vondraysanford/DriftWatch). Part 1 covered data versioning and feature design. Part 2 covered training, tracking in Azure ML via MLflow, and the logistic-regression baseline that beat 50 rounds of tuned XGBoost and became version 1 in the registry. This post is Phase 3 only, a FastAPI service running locally in Docker. Not deployed yet. No CI/CD yet. No drift detection yet. That's Phase 4 and beyond.

## The endpoint shape and why 20 cycles is a hard floor

`POST /predict` takes the last 20 or more raw cycles of one engine. Each row is the 26 columns from the C-MAPSS file: unit number, cycle, three operating settings, 21 sensor readings. The service computes all 99 features from that window, scores the last cycle, and returns a failure probability, a binary label at the model's own operating threshold, the model name and version, a prediction ID, and latency. `GET /model` returns registry provenance. `GET /health` checks both the model and the log sink.

The 20-cycle floor isn't arbitrary. Back in Phase 1 I chose rolling statistics over a 20-cycle window partly because the shortest engine in NASA's official test split is observed for only 31 cycles. You can't compute a 20-cycle rolling statistic from fewer than 20 cycles, so if a request carries fewer rows the API returns 422 rather than guessing. The alternative is computing features over a shorter window without telling anyone, which means the model is scoring something it was never trained on.

Four validation cases return 422, and I verified each of them by hand against the running container. Fewer than 20 cycles. Cycles from more than one engine in the same request. A gap in the cycle sequence. An unknown column. That last one matters more than it looks. Accepting an extra field without complaint is how you end up serving a model that's ignoring an input someone thought was being used. The request schema is generated from the same `data/schema.py` the ingestion code uses, so the API's idea of a valid row can't drift from the pipeline's on its own.

## Baking the model in at build time

A script pulls the registered model version from the Azure ML workspace registry before the image is built, writes it into the build context, and generates a `model_info.json` alongside it. That file carries the run ID, the operating threshold, the held-out metrics, the DVC hash of the training data, and the git commit that produced the run. The container loads from a local path at startup. No registry call at runtime, no credential needed to load the model, and cold starts are fast. That matters, because when this eventually runs on Container Apps at min replicas 0, every cold start counts.

Provenance still travels with the model. `GET /model` returns exactly which run produced it, which data hash it saw, and what threshold it was registered with. It's the same pattern I use in DocQuery and AgentReview: ship the artifact with its lineage attached rather than hoping you can reconstruct it later from logs.

The build script also checks that the registered model doesn't require an estimator library the serving image doesn't pin. If someone registers an XGBoost model after Phase 2 and tries to build without updating the image dependencies, the build fails loudly instead of the container crashing at startup. I'd rather catch that in 22 seconds during `docker build` than in a midnight incident.

One build context detail: because the image needs the shared `data/` package for `build_features`, the Docker build runs from the repo root with `-f serving/Dockerfile`, not from inside `serving/`. A `.dockerignore` keeps the context to the code the image needs. The raw dataset, notebooks, Bicep templates, and training scripts never enter the build. Both base images are pinned by digest, not just by tag.

## Prediction logging is not optional, and a failed write fails the request

Every prediction writes one record before the response goes out. The record contains the raw inputs as received (all 20 cycles, all 26 columns), all 99 computed features, the probability, the label, the threshold, the model version, and a UTC timestamp. The sink is configuration-driven: Postgres in Docker locally, JSONL appended to Azure Blob Storage via managed identity once this runs on Container Apps. The Container Apps filesystem is ephemeral at min replicas 0, so local files aren't an option there. Same record shape either way.

I log the operating settings even though the model doesn't use them as features. They're constant in FD001, the dataset this model was trained on. But they're the clearest fingerprint of a regime change when FD002 gets replayed in Phase 5, and if I don't log them now I can't add them retroactively. The general rule: log more than you think you need, because the data you didn't capture is gone.

Logging is synchronous, and a failed write fails the request. If the database is down, `/predict` returns 500 with "prediction could not be logged; not returning an unlogged result," and `/health` returns 503. I verified this by stopping the database container mid-session. The service reconnected and served again within 2 seconds after I brought it back up, and 53 predictions were logged across the session with none dropped.

The alternative, returning the prediction and dropping the log line, means the drift monitor goes blind at exactly the moment something is going wrong with your infrastructure. I'd rather the caller see an error and retry. It's the same instinct as failing a deploy loudly instead of shipping something half-configured. A partial system that looks healthy is worse than a broken system that says so.

## The one parity check that proves the claim

Once the service was running, I did the one check that matters for the "one function" guarantee. I took the probability the API returned for engine 8 at cycle 150 and compared it to what the registered model gives for the same engine and cycle in the feature table the DVC pipeline built during training. They agree to `6e-15`, which is floating-point noise. That's the proof that the training path and the serving path compute the same thing. It's also a test I can run again after any change to the feature code, which is the point.

For the sanity check on the model itself: engine 8 at its final observed cycle scores a failure probability of 1.0000 with label 1. The same engine at cycles 41 to 60, well before failure, scores 0.0352 with label 0. The model knows where it is in the lifecycle.

Measured on an Apple Silicon Mac via `docker compose`, the image builds in 22 seconds. The stack is healthy 2 seconds after start. Fifty sequential requests through the container averaged p50 6.7 ms and p95 8.8 ms end to end, including feature computation, inference, and the synchronous database write. Fast enough for what this pipeline needs.

## What's next and what to take from this phase

Phase 4 is the GitHub Actions pipeline using OIDC federated credentials, with no stored cloud secrets, deploying this container to Azure Container Apps at min replicas 0, plus a one-off Azure ML managed online endpoint that gets stood up and torn down in the same session because it bills around the clock. That post will cover the OIDC setup in detail, because "secretless CI/CD" is easy to say and mildly annoying to wire up correctly.

The practical things I'd pull from this phase if you're building something similar:

Accept raw inputs and compute features server-side using the training code. Then write a test that compares the API's probability to the training pipeline's probability for the same input and verify they agree numerically. If you have two copies of the feature code, you have two opportunities for them to disagree.

Bake the model into the image at build time and carry its provenance with it. A `model_info.json` with the run ID, data hash, and git commit costs almost nothing to generate and makes every question about "what did this container know" answerable without digging through logs.

Log the raw inputs and computed features, not just the output. You cannot add that data retroactively. When drift detection comes online in Phase 5, it's going to need the full record (inputs, features, predictions, timestamps) to tell you whether the model is seeing data it wasn't trained on.

Make an unloggable prediction an error rather than a silent gap. The drift monitor is only as good as the data it sees. A gap in the log at exactly the moment something broke is the worst possible outcome.