# Data Quarantine, Label Design, and the Phase Where You Fool Yourself: DriftWatch Part 1

Published: 2026-09-02 · Tag: MLOps
Canonical: https://vondraysanford.com/writing/2026-09-02-before-the-model-data-quarantine-label-design-and-the-phase-.html
Author: Vondray Sanford (https://vondraysanford.com)

> DriftWatch Phase 1 had no model in it. It was still the phase where the assumptions that wreck ML projects get made, or don't: label leakage, contaminated holdouts, feature code that drifts from serving. I treated all of it as infrastructure work.

The model is maybe 20% of an ML system. That started as a hunch. After Phase 1 of DriftWatch, my end-to-end MLOps pipeline for predicting turbofan engine failure from NASA's C-MAPSS sensor data, it reads more like a measurement. Phase 1 had zero modeling in it. No training loop, no evaluation metric, no loss curve. It was still the phase where most of the decisions that could sink the project got made. Label leakage, contaminated holdout sets, feature code that diverges between training and serving, data you can't reproduce: none of those are model problems. They're infrastructure problems, and I treated them that way.

This is a build log. The repo is at [github.com/vondraysanford/DriftWatch](https://github.com/vondraysanford/DriftWatch). I'm covering Phase 1 only: data versioning, exploration, and feature engineering. No accuracy numbers, no deployment details, no drift results. Those come in later posts, once they exist.

## Quarantine Is a Code Constraint, Not a README Note

C-MAPSS has four subsets, FD001 through FD004, with different operating condition counts and different fault modes. I train on FD001 only. FD002 and FD004 are quarantined: no exploring, no peeking, no "just a quick look at the distribution." They get replayed through the live endpoint later as production traffic, so the drift detector has a real regime change to catch instead of synthetic jitter I manufactured to make the demo look good.

The quarantine is enforced in code. The ingest script has an allow-list containing only FD001, and if you pass it FD002 or FD004 it refuses, with a message explaining why. The DVC pipeline stages depend on the FD001 files by name rather than on the raw folder, so the DAG itself documents that nothing else is read. A README note would have been advisory. The allow-list is a constraint. That difference matters at 11pm when you're working alone and tempted to "just see" what FD002 looks like.

FD003 goes unused entirely. Four subsets sounds like plenty of data. Once you decide what's training data and what's future drift signal, there isn't much left, and keeping the drift test clean matters more than squeezing out extra training rows.

## Versioning Without Secrets (and Two Infra Bugs I Had to Fix)

Raw data is tracked with DVC. The remote is an Azure Blob container created by the same Bicep template as the rest of the infrastructure. Auth is the Azure CLI login. No storage key, no SAS token, no connection string anywhere in the repo. Getting there took two fixes.

**Gotcha 1:** being subscription Owner does not let you read or write blobs. Owner is a control-plane role. Blob data access needs a data-plane role, `Storage Blob Data Contributor`, and without it the first `dvc push` would have failed with a 403 before a single byte moved. I added the role assignment to the Bicep module with the principal object ID as a deploy-time parameter, the same pattern I was already using for the budget alert email.

**Gotcha 2:** the redeploy failed because the budget module derived its start date from `utcNow()`. Azure refuses to change a budget's start date once it exists, so the template only worked in the month it was first deployed. I pinned the start date as a parameter instead of computing it at deploy time. Both of these are bugs in the infrastructure template, and I logged them that way rather than as "cloud quirks to be aware of." If the template doesn't redeploy cleanly, it's broken.

Verification: a `dvc pull` into a fresh empty repo, with nothing but the config and the pointer files, matched the originals byte for byte across all 14 files. If you can't reproduce the exact data, you don't have versioning. You have a storage bucket and a vague memory.

## Exploration, and the Filter I Almost Got Wrong

FD001 gives you 100 engines, 20,631 cycles, no nulls. Engine lifetimes run 128 to 362 cycles, median 199. The three operating settings are constant across the dataset, which confirms a single operating regime. I excluded them from features but kept them in the raw prediction log, because they become the obvious drift fingerprint when FD002's six regimes show up in production traffic later.

Six sensors never change across the dataset and one flips between two values, so 7 of 21 sensors are dropped by a simple rule: two or fewer distinct values. That rule is obvious. What's less obvious is that two other sensors, `s_8` and `s_13`, barely move in absolute terms, with a standard deviation around 0.07. A variance threshold would have cut them. But they track remaining useful life at about -0.57 Spearman correlation. That's real signal. Distinct-value count was the right filter and variance threshold was not, and the only way I found that out was by looking before deciding.

The label horizon question, "fails within N cycles?", got the same treatment. I measured how far each sensor's mean sits from its healthy baseline (more than 120 cycles from failure) in standard deviations, broken into bands. Inside the last 30 cycles, the strongest sensors are 2 to 4 SDs out. Between 30 and 60 cycles, they're 1.5 to 2. Past 60 they drop under 1 SD and blend into noise by 100 cycles. So N = 30 is a label the sensors can support. N = 60 would ask the model to flag engines where the signal is barely above noise. N = 15 would waste the clearest part of the degradation curve. Positive rate at N = 30 is 15%, and every training engine contributes exactly 31 positive rows because every engine in C-MAPSS runs to failure. There is no censoring in the training set.

## Rolling Window, Split Design, and Parity with Serving

Features summarize the last k cycles per engine. The binding constraint wasn't the training data. It was serving. The endpoint will receive the last k raw cycles of one engine, and the shortest engine in NASA's official test split is observed for only 31 cycles. I set k = 20, which scores every engine with headroom and keeps the request payload small. Per kept sensor: current value, rolling mean, std, min, and max over 20 cycles, deltas at lags 5 and 10, plus engine age. That's 99 feature columns and 18,731 rows after dropping 19 warm-up rows per engine.

The split is by engine unit, never by row. Cycles within one engine are near-copies of each other, so a row-based split would leak the engine's own history into the test set. The final split is 80 engines and 14,870 rows for training, 20 engines and 3,861 rows for test, positive rates 0.167 and 0.161 respectively. Those 20 test engines stay held out for the life of the project; nothing tunes against them. NASA's official test split is prepared as a secondary holdout: 100 different engines, cut off before failure, 11,196 rows, only 3% positive since just 25 of those engines ever come within 30 cycles of failure.

I verified training/serving parity the direct way. There's one feature module shared by the DVC pipeline and the future endpoint. I took the last 20 raw cycles of one engine, dropped the label column, called the same function the way the endpoint will call it, and compared the output to the training table's row for that cycle. Max difference was 3e-11, floating point noise. A 19-cycle request yields zero rows, which is how the endpoint will reject inputs that are too short to score. The pipeline runs in a few seconds with `dvc repro` and is a no-op on the second run. `dvc push` sent the six outputs to Azure.

## What Skipping This Phase Would Have Cost

If I had skipped Phase 1 and jumped to modeling, I would have a pipeline with a variance-based feature filter that threw away two useful sensors, a row-based split leaking engine history into test, a label horizon chosen by convention rather than signal strength, and a feature function that diverges from the serving path the first time someone touches it. None of those failures would show up as an obvious error. They'd inflate the metrics for a while and then fall apart in production.

Enforce data quarantines in code. Split by entity, never by row. Pick the label horizon from measured signal strength rather than from what other notebooks did. Share one feature function between training and serving, and test that it agrees with itself numerically. Treat infra-as-code redeploy failures as bugs in the template, because that's what they are.

Phase 2 is next: a logistic-regression baseline, then XGBoost tuned with Optuna, every run logged to MLflow in the Azure ML workspace, best model registered. That's where the modeling starts, and by then the ground under it should be solid.