# Fifty Trials Couldn't Beat Logistic Regression: DriftWatch Part 2

Published: 2026-09-03 · Tag: MLOps
Canonical: https://vondraysanford.com/writing/2026-09-03-fifty-trials-couldn-t-beat-logistic-regression-driftwatch-pa.html
Author: Vondray Sanford (https://vondraysanford.com)

> Phase 2 of DriftWatch covered training, MLflow tracking, and model registration in Azure ML. It ended with a surprise: a scaled logistic regression beat XGBoost, including XGBoost after 50 Optuna tuning trials, on engines none of the models had seen.

Phase 1 of [DriftWatch](https://github.com/vondraysanford/DriftWatch) covered data versioning, exploration, and feature engineering. Phase 2 was supposed to be the part where I trained a decent XGBoost, tuned it with Optuna, logged everything to Azure ML, and registered the winner. All of that happened. The winner just wasn't XGBoost.

A scaled logistic regression, `StandardScaler` + `LogisticRegression(C=1)` and nothing exotic, beat both an XGBoost with sensible defaults and the same XGBoost after 50 Optuna trials, on 20 held-out engines that no model touched during training or tuning. The registration script picks the winner by held-out ROC-AUC automatically. I didn't choose by hand. The numbers came back that way, and I registered the baseline as version 1.

This post covers the numbers, why I think the linear model won, how the tracking and registry are wired, and two gotchas I hit with MLflow 3 on Azure ML. Nothing is deployed yet. No endpoint, no drift detection, no retraining. That's Phase 3 and beyond.

## The Numbers

All three models train on 80 FD001 engines and get scored on 20 held-out engines split by engine unit, plus NASA's official test split of 100 unseen engines cut off before failure. The label is binary: does this engine fail within the next 30 cycles? Measurements are from 2026-09-02 on FD001 only.

**Logistic regression** (the baseline): held-out ROC-AUC 0.9923, PR-AUC 0.9673, recall 0.897 / precision 0.914 at a threshold of 0.407. Official NASA holdout ROC-AUC: 0.9929.

XGBoost with sensible defaults (300 trees, depth 4, learning rate 0.1): held-out ROC-AUC 0.9899, PR-AUC 0.9589, recall 0.873 / precision 0.897 at threshold 0.461. Official holdout: 0.9908.

XGBoost tuned by Optuna, 50 trials, TPE sampler: best cross-validated ROC-AUC was 0.9886. Held-out ROC-AUC 0.9899, PR-AUC 0.9593, recall 0.845 / precision 0.903 at threshold 0.537. Official holdout: 0.9912. Best settings landed at 400 trees, depth 5, learning rate 0.044, subsample 0.80, column sample 0.67, min child weight 10, tiny L2, positive-class weight around 1.0.

The grouped cross-validation inside the training engines told the same story before the test set did: logistic regression was 0.990 out-of-fold vs. 0.9886 for the best XGBoost trial. The grouped CV was trustworthy enough that I could have registered the baseline without looking at the held-out set. I looked anyway.

## Why the Linear Model Won

My read: the 20-cycle rolling features (mean, std, min, max, deltas at lags 5 and 10 per sensor, plus engine age) already linearize the degradation curve. FD001 is a single operating regime with 14 informative sensors that all drift monotonically toward failure. By the time those features reach a model, there isn't much nonlinearity left for trees to find, and trees add variance while looking for it.

Tuning cannot fix a model-family mismatch. Fifty Optuna trials moved cross-validated ROC-AUC from 0.9880 to 0.9886. Six basis points. The default XGBoost and the tuned XGBoost tied on held-out ROC-AUC at 0.9899. The tuner found the right neighborhood quickly and then shuffled around in it.

None of this means XGBoost is bad. XGBoost is great. It means you establish the baseline first and let the numbers pick the winner. If I'd started with XGBoost and tuned it for a week, I'd have a slightly worse model and a much more complicated system. Running the baseline takes ten seconds and sets the bar everything else has to clear.

## Evaluation Discipline

Every tuning trial is scored by 5-fold `GroupKFold` by engine unit inside the 80 training engines, so the 20 held-out engines never influence a single hyperparameter setting. The operating threshold is picked the same way: max F1 on out-of-fold predictions inside the training engines, then reported on held-out engines at that threshold and at 0.5. Those 20 held-out engines stay locked for the life of the project. They're the benchmark when a challenger model shows up in the drift phase.

The official NASA holdout adds a wrinkle. On NASA's test split the baseline keeps ROC-AUC 0.993, but precision and recall at the training-chosen threshold fall to 0.73 / 0.73, and PR-AUC drops to 0.84. The cause is class prior shift: the training table is 16% positive, but the official holdout is only 3% positive because those engines are cut off before failure and only 25 of 100 ever get within 30 cycles. Ranking quality held. A fixed threshold didn't.

This is exactly the kind of shift the drift monitor is for. It's also a solid argument for logging probabilities rather than labels. Bake the threshold into the output and you've thrown away the information you need to recalibrate when the class prior moves.

## Tracking, Lineage, and the Registry

Every run lands in the Azure ML workspace, which acts as both the MLflow tracking server and the model registry, so there's no separate MLflow server to stand up. Each run is tagged with the DVC hash of the training table it saw (`md5: ea34a561...`), the git commit, the label horizon, and the rolling window size. That's the lineage chain: data version → code version → run → model version. If a model breaks in production, I can trace it all the way back.

The Optuna search is logged as one parent run with 50 nested trial runs. The final model fitted on the best settings lives in the parent run, so it sits beside the baseline in Azure ML Studio and compares cleanly. The registered model version is tagged with the source run ID, the held-out metric, the data hash, and the commit.

All training ran on my Mac in minutes: roughly 8 seconds per Optuna trial, about 6.5 minutes for the full search. The workspace only tracked it, so no Azure compute was billed for Phase 2.

## Two Gotchas I Hit Along the Way

**Gotcha 1 (the real one): MLflow 3 on Azure ML.** MLflow's `log_model` in MLflow 3 first creates a "logged model" entity by hitting `/api/2.0/mlflow/logged-models`. Azure ML's tracking server returns 404 for that endpoint. The run had params, metrics, and plots, and no model artifact.

The fix: save the model in the sklearn flavor to a temp directory with `mlflow.sklearn.save_model` and upload it with `log_artifacts` under `model/`. Identical layout on disk, no broken API call. The same root cause bites `runs:/<id>/model` URIs, because listing or registering through them queries the logged-models search endpoint and 404s. Address the model by the run's artifact URI instead, and pass `run_id` explicitly when creating the model version so the lineage link survives. Loading via `models:/<name>/1` works fine once the version is registered.

**Gotcha 2 (small):** the Azure SDK logs every HTTP request at `INFO`, which buried the training output entirely. Set the `azure`, `azureml`, `urllib3`, and `msal` loggers to `WARNING` at the top of the script and move on.

```
import logging
for name in ("azure", "azureml", "urllib3", "msal"):
    logging.getLogger(name).setLevel(logging.WARNING)

```

## Configuration and What's Next

Nothing Azure-related is hardcoded. Three environment variables (tracking URI from `az ml workspace show`, experiment name, registry model name) with a committed `.env.example` and a gitignored `.env`. None of them are secrets, but they still don't belong in the codebase.

Phase 2 deliverables: a baseline, a tuned XGBoost, MLflow tracking with full lineage, and a registered model version 1. The model was always going to be a small part of the system. This time it was a small model too.

Phase 3 is a FastAPI `/predict` endpoint that takes the last 20 raw cycles of one engine, computes features with the same module used in training, scores against registry version 1 baked into the image at build time, and logs every prediction: raw inputs, features, output, timestamp. That log is what makes the drift phase possible. Without it there's nothing to compare against the training distribution. I'll write that up once it's built.

If you want to poke at the code, the repo is at [github.com/vondraysanford/DriftWatch](https://github.com/vondraysanford/DriftWatch).