# Four Failures, One Green Run, and the Teardown That Held Every Time

Published: 2026-09-04 · Tag: DevOps
Canonical: https://vondraysanford.com/writing/2026-09-04-four-failures-one-green-run-and-the-teardown-that-held-every.html
Author: Vondray Sanford (https://vondraysanford.com)

> DriftWatch Phase 4 build log: how enterprise DevOps experience, not ML theory, solved secretless CI/CD, identity scoping, and a managed endpoint that had to prove it worked before it was allowed to disappear.

Parts 1 through 3 of DriftWatch covered the data quarantine, the model that beat everything fancier, and the containerized API. Phase 4 is where the ML work mostly stops and the DevOps work starts in earnest. The interesting engineering here was not the model, which was already registered in Azure ML. It was identity with nothing to leak, infrastructure split so CI can't corrupt production, a pipeline that fails when the endpoint answers wrong, and a cost guardrail that got tested by three real failures before it ever saw a green run. This is a build log, not a tutorial. The failures are the point.

## Nothing Stored, Nothing to Rotate

GitHub authenticates to Azure with an OIDC federated credential. The Entra app registration holds zero password credentials. I verified this with `az ad app credential list`, which returns an empty array. GitHub signs a short-lived token that names the repository and branch, and Azure trusts it only when the subject matches the federated credential exactly. No secret is ever stored anywhere.

The eight values the workflow needs (client ID, tenant ID, subscription ID, resource names) are GitHub repository *variables*, not secrets, because possession of them grants nothing without a token only GitHub can mint. The repo is public, the workflow logs are public, and those identifiers are visible by design. My [KodeKloud contribution](https://github.com/vondraysanford) was a hardcoded-credentials fix, so a secretless pipeline of my own isn't an aesthetic choice. It's the whole point.

RBAC is scoped per resource rather than blanket subscription access: `AcrPush` on the registry, `Contributor` on the resource group, `AzureML Data Scientist` on the workspace, `Storage Blob Data Contributor` on the storage account. `Contributor` is control-plane only, so each data-plane action needs its own role assignment. Every command is recorded in `infra/oidc-setup.md`, including teardown.

The first run failed immediately with `AADSTS700213`. GitHub was presenting the subject as `repo:vondraysanford@101304529/DriftWatch@1333783711:ref:refs/heads/main`, with the account ID and repository ID embedded, rather than the name-based form most documentation shows. I verified both numbers against the GitHub API before trusting the string, then updated the federated credential to match exactly. The ID-qualified form is stronger: renaming the repo or account can't transfer trust, and nobody who later claims a freed-up name can authenticate. The lesson is to read the subject out of the error instead of assuming a format.

## Split Infrastructure So CI Can't Revert Production

The Container Apps environment, a user-assigned managed identity, and that identity's two roles (`AcrPull`, `Storage Blob Data Contributor`) live in `main.bicep` because they never change. The container app itself lives in a separate `containerapp.bicep` that takes the image tag as a required parameter. That is the only template CI touches.

If the app lived in `main.bicep` with a placeholder image default, any later redeploy of the durable footprint would revert production to the placeholder without anyone noticing. I've seen that class of bug in enterprise deployments: a well-intentioned infra change rolls back an application version because the template wasn't split. User-assigned identity rather than system-assigned means the role assignments are made once at setup and not recreated on every app creation.

I proved the deploy path by hand before wiring CI. That caught a real bug: my local image was `arm64` (Apple Silicon), and Container Apps runs `amd64`. The build now states `linux/amd64` explicitly. A pipeline failure and an application failure look identical unless you've already ruled one out.

The pipeline itself, `deploy.yml`, triggers on push to `main` and runs 18 offline checks first: feature contract (99 columns, dropped sensors never leak in, settings never become features), window-length rules, and request-schema rejections for too few cycles, two engines at once, a sensor gap, an unknown column, and a missing sensor. No cloud access, no data pull, done in seconds. Then it pulls the registered model, builds and pushes to ACR, deploys Bicep, and smoke-tests the live endpoint. The smoke test fails the run if the near-failure example isn't flagged. A deploy that reports success with a broken endpoint is worse than a failed deploy. Every action is pinned to a full commit SHA resolved from the GitHub API, and one lookup returned a tag object's SHA rather than the commit it points to, which would have failed at runtime. The verification step paid for itself on the first pass. A docs-only commit is skipped by `paths-ignore`, so evidence screenshots never trigger a redeploy.

## The Persistent Demo, Measured

The always-on endpoint runs on Azure Container Apps in `eastus2`, min replicas 0, max 1. Probabilities are identical to local inference: `1.0000` for the near-failure window, `0.0352` for the healthy one. Server-side latency is 12 ms per request (features + inference + synchronous log write). Round trip from a laptop is p50 93 ms, p95 101 ms. Most of that is network distance, which is why I report both numbers instead of one.

The app scales to zero five minutes after the last request. The next cold request takes 32.7 seconds while a replica starts. That's the real cost of $0 idle. One minimum replica would remove the cold start for roughly $30/month. I'm keeping min replicas at 0 for this phase because this is a demo. Nobody is holding it to a production SLA.

Prediction logs go to Blob Storage via the app's own managed identity, with no connection string or account key anywhere. After the first green run: 25 predictions logged in one JSONL file per replica per hour, carrying all 20 raw cycles, all 99 features, the output, the threshold, and the model version. Those records are the future drift fingerprint. Phase 5 will replay the quarantined FD002 regime through the live endpoint as production traffic so Evidently has a real regime change to catch.

## Stand It Up, Prove It, Tear It Down

Azure ML managed online endpoints bill per instance-hour with no scale-to-zero, roughly $70+/month if forgotten. So the demonstration is a separate, manual-only workflow that stands one up, proves it serves, captures evidence, and deletes it in the same run. Teardown is under `always()`. A final step polls until the endpoint is confirmed gone and fails loudly if it isn't. It never runs on merge.

It took four attempts. The teardown held on every failure. That result matters more to me than the green run.

**Attempt 1:** `SubscriptionNotRegistered ... Resource provider [N/A]`. Managed endpoints require the `Microsoft.PolicyInsights` provider, and Azure's error named neither the provider nor the subscription. Registered it (75 seconds, free). The workflow now checks providers up front and names the missing one.

**Attempt 2:** `az ml online-deployment create` requires a YAML file even though `az ml online-endpoint create` happily takes `--name`. A CLI asymmetry. Added the YAML.

**Attempt 3:** Azure's no-code MLflow deployment crashed at startup. Its own scoring script, `/var/mlflow_resources/mlflow_score_script.py`, imports `azureml.ai.monitoring`, and the environment Azure built for the model didn't contain that package. Image pull, model download, and container start all succeeded. The scoring server died on Azure's own dependency. The liveness probe returned 502 for twenty minutes before Azure gave up. I diagnosed it from the container log the workflow had captured as an artifact right before teardown. Without that capture step there would have been nothing to read. Fix: own the scoring script and the environment (bring-your-own), which is the production pattern anyway. The managed endpoint now takes raw cycles and computes features with the same `data/features.py` as training, so both endpoints share one contract.

**Attempt 4:** `Standard_DS3_v2` was rejected at validation. Managed endpoints check quota at double the SKU's vCPU count ("Current usage/limit: 0/6. Additional needed: 8"), and the subscription's DSv2 family limit is 6. `DS2_v2` needs 4 and fits. Azure's warning that `DS2_v2` "may be too small" is generic; the model peaks at 254 MB against the SKU's 7 GB.

**Run 4, green in 18m 29s:** endpoint created in 1m 6s; custom environment built and deployment live in 9m 25s; five invocations, all correct (`1.0000` / label 1, `0.0352` / label 0, identical to the Container App, and the run fails on a wrong label); logs captured; teardown; "Confirm nothing is left billing" passing after 6m 47s. Round trips were 2.9 to 3.3 seconds around the `az ml online-endpoint invoke` CLI call, dominated by CLI startup and token acquisition. The model scores in about 10 ms locally. Server-side latency wasn't isolated in this run, and I'm not going to make up a number. Month-to-date spend at phase close: under $1 against a $30 budget alert, with the managed endpoint's total lifetime across four attempts measured in minutes. The operating threshold (0.4068) travels with the registered model version as a tag, so any deployment reads it from the registry instead of hardcoding it.

## What I'd Tell Someone Starting This

Federated credentials mean nothing to rotate and nothing to leak. The identifiers can be public. Match the OIDC subject GitHub actually presents rather than the one the docs show. Read it out of the error.

Split durable infrastructure from the thing CI redeploys. A single Bicep file with a placeholder image default is a production rollback waiting to happen.

Prove the deploy path by hand once before automating it. You'll find the `arm64` vs `amd64` problem, or whatever your equivalent is, in ten minutes instead of in a pipeline log at 11 PM.

Make the smoke test assert correctness. A 200 proves nothing. Pin actions to commit SHAs and verify the SHA resolves to a commit rather than a tag object. Capture container logs before teardown or you diagnose blind. Put teardown under `always()` and poll until it's really gone. When a platform's own convenience path breaks on its own dependency, own the script.

Phase 5 is Evidently reference-vs-current on the prediction logs, FD002 replayed as production traffic, and a threshold crossing that fires `repository_dispatch` into a retrain workflow: challenger vs. champion, register the winner. That's where the drift detection gets built. The code for Phase 4 is at [github.com/vondraysanford/DriftWatch](https://github.com/vondraysanford/DriftWatch) and the evidence screenshots are under `docs/evidence/phase-4/`.