Hosted language models change under stable names. A provider updates weights, a system prompt, a decoding default or a safety layer, and the application built on top keeps running as if nothing happened, until a refusal rate climbs, a tool call changes shape, or a number in a report stops matching. Most monitoring stacks answer this with an alert. This talk is about the step after the alert: making the runtime refuse to proceed on a model that has drifted, and recording why.
epicormic is an open-source Python library that measures provider drift with canary panels: fixed probe sets replayed in windows against a hosted model and, in parallel, against a locally pinned control model that cannot drift. Per-probe and pooled tests (Mann-Whitney with Cliff’s delta, Fisher’s exact test, a stratified permutation test, and an anytime-valid e-process across windows) produce a verdict of stable, warning, drift or unknown at a declared false-alarm rate, and separate declared changes, where the provider’s contract fingerprint moved, from silent ones. The verdict drives fail-closed levers in pollard, an open-source governed execution runtime: dispatch is refused on drift, side-effectful tool calls require confirmation, and unacknowledged fingerprint changes are gated. Every verdict is a value-free, sealable evidence record that can be exported and audited.

Attendees will leave with a method they can apply with or without the library: a canary plus a pinned control, a calibrated verdict, and a lever that stops the run.