Evals in CI: How to Catch Agent Regressions Before Production Does

Your deploy pipeline versions everything except the thing that matters most: what the agent actually does.

Code is versioned. Configs are versioned. The prompt is probably in a repo somewhere, hopefully with a review attached. But the agent's behavior, the mapping from inputs to actions, shifts whenever any of its inputs change: a model version bump you did not ask for, a retrieval corpus that gained three thousand documents overnight, a tool schema change from another team, a "small" prompt edit to fix one bad answer in a demo. None of those go through a behavioral regression test. They go straight to production, where your users run the eval suite for you, one incident at a time.