How Cache Invalidation Caused Memory Pressure and API Failures at Scale

Most production systems do not fail loudly. They degrade quietly.

This case started with an API that was technically “healthy.” It was serving traffic, success rates were high, and no critical alerts were firing. Yet recurring degradation periods caused the service to miss its service-level agreement (SLA) target for request success. During the worst degradation periods, request success could fall to approximately 98%, against a target of 99.9%. Some requests were simply failing. Because the API sat directly in the purchase path, the gap between 98% and 99.9% meant real checkout failures, abandoned transactions, and lost product revenue.