Decoupling a Data Pipeline From the Tables It Was Never Supposed to Depend On

I built a pipeline framework on top of a convenience feature, and for about a year it looked like the best decision in the design. The framework generates analytics pipelines from declarative configuration: a user describes the events, stages, and metrics they care about, and the framework produces the Spark jobs, output tables, and dashboards. To make those pipelines incremental, I required every upstream source to be a table with change data feed enabled. The table format's changelog told us exactly which rows had been inserted, updated, or deleted since the last run, so every generated pipeline processed only deltas, for free, with no bookkeeping code of its own.

The fragility hidden in that decision took longer to surface than it should have because it never showed up as a framework bug. It showed up as other people's schema changes.