Problem
A backwards-incompatible production database migration renamed tables and columns, including users.autobuild_wizard_done, before the production API was updated.
Impact
Production API pods emitted repeated PostgreSQL “column does not exist” errors. Production was disrupted from approximately 9:12am to 9:51am EDT.
Cause
The migration reached the production database through Edge while production was still running code that referenced the old schema. The migration validation job allowed the rename operations through an unsafe-migration bypass because they were marked reversible.
Resolution
The team rolled production forward by deploying the latest image, restoring schema/code compatibility. Production was confirmed back at 9:51am EDT.
Resolved
Problem
A backwards-incompatible production database migration renamed tables and columns, including users.autobuild_wizard_done, before the production API was updated.
Impact
Production API pods emitted repeated PostgreSQL “column does not exist” errors. Production was disrupted from approximately 9:12am to 9:51am EDT.
Cause
The migration reached the production database through Edge while production was still running code that referenced the old schema. The migration validation job allowed the rename operations through an unsafe-migration bypass because they were marked reversible.
Resolution
The team rolled production forward by deploying the latest image, restoring schema/code compatibility. Production was confirmed back at 9:51am EDT.
