Skip to content

Runbook: Database migration issues

This guide shows how to runbook: Database migration issues in a reliable, repeatable way.

Symptoms

  • Service crashes or fails to start during deploy/upgrade
  • Requests fail with schema errors (missing table/column, constraint mismatch)
  • Migration job exits non-zero or reports failed migrations

Decision tree (fast path)

flowchart TD
  A[Upgrade failing] --> B{DB reachable?}
  B -->|No| C[Fix connectivity/creds/network policy]
  B -->|Yes| D{Failed migrations present?}
  D -->|Yes| E[Investigate the failed migration and roll forward]
  D -->|No| F{Migrations behind the code?}
  F -->|Yes| G[Apply migrations (up)]
  F -->|No| H[Check app logs for non-migration causes]

Quick triage (copy/paste)

Local dev:

cd api
make migrate-preflight
make migrate-status

Production:

  • Run your migration job/container and capture output.
  • If you have DB access, run the same migrate commands in a controlled job environment.

Safe remediations

  • Prefer roll forward:
  • fix the migration and deploy a new migration
  • re-run up
  • Avoid down in production unless you have an explicit rollback plan and DB backups.

See the migration policy and commands: Migrations (safe upgrade)

Verification

  • migrate status shows no failed migrations.
  • Service starts cleanly and /readyz becomes 200 OK.