Zero downtime deployment, including the database

The rule underneath everything

During any deploy, there is a window when old and new code run side by side: old pods still draining, new ones starting, background workers on the previous release, jobs enqueued by old code processed by new code. So every release has to satisfy two conditions:

• The new code works with the old schema (or the schema change ships first).
• The old code works with the new schema.

Any change that breaks one of these – renaming a column, dropping a table, changing a job’s arguments – has to be split across multiple releases. That’s the whole discipline. Everything else is technique.

Deploy strategies: rolling, blue-green, canary

• Rolling deploys replace instances a few at a time. Default on Kubernetes, ECS and most PaaS platforms. Cheap, but old and new run together for the whole rollout.
• Blue-green runs two full environments and switches the router from blue to green at once. Rollback is switching back – as long as the database change was backward compatible, because both environments share the database.
• Canary sends a small percentage of traffic to the new version, watches error rates and latency, then widens. Best for catching problems that only show under real traffic.
• Feature flags separate deploy from release: code ships dark and is turned on per tenant or percentage. This is how you make risky changes boring.

Whichever you use, three details decide whether it’s actually zero downtime: health checks that verify the app can serve (database reachable, migrations compatible), not just that the process started; connection draining so in-flight requests finish before an instance stops; and graceful worker shutdown so background jobs finish or get re-queued rather than killed mid-way.

Expand and contract: the core zero downtime migration technique

Also called parallel change. Take renaming users.name to users.full_name, which done naively breaks every running instance of the old code. Done safely, it’s a sequence of deploys:

1. Expand: add full_name as a nullable column. Old code ignores it.
2. Dual write: deploy code that writes both name and full_name, still reading name.
3. Backfill: copy name into full_name for existing rows, in batches.
4. Switch reads: deploy code that reads full_name (still writing both, so a rollback is safe).
5. Stop writing the old column: deploy code that only uses full_name. Tell the ORM to ignore name – ignored_columns in Rails, or dropping it from the Django model while keeping the column via SeparateDatabaseAndState.
6. Contract: drop the name column in a later release.

Six steps for a rename feels like a lot. It’s also why you should rename columns rarely. The same pattern handles splitting a table, changing a column’s type, moving data between modules, or retrofitting a tenant_id onto every table.

Postgres operations that take locks you don’t expect

Most schema changes need an ACCESS EXCLUSIVE lock, at least briefly. The problem isn’t the lock itself; it’s that the migration waits behind a long-running query, and every other query then queues behind the migration. A ten-second report can turn a millisecond ALTER into a full outage. The defenses:

• Always set lock_timeout (a few seconds) for migrations, and retry on failure. A failed migration is better than a queued one.
• Create indexes with CREATE INDEX CONCURRENTLY. It can’t run inside a transaction, so disable the migration transaction for that step (disable_ddl_transaction! in Rails, atomic = False in Django), and check for an INVALID index if it fails.
• Add foreign keys and check constraints as NOT VALID, then VALIDATE CONSTRAINT in a separate step. Validation takes a weaker lock and doesn’t block writes.
• Adding NOT NULL to an existing column scans the table under an exclusive lock. On Postgres 12 and later, add a CHECK (column IS NOT NULL) NOT VALID constraint, validate it, then SET NOT NULL, which uses the validated constraint to skip the scan.
• Adding a column with a default is a fast metadata change on Postgres 11 and later for non-volatile defaults. Older versions rewrite the table.
• Changing a column type usually rewrites the table. Use expand and contract instead.
• Lint migrations in CI. strong_migrations for Rails and django-pg-zero-downtime-migrations or similar tools for Django catch most of these before review.

MySQL has its own set of rules (online DDL algorithms, gh-ost and pt-online-schema-change for large tables), but the principles are the same: avoid table rewrites, keep locks short, and never let a migration wait indefinitely.

Backfills that don’t hurt production

• Run them as background jobs, not inside the migration. A migration that updates ten million rows holds locks and blocks the deploy.
• Batch by primary key range – a few thousand rows per batch – and commit each batch. Avoid OFFSET pagination; it gets slower with every page.
• Throttle with a short sleep between batches, and watch replication lag if you have replicas.
• Make it idempotent and resumable. Only update rows that still need it (WHERE full_name IS NULL), so a restart picks up where it left off.
• Verify before contracting. Count rows that still don’t match before you deploy the step that depends on the backfill being complete.

The parts people forget

• Background jobs. A job enqueued by old code with old arguments will be processed by new code. Keep job signatures backward compatible for at least one release, and don’t delete a job class while instances might still be in the queue.
• Caches. Serialized objects in Redis from the old version may not deserialize in the new one. Version cache keys when the shape changes – see caching strategy.
• API clients and mobile apps. Old app versions stay in the wild for months. Additive API changes only, and version anything breaking.
• Static assets. Fingerprint them and keep the previous release’s assets available, or users with an open tab get 404s on the next click.
• Rollback. Practice it. A rollback that’s never been tested is a second outage waiting to happen.

Where to start

If deploys currently need a maintenance window, start with three changes: lock_timeout on every migration, a migration linter in CI, and a written rule that destructive changes ship in a separate release from the code that stops using them. Those alone remove most deploy-related downtime.

Deploy safety also shows up directly in your DORA metrics – change fail rate and failed deployment recovery time – so it’s worth tracking before and after. If you want hands-on help getting there, that’s part of my DevOps consulting work, and it’s a common first step in Heroku migration projects where the new platform makes these patterns easier.

Related

• DevOps consulting for teams without a platform team
• Picking a Heroku alternative and moving to it without downtime
• DORA metrics for small teams: what to measure and how
• PostgreSQL performance tuning when the database is the bottleneck
• Choosing a caching strategy that won’t bite you later
• Legacy modernization services that keep the business running