Never Stop the Old Container First

Every deployment guide starts with the happy path. This one starts with the gap.

You have a container serving production. You push a release. Compose stops the old container, starts a new one, and waits for it to report healthy. Somewhere in the middle of that sentence is a window - thirty seconds, sixty on a bad day - where the reverse proxy has nothing to route to and every request in flight comes back a 502.

For a marketing site that window is invisible. For anything transactional it is not. If the deploy window catches a checkout, a payment callback, or a desktop client flushing its offline queue, the failure is not cosmetic - it is a retry storm and a support ticket.

This is the pipeline I use to close that window: GitHub Actions for the quality gate, rsync for delivery, Docker Compose for the containers, and Traefik as the thing that makes cutover free. No Kubernetes, no second load balancer, one VPS.

The gap nobody budgets for

The naive deploy is one line, and it is one line because Compose is doing something reasonable:

docker compose up -d --no-deps api

Compose sees a changed image, so it recreates the container. Recreate means stop, then start. The proxy watches the Docker socket, sees the container disappear, and pulls it from the routing pool. There is nothing else in the pool.

Timeline of a naive recreate
────────────────────────────────────────────────────────

  Old container  ████████████████████
                                     │ stop
                                     ▼
  Gap                                ··············
                                                   │ healthy
                                                   ▼
  New container                                    ████████████████████

  Client sees:   200 200 200 ...  502 502 502 ...  200 200 200

Two common workarounds do not work here.

docker compose up --scale api=2 is the textbook answer, and it fails the moment you set a fixed container_name. Compose cannot start two containers with the same name, and dropping the fixed name costs you every ops habit built on docker logs api instead of docker logs project-api-1.

A longer healthcheck interval does not help either. The gap is not a health-detection problem, it is an ordering problem. The old container is already gone.

The fix is to invert the order. Never stop the old container until something else is already healthy and already registered with the proxy.

What zero downtime actually means

Worth being precise, because the phrase gets used loosely. My working definition:

For the entire duration of a deploy, every client request receives a successful response from some version of the application.

Note what that does not promise. It does not promise that all clients see the same version at the same instant - during the overlap window, two requests from the same browser can land on different builds. It does not promise atomicity across services. And it does not extend to your background workers or your database, which have their own answers.

That looseness is the price. In exchange you get a deploy you can run at 2pm on a Tuesday instead of scheduling for 3am.

The shape of the system

Two repositories - an API and a web front end - each with its own workflow, both deploying to the same VPS and attaching to an external Docker network the proxy owns.

                    ┌──────────────────────────────────┐
                    │       Internet / clients         │
                    └────────────────┬─────────────────┘
                                     │ :80 / :443
                                     ▼
                    ┌──────────────────────────────────┐
                    │  Traefik                         │
                    │  TLS, routing, middleware, LB    │
                    │  network: edge                   │
                    └──────┬────────────────┬──────────┘
                           │                │
              PathPrefix(/api)              │ everything else
                           ▼                ▼
                   ┌──────────────┐  ┌──────────────┐
                   │ app-api      │  │ app-web      │
                   │ :8000        │  │ :80 (nginx)  │
                   └──────┬───────┘  └──────────────┘
                          │
             ┌────────────┼────────────┐
             ▼            ▼            ▼
        PostgreSQL    PgBouncer      Redis
        (+ background workers, recreated after API cutover)

Each layer has exactly one job in the zero-downtime story:

Layer	Job
GitHub Actions	Gate on lint, types and tests, then deliver and invoke
rsync over SSH	Put the CI checkout on the server - no git remote on the VPS
Docker Compose	Build images, run the candidate alongside the live container
Traefik	Discover containers by label; balance across both during overlap
Healthchecks	Gate promotion - the only thing standing between a bad build and users
`stop_grace_period`	Let in-flight requests finish when the old container is drained

Secrets live only on the server. The .env files are never in git and never in the rsync payload - CI has no way to overwrite them, which is deliberate. A deploy pipeline that can rewrite production credentials is a deploy pipeline that will, once.

Blue-green without Kubernetes

Blue-green usually implies two full environments and a switch between them. On a single VPS that is unaffordable, and it is also more than the problem needs. What I actually run is blue-green scoped to one service, for about ninety seconds:


Live	The container serving production right now, under the canonical name
Candidate	A second container from this deploy’s image, under a `-candidate` name

The candidate comes up beside the live container. Both serve. Then the live one goes away and the canonical name is recreated on the new image. The candidate is scaffolding - it exists for the duration of the swap and is removed at the end.

Phase A - build            Phase B - overlap          Phase C - promote
────────────────           ──────────────────         ─────────────────

 Traefik ──► live ●         Traefik ──► live ●         Traefik ──► live ● (new image)
                            Traefik ──► candidate ●
 candidate: building        (both healthy)             candidate: removed

The candidate is defined in an overlay file, docker-compose.deploy.yml, layered on top of the production Compose file. It extends the live service and changes four things:

services:
  api_candidate:
    extends: { file: docker-compose.prod.yml, service: api }
    container_name: app-api-candidate
    image: app-api:${IMAGE_TAG}
    restart: 'no'
    labels:
      - traefik.http.routers.api-candidate.rule=PathPrefix(`/api`)
      - traefik.http.routers.api-candidate.service=api

restart: 'no' matters. A candidate that crash-loops should stay down so the healthcheck gate catches it, not restart until the timeout expires and confuses the diagnosis.

One service name, two containers

This is the part that makes the whole thing work, and it is three lines of labels.

Traefik requires router names to be unique - two containers cannot both claim api. But routers and services are separate concepts, and nothing stops two distinct routers from pointing at the same Traefik service:

# on the live container
traefik.http.routers.api.rule=PathPrefix(`/api`)
traefik.http.routers.api.service=api

# on the candidate
traefik.http.routers.api-candidate.rule=PathPrefix(`/api`)
traefik.http.routers.api-candidate.service=api      # <- same service

traefik.http.services.api.loadbalancer.server.port=8000

Traefik resolves both containers into one service’s server pool and round-robins across them. You get load balancing during the overlap for free, from the proxy that is already terminating TLS.

During overlap

  Request ──► Traefik service "api"
                    │
          ┌─────────┴─────────┐
          ▼                   ▼
      app-api            app-api-candidate
      (old image)        (IMAGE_TAG = git SHA)

When the live container is stopped, Traefik gets the Docker event, drops that server from the pool, and sends 100% of traffic to a candidate that has already been warm and serving for a minute. There is no cold start at the cutover instant - which is the other half of why this is smooth, and the half that a DNS or symlink switch does not give you.

The pipeline, end to end

Each repo has one workflow on the production branch.

Event	What runs
PR into `production`	CI only - lint, types, tests, build
Push to `production`	CI, then deploy
`workflow_dispatch`	Manual re-run, which is also the rollback path

push to production
        │
        ▼
┌─────────────────────────────────────┐
│ job: ci                             │
│  API: lint, format, types,          │
│       migration drift check         │
│  Web: eslint, tsc, unit, build      │
└──────────────┬──────────────────────┘
               │ needs: ci
               ▼
┌─────────────────────────────────────┐
│ job: deploy                         │
│ concurrency: production-deploy      │
│   cancel-in-progress: false         │
│                                     │
│  1. checkout                        │
│  2. ssh-agent, load deploy key      │
│  3. rsync  →  DEPLOY_PATH           │
│  4. ssh: IMAGE_TAG=$GITHUB_SHA \    │
│          ./scripts/deploy/prod.sh   │
└─────────────────────────────────────┘

cancel-in-progress: false is not a detail. The default in most concurrency configs is to cancel the older run, and cancelling a deploy mid-cutover is how you end up with no live container and a half-promoted candidate. Deploys queue. They never race.

Why rsync and not `git pull`

The obvious pattern is to SSH in and git pull. I do not, for three reasons.

Actions has already checked the code out - pulling again means putting a deploy key with repo access on the server, which is a credential I would rather not have sitting there. It also makes the server’s state depend on a network call to GitHub at deploy time, adding a failure mode to a step that has enough of them. And a git working tree on a production box invites someone to fix a bug in place at 1am, which then vanishes on the next deploy with no record it existed.

rsync -az --delete \
  --exclude '.git/' \
  --exclude '.env*' \
  --exclude 'node_modules/' \
  --exclude 'media/' \
  ./ "${VPS_USER}@${VPS_HOST}:${DEPLOY_PATH}/"

ssh "${VPS_USER}@${VPS_HOST}" \
  "cd '${DEPLOY_PATH}' && IMAGE_TAG='${GITHUB_SHA}' ./scripts/deploy/production.sh"

Consequences worth writing on the runbook: the deploy directory is rsync-managed, not a git checkout. --delete means anything not in the repo is removed, so runtime state - uploads, .env, caches - must be excluded or mounted from outside the tree. And since there is no git history on the server, the deploy script writes the shipped SHA to .deploy-revision, which is the only way to answer “what is actually running right now”.

The tag is the contract

IMAGE_TAG = GITHUB_SHA

The candidate is built with that tag and the promotion step uses the same tag rather than rebuilding. This sounds pedantic and is not: if promote rebuilds, you have health-checked one artefact and shipped a different one. Same idea as building a release artefact once and promoting it rather than rebuilding per target.

The API deploy, phase by phase

Phase	Action	Live API up?
0	Clean the deploy tree, write `.deploy-revision`	yes
1	`compose build api`	yes
2	Run migrations as a one-off container	yes
3	Start the candidate	yes
4	Wait for candidate Docker health, ~300s timeout	yes
5	Smoke test `https://example.com/api/health/` through the proxy	yes
6	`docker stop -t 30` the live container	candidate serves 100%
7	`compose up -d --no-deps api` on the new image	yes
8	Wait for the canonical container to report healthy	yes
9	Remove the candidate	yes
10	Recreate background workers	yes
11	Final smoke test	yes

Time ─────────────────────────────────────────────────────────────►

Live        ████████████████████████████████████░░ stop
                                                 │
Candidate        ░░ build ░░ ████ health ██████████████░░ remove
                                  │ smoke      │ promote
Migrate      ████ one-off         │            │
Workers                                          ░░ recreate ░░

Client traffic ───────────────────────────────────────────────────►

Two phases carry most of the weight.

Phase 4 is the gate. Docker’s own health status is the signal, which means the healthcheck in the Dockerfile is load-bearing infrastructure, not decoration. A healthcheck that only checks the process is alive will cheerfully promote a container that cannot reach the database. Mine hits a readiness endpoint that touches the DB and the cache.

Phase 5 is the second gate, and it exists because Phase 4 can pass while the request still fails. The container being healthy on its own network is not the same as the request path working - labels can be wrong, the container can be on the wrong network, TLS can be misconfigured. Phase 5 goes out through the public hostname and comes back in through Traefik, which is the path a user takes.

Only after both gates pass does anything stop. Everything before Phase 6 is reversible by deleting a container nobody was told about.

The web deploy

Same shape, fewer moving parts - no migrations, no workers.

Phase	Action	Live UI up?
1	Build the image at `IMAGE_TAG`	yes
2	Start the candidate	yes
3	Wait for health	yes
4	In-container smoke: `wget -S /`, expect 200, not a redirect	yes
5	`docker stop -t 30` live	candidate serves 100%
6	Promote canonical	yes
7	Wait for health, remove candidate	yes
8	Brief pause for label rediscovery, then external smoke	yes

The front end’s smoke test is fussier about content than the API’s. A static server will happily return 200 for a build that routed / to the sign-in page, or shipped an asset manifest referencing chunks that are not in the image. Checking the status code alone passes both. So the in-container check asserts the response is a 200 and not a redirect, before anything is drained.

Routing stays unambiguous by keeping the API rule specific and the web rule broad:

PathPrefix(`/api`)  → service "api"    (higher priority)
Host(`example.com`) → service "web"    (everything else)

The web front end never gets a chance to swallow /api on a bad day.

Migrating while the old code still serves

Phase 2 runs migrations while the old version is still the only thing serving traffic. That is a deliberate choice with a hard constraint attached:

Between Phase 2 and Phase 6, the old code is running against the new schema. If that combination does not work, you do not have a zero-downtime deploy - you have a broken one with extra steps.

Which sorts every schema change into two piles:

Safe during overlap	Needs a window, or a second release
Add a nullable column	Drop a column the old code still selects
Add a table	Rename a column in one release
Add an index concurrently	`NOT NULL` without a default while old writers omit it
Backfill in a separate pass	Destructive rewrites only the new code understands

The discipline is expand, then contract. Release one adds the new column and writes to both. Release two reads from the new column. Release three drops the old one. Three deploys to rename a field feels absurd until the alternative is a maintenance window, and the whole point of this pipeline is not needing one.

The migration step is also the last fully safe abort point. It runs before the candidate exists, so a failure there leaves the live container untouched and the deploy simply stops. On multi-tenant setups where migrations run per schema, this phase is the long pole - and worth thinking about early, because a per-tenant migration that takes minutes changes the shape of the overlap window.

Failing closed, and rolling back

The deploy script traps errors, and the response depends on one boolean: has the live container been stopped yet?

                  deploy error
                       │
         ┌─────────────┴─────────────┐
         ▼                           ▼
   DRAINED = 0                  DRAINED = 1
   (before phase 6)             (after live was stopped)
         │                           │
         ▼                           ▼
   remove candidate            do NOT clean up
   live never stopped          log loudly, exit non-zero
   users unaffected            operator checks canonical

Before drain, cleanup is safe and automatic - the candidate is removed and nobody outside the pipeline ever knew it existed. After drain, the script deliberately stops helping. An automated cleanup at that point could remove the only container serving traffic. Loud failure beats clever recovery when the blast radius is production.

Failure	Outcome
Build fails	Live untouched
Migration fails	Live untouched, deploy aborts
Candidate never healthy	Candidate removed, live untouched
Smoke fails before drain	Candidate removed, live untouched
Failure after drain	Rare, and the one case that pages a human

Rollback uses the same machinery, which is the point of building it this way. Point production at the last known-good commit and re-run the workflow - it blue-green deploys that SHA exactly like any other release, with the same gates. There is no separate, less-tested rollback path to discover is broken at the worst possible moment.

What you cannot do is git reset on the server. The deploy directory is not a git repo, and treating it like one is how you end up with files from two releases. Roll back through the pipeline.

If the deploy failed before drain, there is usually nothing to roll back at all. Live never changed.

Proving it actually held

Claiming zero downtime and measuring it are different activities. From any machine with network access, during a real deploy:

while curl -sf https://example.com/api/health/ >/dev/null; do
  printf '.'
  sleep 1
done
echo "FAILED at $(date -Is)"

The loop exits on the first non-success. A clean cutover keeps it running for the entire deploy. Run it in a second terminal while you watch the Actions log - it takes ten seconds to set up and it is the only evidence that matters.

Afterwards, on the server:

cd "$DEPLOY_PATH"
cat .deploy-revision          # matches the Actions run SHA?
docker ps                     # canonical container healthy, no orphan candidate
curl -sf https://example.com/api/health/

The one drill worth running deliberately: break a candidate’s healthcheck on purpose in a staging copy and run the deploy. Confirm it aborts at Phase 4 and the live container is still serving. A safety mechanism you have never seen fire is a safety mechanism you are guessing about.

What this does not cover

Being honest about the boundary is more useful than the claim.

Component	During an app deploy	Why
API	Up throughout	Blue-green plus proxy load balancing
Web UI	Up throughout	Same
Background workers	Brief restart	Recreated after cutover - queued jobs delay, they do not vanish
PostgreSQL, Redis, PgBouncer	Untouched	`--no-deps` keeps the data plane out of the app recreate
The proxy itself	Separate lifecycle	Traefik is not rebuilt on every app push
Desktop or mobile clients	Separate pipeline	Different release cadence entirely

Workers are the honest gap. A brief worker restart is acceptable because a durable queue makes it a latency event rather than a correctness one - which only holds if the jobs are idempotent and the queue survives the restart. If that is not true of your workers, fix that before you worry about the API’s 502s. It is the larger problem.

The other gap is cross-service atomicity. Two repos deploy independently, so there is no instant where the front end and API switch together. If a release needs both, the API ships the new field first in a backward-compatible release, and the UI follows once it is live. That is release discipline, not infrastructure - no pipeline will save you from a breaking API change shipped in lockstep.

The operator checklist

Once per server

Docker and the Compose plugin installed, deploy user in the docker group
The external proxy network exists and the proxy is running on it
Deploy directories created, .env files present and not in git
Initial stack brought up once by hand, so the deploy script only ever swaps

Once per repo

Secrets: host, user, SSH key, deploy path
Workflow on the production branch, with concurrency set to queue rather than cancel
Branch protection so the only way in is a reviewed merge

Every release

CI green before merge
Migrations reviewed against the expand-only table above
Watch the phase log through cutover
Confirm .deploy-revision and the external health URL
Optional, and cheap: run the curl loop during the deploy

When something looks wrong

Symptom	First thing to check
SSH fails in Actions	Host, key, firewall, user in the docker group
Candidate never healthy	`docker logs app-*-candidate` - almost always config or DB reachability
502 after cutover	Container health, proxy labels, is it on the right Docker network
Two versions serving forever	Orphaned candidate from a failed run - `docker ps` will show it
Mystery files on the server	Something was edited in place; rsync `--delete` will remove it next deploy

The sequence is the guarantee

There is no flag for this. Strip the pipeline down and what is left is an ordering rule and two gates:

  gate on quality in CI
        →
  deliver code, build the new image     (live still serving)
        →
  migrate compatibly                    (live still serving)
        →
  start candidate → health → smoke      (live still serving)
        →
  drain live → promote → clean up
        →
  smoke again

Every step before the drain is reversible. The drain only happens after two independent checks agree the new version works, one of them through the same path a user takes. And the data services stay out of the recreate entirely.

The rest is discipline: migrations that respect the overlap window, workers whose restart is a latency event rather than a data-loss event, and a rollback path that is the deploy path so it cannot rot. Get those right and deploying stops being an event you schedule around, which is worth more than the uptime figure.