The instinct when a platform slows down is to make the server bigger. It works, briefly, and it teaches you nothing - which means you will be doing it again next quarter.

FitsSort went from a few hundred to over ten thousand daily active users without a proportional increase in hardware. Four things did the work, in order of how much they mattered.

1. Connection exhaustion, which is not load

The first symptom looked like an outage: FATAL: sorry, too many clients already. CPU was comfortable. Memory was comfortable. The database was refusing connections.

Django opens a connection per request per worker and holds it for the request’s duration. With CONN_MAX_AGE set, it holds it between requests too. Multiply by gunicorn workers, multiply by application instances, and you cross Postgres’s max_connections while the database is doing almost nothing.

The tempting fix is raising max_connections. Don’t. Every Postgres connection is a backend process with its own memory, and a thousand mostly-idle connections cost real RAM and real scheduler pressure while making the eventual failure worse.

The actual fix is PgBouncer in transaction pooling mode. Applications connect to PgBouncer; PgBouncer maintains a small pool of real backends and hands one to a transaction only while it is running. Hundreds of application connections multiplex onto a few dozen real ones, because at any instant most “connections” are idle.

Transaction pooling has rules you must respect:

  • No session state. Session-level SET, advisory locks, LISTEN/NOTIFY and WITH HOLD cursors do not survive, because the next statement may land on a different backend. If you use search_path for multi-tenancy, it has to be transaction-local.
  • CONN_MAX_AGE = 0 in Django. Let PgBouncer own pooling. Two poolers stacked on each other is a way to have the problems of both.
  • Prepared statements need care. Disable Django’s server-side prepared statements, or adopt PgBouncer’s prepared-statement support deliberately rather than by accident.

This single change removed the failure mode entirely, and cost one small process.

2. N+1 queries, found rather than guessed

Every ORM makes N+1 queries easy to write and invisible to read. A member list page that looked fine at fifty members issued four hundred queries at four hundred.

The fix is select_related for forward foreign keys and prefetch_related for reverse and many-to-many relations. That part is well known. The part that actually mattered was not guessing:

from django.db import connection
from django.test.utils import CaptureQueriesContext

with CaptureQueriesContext(connection) as ctx:
    response = client.get("/api/members/?branch=3")

assert len(ctx.captured_queries) < 12, f"{len(ctx.captured_queries)} queries"

An assertion on query count, in the test suite, per endpoint. It turns a performance regression into a failing test - which is the only mechanism I have found that keeps N+1s from coming back the moment someone adds a nested field to a serializer.

3. Indexes that match the queries you actually run

Attendance is the highest-volume table in the product, and every report filters it the same way: one branch, one date range.

A single-column index on branch_id finds all attendance for a branch - thousands of rows - and then filters them by date in memory. The index that matches the query is composite, and column order is not arbitrary:

CREATE INDEX attendance_branch_time
  ON attendance (branch_id, checked_in_at DESC);

Equality columns first, range columns second. Postgres can seek to branch_id = 3 and then walk the date range in index order. Reverse the order and it cannot.

Add DESC when your queries are “most recent first,” which for attendance is all of them - it lets the planner satisfy the ORDER BY from the index and skip the sort entirely.

The other habit worth having: EXPLAIN (ANALYZE, BUFFERS) on the query as it is actually issued, with realistic parameters. A plan over a hundred rows tells you nothing about a plan over a million, and the planner switches strategies between them.

4. Stop recomputing what has not changed

The gym owner’s dashboard aggregated today’s check-ins, active memberships, expiring plans and revenue. Four aggregate queries over large tables, on every page load, for every owner, all day.

None of that data needs to be a second old. Caching it in Redis with a short TTL, invalidated on the writes that actually affect it, removed the largest single source of database load in the system.

The subtlety is stampede protection. When a hot key expires, every concurrent request misses simultaneously and all of them recompute. Under load that is worse than no cache. A short lock around recomputation - first requester computes, everyone else waits briefly and reads the fresh value - is a few lines and prevents a thundering herd on your most expensive query.

And the related change: WebSocket instead of polling. The live check-in feed used to poll every five seconds per open dashboard. Pushing on the event that already existed removed a constant, useless query floor that scaled with the number of screens open in the building.

Deploys that don’t count as downtime

None of the above helps if every release is a gap in service. Rolling container swaps behind Traefik with health gating: the new container starts, its health check passes, traffic shifts, the old one drains. A build that fails its health check never receives a request. Rollback is deploying the previous tag.

The constraint this puts on you is that both versions run simultaneously during the swap. Every migration must be compatible with the code on either side of it. Additive changes deploy freely; destructive ones become a three-step dance across releases - add, migrate, remove. It is slower to write, and it is the reason deploys are boring.

The order matters

Notice that hardware never appears. Connection exhaustion looked like load and was not. N+1s looked like slow queries and were a count problem. Missing indexes looked like table size and were a shape problem. The dashboard looked like traffic and was recomputation.

Scaling up would have masked all four for a quarter, at a permanent cost, and I would have learned nothing about the system I run.