If your pull requests sit for fifteen minutes waiting on a green check, your team is paying for it many times a day. Engineers switch context, batch up changes to avoid the wait, and start treating CI as something to route around instead of something that protects them. Slow CI is rarely one big problem. It is usually five or six small ones that add up: dependencies downloaded from scratch on every run, a test suite running on a single core, the backend tests running when only a React component changed, Docker images rebuilt from the first layer, and nobody measuring any of it.

This post walks through the fixes we reach for first on Django and React projects that use GitHub Actions. Every example is plain YAML you can adapt. None of it needs a paid CI product.

Start by measuring, not guessing

Before you change anything, find out where the time goes. Open a recent workflow run and look at the step timings. In most Django and React repositories we look at, the time splits into a few buckets:

• Installing Python and Node dependencies
• Building assets or Docker images
• Running the test suites
• Waiting for a runner to pick up the job

Write down the median duration of your main workflow over the last few weeks. The GitHub CLI makes this easy:

gh run list --workflow ci.yml --branch main --limit 50 \
  --json databaseId,conclusion,createdAt,updatedAt \
  > runs.json

The difference between createdAt and updatedAt gives you a rough wall-clock time per run (it includes queue time and any re-runs, so treat it as an approximation). Keep this number. Every change below should move it, and if a change does not, revert it. CI configuration has a habit of collecting clever tricks that nobody can explain a year later.

1. Cache dependencies properly

The cheapest win is to stop downloading the same packages on every run. The official setup-python and setup-node actions have caching built in, keyed on your lock files.

name: ci

on:
  pull_request:
  push:
    branches: [main]

jobs:
  backend:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
          cache-dependency-path: requirements/*.txt

      - run: pip install -r requirements/dev.txt

  frontend:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: frontend
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
          cache-dependency-path: frontend/package-lock.json

      - run: npm ci

A few details matter here:

• Cache keys come from lock files. If your Python dependencies are not pinned, the cache key never changes, and you will get stale or inconsistent installs. Pin with a lock file (pip-tools, uv or Poetry) so the key changes only when dependencies actually change.
• npm ci, not npm install. npm ci installs exactly what the lock file says and fails if package.json and the lock file disagree. That is what you want in CI.
• The built-in cache stores the package manager’s download cache, not the installed environment. That is usually enough. If installs are still slow because of packages that compile native code, you can cache the virtual environment itself with actions/cache, keyed on the Python version and the lock file hash.

GitHub documents how cache keys, restore keys and size limits work in its guide to caching dependencies to speed up workflows. It is worth reading once, mainly so you understand when caches get evicted and why a branch can read the cache from main but not the other way around.

2. Run Django tests in parallel

Most GitHub-hosted Linux runners have more than one CPU core, and a default pytest run uses one. If you use pytest with pytest-django, adding pytest-xdist is often the single biggest speedup for the backend job.

pip install pytest-xdist
pytest -n auto

-n auto starts one worker per available CPU. pytest-django handles the database side for you: each worker gets its own test database (with a suffix such as _gw0, _gw1), so workers do not trample each other's data.

What usually breaks when you turn this on:

• Tests that depend on order. A test that only passes because an earlier test created some data will fail randomly under xdist. These are real bugs in the test suite, and parallelism just exposes them. Fix them rather than turning parallelism off.
• Shared external state. Tests that write to the same file path, the same Redis key prefix or the same S3 bucket prefix will collide. Use temporary directories (tmp_path) and per-worker prefixes.
• Database service limits. If you run PostgreSQL as a service container, each worker opens its own connections. The defaults are normally fine for a handful of workers, but keep an eye on max_connections if you also run Celery tasks eagerly inside tests.

Here is a backend job with a real PostgreSQL service and parallel tests:

backend:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env:
          POSTGRES_USER: app
          POSTGRES_PASSWORD: app
          POSTGRES_DB: app
        ports:
          - 5432:5432
        options: >-
          --health-cmd "pg_isready -U app"
          --health-interval 5s
          --health-timeout 5s
          --health-retries 10
    env:
      DATABASE_URL: postgres://app:app@localhost:5432/app
      DJANGO_SETTINGS_MODULE: config.settings.test
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
          cache-dependency-path: requirements/*.txt
      - run: pip install -r requirements/dev.txt
      - run: pytest -n auto --durations=20

The --durations=20 flag prints the twenty slowest tests at the end of each run. That list is your to-do list. In our experience a small number of slow tests (often ones that hit real network calls, sleep, or build large fixtures) account for a big share of the suite's time.

3. Split tests across several jobs

Parallelism inside one runner has a ceiling. When the suite is large enough, split it across several runners with a matrix. For Python, the pytest-split plugin divides tests into groups using recorded durations, so each group takes roughly the same time:

backend-tests:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        group: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip
          cache-dependency-path: requirements/*.txt
      - run: pip install -r requirements/dev.txt
      - run: pytest --splits 4 --group ${{ matrix.group }} -n auto

You generate the durations file locally (or in a scheduled job) with pytest --store-durations and commit it. Without it, the plugin still splits, just less evenly.

On the React side, both Jest (version 28 and later) and Vitest support sharding out of the box:

frontend-tests:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shard: [1, 2, 3]
    defaults:
      run:
        working-directory: frontend
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: npm
          cache-dependency-path: frontend/package-lock.json
      - run: npm ci
      - run: npx vitest run --shard=${{ matrix.shard }}/3

Swap npx vitest run for npx jest if you use Jest; the --shard=1/3 syntax is the same.

A word of caution: more jobs means more setup overhead. Each shard checks out code and installs dependencies again. Splitting a three-minute suite into four shards can make the total slower. Split only when a single job is clearly the bottleneck, and check the measured wall-clock time afterwards.

4. Skip work that has not changed

In a repository that holds both a Django backend and a React frontend, many pull requests touch only one side. Running everything on every change is the easiest waste to remove.

GitHub Actions supports paths filters on triggers, but they apply to the whole workflow, and a workflow that never runs cannot report a status. That causes trouble with required checks. A more flexible pattern is to run a small "changes" job first and let the other jobs depend on its outputs. The dorny/paths-filter action is a common choice:

jobs:
  changes:
    runs-on: ubuntu-latest
    outputs:
      backend: ${{ steps.filter.outputs.backend }}
      frontend: ${{ steps.filter.outputs.frontend }}
    steps:
      - uses: actions/checkout@v4
      - uses: dorny/paths-filter@v3
        id: filter
        with:
          filters: |
            backend:
              - 'backend/**'
              - 'requirements/**'
              - '.github/workflows/**'
            frontend:
              - 'frontend/**'
              - '.github/workflows/**'

  backend:
    needs: changes
    if: needs.changes.outputs.backend == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # ... backend steps from above

  frontend:
    needs: changes
    if: needs.changes.outputs.frontend == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # ... frontend steps from above

Two rules keep this safe:

• Include the workflow files themselves in every filter. If someone edits the CI configuration, everything should run.
• Always run the full suite on main. Path filters are an optimisation for pull requests. Add a condition such as if: github.ref == 'refs/heads/main' || needs.changes.outputs.backend == 'true' so merges get the complete check.

If you use branch protection with required checks, add a final “all green” job that needs every other job and passes when they either succeeded or were skipped. Make that the single required check, so skipped jobs do not block merges.

One more cheap win in the same spirit: cancel outdated runs when someone pushes again to the same pull request.

concurrency:
  group: ci-${{ github.ref }}
  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}

This stops three stale runs from queuing behind the one you actually care about.

5. Cache Docker layers

If your pipeline builds a Docker image for Django (to run integration tests or to push to a registry), the build is often the slowest step. Docker BuildKit can store its layer cache in the GitHub Actions cache, so unchanged layers are reused across runs:

image:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: false
          tags: app:ci
          cache-from: type=gha
          cache-to: type=gha,mode=max

The details are in Docker’s documentation for the GitHub Actions cache backend. The cache only helps if your Dockerfile is ordered so that the expensive, rarely changing steps come first:

FROM python:3.12-slim

WORKDIR /app

# Dependencies change rarely: install them in their own layer
COPY requirements/base.txt requirements/base.txt
RUN pip install --no-cache-dir -r requirements/base.txt

# Application code changes on every commit: copy it last
COPY . .

CMD ["gunicorn", "config.wsgi:application", "--bind", "0.0.0.0:8000"]

If you write COPY . . before installing requirements, every code change invalidates the dependency layer, and no cache backend can save you. Add a .dockerignore that excludes node_modules, .git, local media and virtual environments as well, so unrelated files do not bust the cache or bloat the build context.

6. Keep the slow checks, move them

Some checks are slow and valuable: end-to-end browser tests, full migrations against a production-sized schema, dependency audits. Instead of deleting them, decide where they belong.

• On every pull request: linting, type checks, unit tests, and the tests for whatever changed.
• On merge to main: the full suite, image build and integration tests.
• On a schedule (nightly): long end-to-end runs, dependency and security scans, and regenerating the test durations file.

This is a judgement call, not a rule. Our view is that a pull request check should finish in under ten minutes, because past that point people stop waiting for it. Your team may set a different line, but pick one deliberately.

7. Make the test setup itself cheaper

Once the suite runs in parallel, per-test overhead starts to dominate, and Django has a few settings that cost nothing to change for tests:

# config/settings/test.py
from .base import *  # noqa
 
PASSWORD_HASHERS = ["django.contrib.auth.hashers.MD5PasswordHasher"]
EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"
CELERY_TASK_ALWAYS_EAGER = True
LOGGING_CONFIG = None

The default password hasher is deliberately slow, which is exactly right in production and pure waste in a test suite that creates hundreds of users. A fast hasher is fine for tests because the hashes never leave the test database.

Database creation is the other fixed cost. pytest-django’s --nomigrations flag builds the test schema directly from the models instead of replaying every migration, which can save a minute on projects with a long migration history. If you use it, add a separate, cheap job that runs python manage.py makemigrations --check --dry-run and applies migrations to an empty database, so broken or missing migrations are still caught.

8. Use faster tools where they are drop-in

Some speedups come from swapping a tool rather than restructuring the pipeline. uv installs Python packages from a lock file many times faster than pip, and the official astral-sh/setup-uv action caches its own downloads. Ruff replaces flake8, isort and many pylint checks with a single fast binary, which turns a lint step from tens of seconds into a second or two. On the frontend, Vitest is usually faster to start than Jest for projects already built with Vite.

Treat these as optional. A tool swap is worth it when the step it replaces shows up near the top of your timings, not because it is fashionable. Measure the step before and after, the same way you measured the whole run.

9. Deal with flaky tests before they deal with you

Parallel and sharded runs expose tests that pass or fail depending on timing, ordering or shared state. A flaky test does more damage than a slow one: people learn to press re-run without reading the failure, and eventually a real failure gets re-run into green.

Track flakiness explicitly. When a test fails and then passes on retry with no code change, record it. Quarantine known flaky tests in a separate, non-blocking job with a ticket and an owner, so the main check stays trustworthy while they are fixed. Avoid blanket automatic retries on the main suite; they hide the problem instead of measuring it. The goal is a check that is both fast and believed.

10. Buy bigger runners last, and with numbers

Larger runners with more cores are the easiest speedup to buy, and the easiest to waste money on. They help when the suite is already parallel and CPU-bound: a test run that saturates four cores will finish faster on eight. They do almost nothing for a pipeline that spends its time downloading dependencies, waiting on a single slow test or building an image without a layer cache.

So apply the earlier steps first, then look at the timings again. If the remaining time is spread across tests running in parallel and the runner’s CPU is fully used, try a larger runner for that one job and compare the wall-clock time and the cost per run. Keep it only if both numbers justify it. Self-hosted runners are another option for teams with predictable load, but they bring their own maintenance and security work, especially for public repositories, so treat them as an infrastructure project rather than a quick fix.

A realistic order of work

If you are starting from a single, slow workflow, this is the order we would tackle it in:

1. Measure the median run time and the slowest steps.
2. Turn on dependency caching for pip and npm.
3. Add concurrency with cancel-in-progress for pull requests.
4. Add pytest-xdist and fix the order-dependent tests it exposes.
5. Add path filters so frontend-only changes skip the backend job, and the other way around.
6. Reorder the Dockerfile and enable the BuildKit cache.
7. Only then, if one job is still the bottleneck, split it across a matrix.
8. Measure again and write the before and after numbers in the pull request that changed CI.

Each step is small enough to review in one sitting, and each one can be reverted on its own if it causes trouble.

Final thought

Fast CI is not about a clever trick. It is about refusing to do the same work twice: do not download what you already have, do not test what did not change, do not rebuild layers that are identical, and do not run on one core what can run on four. Measure first, change one thing at a time, and keep the numbers. A pipeline that gives feedback in five minutes changes how a team works more than almost any tool you can buy.

If your team is fighting a slow pipeline on a Django and React stack, we are happy to compare notes in the comments: what was your biggest single win?

Tomorrow, November 12, 2026, PostgreSQL 14 reaches end of life. The official versioning page lists November 12, 2026 as its final release date, and the project’s minor release roadmap has a minor release scheduled for that same day. That makes it the last planned update PostgreSQL 14 will get.

If your Django app still runs on 14, nothing breaks on November 13. The database keeps serving queries the way it did the day before. That’s exactly why this deadline is easy to ignore, and why ignoring it is a mistake.

This post is a practical plan: what EOL actually means, how to decide where to go, three upgrade paths with honest tradeoffs, the Django-specific traps, and how to rehearse and roll back.

What end-of-life actually means

PostgreSQL supports each major version for five years. After that, the community ships one final minor release, and the version is unsupported. In practice:

• No more security fixes. The next scheduled minor release (February 11, 2027) will patch 15, 16, 17, and 18. Not 14.
• No more bug or data-corruption fixes. Minor releases are where those fixes live, and there won’t be any more for 14.
• The ecosystem moves on. Frameworks and drivers drop old versions on their own schedule. For Django, that has already happened.

Here’s the current picture from postgresql.org:

• PostgreSQL 18 (latest minor 18.6 at time of writing): supported until November 14, 2030
• PostgreSQL 17 (17.11): supported until November 8, 2029
• PostgreSQL 16 (16.15): supported until November 9, 2028
• PostgreSQL 15 (15.19): supported until November 11, 2027
• PostgreSQL 14 (14.24): supported until November 12, 2026

PostgreSQL 19 hit its first release candidate on October 15, so it’s either brand new or just around the corner when you read this.

The Django angle: 6.1 already left 14 behind

The Django 6.1 release notes say it plainly: “Dropped support for PostgreSQL 14.” Django 6.1 supports PostgreSQL 15 and higher. Django 6.0 and the 5.2 LTS still support 14.

This isn’t just a documentation note. Django checks the server version when it connects, and on a 14 server Django 6.1 raises NotSupportedError ("PostgreSQL 15 or later is required"). So if your database is still on 14, your Django upgrade is blocked too. The database upgrade has to happen first.

Managed services buy time, at a price

If you’re on a managed service, the clock works differently, but it still runs:

• Amazon RDS for PostgreSQL: the RDS release calendar lists the end of standard support for 14 as February 28, 2027. From March 1, 2027, instances still on 14 move into paid RDS Extended Support.
• Google Cloud SQL: the Cloud SQL version policy shows extended support for 14 starting February 1, 2027, with enrollment and billing applied automatically.

Extended support is a bridge, not a plan. You pay extra to postpone the same work.

The decision checklist

Answer these before you choose a path. Most of them are a single query or a single conversation.

1. Where does the database run? Self-managed VM, Kubernetes, RDS/Aurora, Cloud SQL, Azure? Managed services decide some of your options for you.
2. How big is it, and how much downtime can you take? Five minutes at 3 a.m. and “zero, ever” lead to very different plans.
3. Which extensions are installed? Run SELECT extname, extversion FROM pg_extension; in every database. pgvector, PostGIS, and anything custom must be available for the target version.
4. Do you have streaming replicas? They need a plan too.
5. Is the OS or base image changing at the same time? A new glibc can change collation order underneath your indexes (more on this below).
6. Are data checksums enabled? Run SHOW data_checksums;. This matters for pg_upgrade to 18.
7. Do any roles still use MD5 passwords? PostgreSQL 18 deprecates MD5 password authentication. Check with SELECT rolname FROM pg_authid WHERE rolpassword LIKE 'md5%';.
8. Which Django and driver versions are you on, and where are you heading? If 6.1 is on your roadmap, the target needs to be 15 or newer anyway.

Which target version?

My recommendation for most Django teams: go to 18 if your extensions and platform support it, otherwise 17.

• Not 15. It reaches EOL on November 11, 2027. You’d be doing this again in a year.
• 16 is fine, but it gives you two fewer years of support than 18 for the same migration effort.
• 17 is mature and widely supported by managed services and extension builds.
• 18 gives you the longest runway (to 2030) and real upgrade improvements: pg_upgrade now keeps optimizer statistics, and it adds the --swap transfer mode.
• 19 is fine to test against, but a brand-new major version is not where I’d land a production database under deadline pressure.

You can jump straight from 14 to 18. Skipping versions in between is supported, but read the release notes for every major version you cross. 15, 16, 17, and 18 each have an “incompatibilities” section.

Three upgrade paths

Path 1: pg_upgrade (in place, fast, short downtime)

pg_upgrade builds new system catalogs and reuses your existing data files, so it doesn’t depend much on data size. Always run the new version’s binary, and always run --check first:

# Initialize the new cluster with matching encoding/locale.
# PG18 initdb enables data checksums by default and pg_upgrade requires
# matching settings: keep --no-data-checksums only if SHOW data_checksums
# returns "off" on your 14 cluster.
/usr/lib/postgresql/18/bin/initdb -D /srv/pg18/data \
  --encoding=UTF8 --locale=en_US.UTF-8 --no-data-checksums

# Dry run: checks only, changes nothing.
/usr/lib/postgresql/18/bin/pg_upgrade \
  --old-bindir=/usr/lib/postgresql/14/bin \
  --new-bindir=/usr/lib/postgresql/18/bin \
  --old-datadir=/srv/pg14/data \
  --new-datadir=/srv/pg18/data \
  --link --check

(Debian and Ubuntu users have pg_upgradecluster, which wraps this.)

Transfer modes are the main decision:

• --copy (default): slowest, needs double the disk, and leaves the old cluster untouched.
• --link: uses hard links, so it's fast and uses almost no extra space. Once you start the new cluster, though, the old one is no longer safe to use.
• --clone: as fast as link on file systems with reflink support (for example XFS or Btrfs on Linux, APFS on macOS), and it leaves the old cluster usable.
• --swap (new in 18): moves the data directories, and can be the fastest option when you have many relations. The docs recommend pairing it with --sync-method=fsync. Once the file transfer starts, the old cluster is destructively modified.

Pass the same mode flag together with --check, so pg_upgrade runs the checks specific to that mode.

Tradeoffs: minutes of downtime, and it upgrades the cluster exactly as it is. That includes the OS and its collation behavior, which can be good or bad. Extensions need their new-version shared libraries installed before you run it.

Path 2: logical replication (near-zero downtime, more moving parts)

You stand up a new 18 server, replicate the data into it while 14 keeps serving traffic, then cut over in seconds. On the 14 side:

-- Needs a restart on 14.
ALTER SYSTEM SET wal_level = logical;
-- After restart:
CREATE PUBLICATION upgrade_pub FOR ALL TABLES;

Copy roles and schema with the new version’s tools, then subscribe:

pg_dumpall -h old-db --roles-only | psql -h new-db -d postgres
pg_dump -h old-db -d app --schema-only --no-publications --no-subscriptions \
  | psql -h new-db -d app

-- On the 18 side:
CREATE SUBSCRIPTION upgrade_sub
  CONNECTION 'host=old-db dbname=app user=replicator'
  PUBLICATION upgrade_pub;

The documented logical replication restrictions are where Django teams get caught:

• DDL is not replicated. Freeze migrations from the moment you dump the schema until cutover.
• Sequence data is not replicated. Rows arrive, but sequences on the new side still sit at their start values. You must advance them at cutover, or the first insert will hit a duplicate key.
• Large objects are not replicated. This rarely matters for Django, but check.
• UPDATE and DELETE need a replica identity. Django models have primary keys by default. Tables created by hand or by third-party tools might not.

Tradeoffs: downtime is limited to the cutover, and the old server stays intact as a fallback. In exchange, you have more steps, a migration freeze, and sequence handling to get right.

Path 3: dump and restore (simple, slow, cleanest result)

pg_dumpall -h old-db --globals-only > globals.sql
pg_dump -h old-db -d app -Fd -j 8 -f /backup/app
psql -h new-db -d postgres -f globals.sql
createdb -h new-db -O app_owner app
pg_restore -h new-db -d app -j 8 /backup/app
vacuumdb -h new-db --all --analyze-in-stages

Use the directory format (-Fd) so both dump and restore can run in parallel.

Tradeoffs: downtime grows with data size, so for small and medium databases this is often the simplest correct answer. It also rebuilds every index from scratch, which makes it the safest path when you’re changing OS, CPU architecture, or collation library at the same time.

The Django-specific checklist

Settings and drivers. Django 6.1 requires psycopg 3.1.12+ or psycopg2 2.9.9+, and the Django database docs recommend psycopg 3 and warn that psycopg2 support is likely to be deprecated. If you’re touching the database layer anyway, this is a good moment to switch. Then re-check OPTIONS: connection pooling ("pool") only works with psycopg 3, and if PgBouncer runs in transaction mode, keep DISABLE_SERVER_SIDE_CURSORS = True on that connection.

Migrations. Before the window, confirm nothing is pending, then freeze:

python manage.py makemigrations --check --dry-run
python manage.py migrate --check

Raw SQL and RunSQL. Search for hand-written SQL, especially functions used by expression indexes or materialized views. Since PostgreSQL 17, maintenance operations (ANALYZE, CREATE INDEX, REFRESH MATERIALIZED VIEW, VACUUM, and others) run functions with a safe search_path. Functions that reference non-default schemas need an explicit SET search_path in their definition.

The public schema. Since PostgreSQL 15, new databases no longer give everyone CREATE on public. pg_upgrade and dump/restore preserve your existing permissions, but a freshly created database on 15+ won't have them. That includes a new staging environment. If your app role isn't the database owner, migrate fails with "permission denied for schema public". Fix it by making the app role the owner (createdb -O) or with an explicit GRANT CREATE ON SCHEMA public TO app_role;.

Collations and ICU. If the new server runs a different OS or glibc version, text sort order can change, and B-tree indexes on text columns can silently stop matching. The PostgreSQL wiki page on locale data changes explains the risk. After the upgrade, compare recorded and actual collation versions:

SELECT datname, datcollversion,
       pg_database_collation_actual_version(oid) AS actual
FROM pg_database WHERE datallowconn;

If they differ, REINDEX the affected indexes, then run ALTER DATABASE app REFRESH COLLATION VERSION;. Do the same for any ICU collations you created with Django's CreateCollation operation or db_collation.

Sequences. After a logical-replication cutover (or any partial data copy), let Django generate the fix:

python manage.py sqlsequencereset accounts orders billing \
  | python manage.py dbshell

List every app that has models, including auth, admin, and third-party apps.

Extensions. Install the target-version builds before upgrading, then update them inside each database:

ALTER EXTENSION vector UPDATE;          -- pgvector
SELECT postgis_extensions_upgrade();    -- PostGIS and its companions

For pg_upgrade with PostGIS, make sure the same PostGIS 3.x release is available for both the old and new binaries.

Statistics. Don’t let the first hour of production traffic hit a planner with no statistics. On 18, pg_upgrade carries most statistics over, and the docs recommend:

vacuumdb --all --analyze-in-stages --missing-stats-only
vacuumdb --all --analyze-only

When upgrading to 17 or earlier, or after a dump/restore, run vacuumdb --all --analyze-in-stages before you open the gates.

The rehearsal plan

Do the upgrade at least twice before you do it for real.

Step 1: run your test suite against the target version in CI

With Testcontainers for Python and pytest-django, the server version becomes a single variable:

# conftest.py
import os
import pytest
from testcontainers.postgres import PostgresContainer

PG_IMAGE = os.environ.get("PG_IMAGE", "postgres:18")

@pytest.fixture(scope="session")
def pg_container():
    with PostgresContainer(PG_IMAGE, driver=None) as pg:
        yield pg

@pytest.fixture(scope="session")
def django_db_modify_db_settings(pg_container):
    from django.conf import settings
    settings.DATABASES["default"].update(
        HOST=pg_container.get_container_host_ip(),
        PORT=pg_container.get_exposed_port(5432),
        NAME=pg_container.dbname,
        USER=pg_container.username,
        PASSWORD=pg_container.password,
    )

Run a CI matrix with PG_IMAGE set to postgres:14 and your target. If you use pgvector or PostGIS, use an image that ships the extension for that major version. A one-line guard test that compares connection.pg_version // 10000 with the major version in PG_IMAGE proves the matrix is really testing what you think it is.

Step 2: rehearse on a copy of production

Restore a recent backup to a staging server and run the exact commands from your runbook, including pg_upgrade --check and the post-upgrade scripts it generates. Time every step. The timings you measure are your maintenance window.

Step 3: smoke-test the app against the upgraded copy

Log in, run the critical flows, kick off Celery tasks, and compare row counts on your largest tables. Watch the slow-query log for plans that changed.

Step 4: write down go/no-go criteria

Do it before the night itself, not during it: maximum acceptable window, which smoke tests must pass, and who makes the call.

The rollback plan

Decide your point of no return in advance. It’s the moment the new server accepts writes.

• Before anything else: take a fresh, verified backup or provider snapshot of the 14 cluster.
• pg_upgrade with --copy or --clone: the old cluster is untouched. Rolling back means pointing Django back and starting 14.
• pg_upgrade with --link: if you haven't started the new cluster, the docs explain how to restore the old one (rename global/pg_control.old). Once the new cluster has started, the old one is unsafe, and rollback means restoring from backup.
• --swap: once pg_upgrade reports that the old cluster is no longer safe to start, rollback means restoring from backup.
• Logical replication: 14 stays intact. If you need a rollback that keeps writes made after cutover, set up reverse replication (new to old) before you open traffic, and rehearse that too.

Whatever the path, keep the old DATABASES settings in an environment variable rather than in code. Then rolling back the app is a config change, not a deploy.

Tell people before, during and after

An upgrade that goes perfectly can still feel like an outage if nobody knew it was happening. Announce the window to support, to customer-facing teams and, if you have one, on your status page, a week ahead and again on the day. Say what users will notice: a few minutes of read-only mode, slower pages while statistics rebuild, or nothing at all.

During the window, post short updates in one channel so people stop asking in five. Afterwards, write a brief note: what version you are on now, how long the downtime was compared with the rehearsal, and anything you would change. That note is the first page of the runbook for the next major upgrade, which, with a yearly release cycle, is never as far away as it seems.

Keep that note short enough that people actually read it.

Conclusion

PostgreSQL 14 doesn’t stop working on November 12. It stops getting fixed. For Django teams there’s a second clock as well: Django 6.1 already requires PostgreSQL 15 or newer.

The plan is simple to state. Pick 18 (or 17), choose the path that fits your downtime budget, freeze migrations, handle sequences, extensions, collations, and statistics on purpose, and rehearse until the runbook is boring. A boring upgrade night is the goal.

Most multi-tenant Django applications use shared tables: every tenant’s orders live in one orders_order table, and a tenant_id column says whose each row is. It is the cheapest model to run and the easiest to evolve. If you are still choosing between shared tables, schema-per-tenant and database-per-tenant, our guide to multi-tenant architecture walks through those trade-offs. This post assumes you have chosen shared tables.

The weakness of shared tables is that isolation depends on every query remembering to filter by tenant. Thousands of queries get it right; the one that doesn’t leaks a customer’s data to another customer. PostgreSQL’s row-level security (RLS) adds a second line of defence: once a policy is in place, the database itself refuses to return or modify rows that belong to a different tenant, whatever SQL the application sends.

The gap that application-level filtering leaves

A tenant-aware manager, one that adds filter(tenant=current_tenant) to every queryset, is a good first layer. It still leaves gaps:

• Raw SQL and extra() bypass managers completely. So does any hand-written query in a reporting module.
• Unscoped managers exist for good reasons, in the admin, in management commands, in data migrations, and they are easy to use from the wrong place.
• Aggregations and subqueries can cross tenants without anyone noticing. A Subquery over a related table, or an annotation that joins through a relation, filters only what you told it to filter.
• New developers don’t yet know which manager is safe, and code review won’t catch every case.

Each of these is a single missing WHERE clause. RLS turns that missing clause from a data leak into an empty result.

Use two database roles

RLS does not apply to everyone. PostgreSQL’s documentation is explicit: superusers and roles with the BYPASSRLS attribute always bypass row security, and a table's owner normally does too. If your Django app connects as the role that owns its tables, as most do by default, enabling RLS changes nothing.

So split the roles:

• An owner role owns the tables and runs migrations.
• An app role has SELECT, INSERT, UPDATE and DELETE on those tables and nothing more, and is subject to RLS.

CREATE ROLE app_owner LOGIN PASSWORD '...';
CREATE ROLE app_user  LOGIN PASSWORD '...' NOSUPERUSER NOBYPASSRLS;

GRANT USAGE ON SCHEMA public TO app_user;

-- Tables and sequences created later by app_owner (i.e. by migrations)
-- are automatically usable by app_user.
ALTER DEFAULT PRIVILEGES FOR ROLE app_owner IN SCHEMA public
    GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_user;
ALTER DEFAULT PRIVILEGES FOR ROLE app_owner IN SCHEMA public
    GRANT USAGE, SELECT ON SEQUENCES TO app_user;

That means two sets of credentials. The simplest arrangement is to keep a single default database entry and choose the credentials by environment: the running application gets app_user, and the deploy step that runs migrate gets app_owner.

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": env("DB_NAME"),
        "USER": env("DB_USER"),          # app_user at runtime, app_owner for migrate
        "PASSWORD": env("DB_PASSWORD"),
        "HOST": env("DB_HOST"),
        "CONN_MAX_AGE": 60,
    },
}

Add the policy in a migration

Policies belong in migrations, so every environment gets them and they are reviewed like any other schema change.

from django.db import migrations

POLICY = (
    "tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid"
)

class Migration(migrations.Migration):
    dependencies = [("orders", "0007_order_tenant")]

    operations = [
        migrations.RunSQL(
            sql=f"""
                ALTER TABLE orders_order ENABLE ROW LEVEL SECURITY;
                CREATE POLICY tenant_isolation ON orders_order
                    USING ({POLICY})
                    WITH CHECK ({POLICY});
            """,
            reverse_sql="""
                DROP POLICY IF EXISTS tenant_isolation ON orders_order;
                ALTER TABLE orders_order DISABLE ROW LEVEL SECURITY;
            """,
        ),
    ]

The two clauses do different jobs. USING decides which existing rows are visible to SELECT and which rows UPDATE and DELETE can touch; rows that fail it are silently left out. WITH CHECK is evaluated against the new contents of rows written by INSERT and UPDATE, and a failure raises an error instead of being silently skipped. Together they stop the app from reading another tenant's rows, and from writing a row into another tenant or moving one across by changing its tenant_id.

The expression reads the tenant from a custom setting, app.tenant_id, which the application sets per request (below). current_setting(name, true) returns NULL rather than raising when the setting doesn't exist, and NULLIF(..., '') turns an empty value into NULL too, which matters because casting '' to uuid would be an error. A NULL tenant makes the comparison NULL, and PostgreSQL treats NULL like false: no rows are visible and every write fails the check. If the application forgets to set a tenant, the policy fails closed.

FORCE ROW LEVEL SECURITY

ALTER TABLE orders_order FORCE ROW LEVEL SECURITY makes policies apply to the table owner as well. It is the right tool when you can't separate roles, for example when a platform gives you a single database user, or when you want owner-run scripts to get the same protection as the app. Expect side effects: a data migration running as the owner will see no rows unless it sets a tenant first. Superusers and BYPASSRLS roles still bypass the policy, so FORCE does not replace the role split. It protects against one particular mistake.

Policies on many tables

Real schemas have dozens of tenant-scoped tables. Policy names only need to be unique per table, so the same SQL can be generated for each:

# tenancy/rls.py
from django.db import migrations

EXPR = "tenant_id = NULLIF(current_setting('app.tenant_id', true), '')::uuid"

def tenant_policy(table):
    return migrations.RunSQL(
        sql=f"""
            ALTER TABLE {table} ENABLE ROW LEVEL SECURITY;
            CREATE POLICY tenant_isolation ON {table}
                USING ({EXPR}) WITH CHECK ({EXPR});
        """,
        reverse_sql=f"""
            DROP POLICY IF EXISTS tenant_isolation ON {table};
            ALTER TABLE {table} DISABLE ROW LEVEL SECURITY;
        """,
    )

A migration then lists its tables: operations = [tenant_policy(t) for t in ["orders_order", "orders_orderline", "billing_invoice"]]. To make sure a new table never ships without a policy, add a test that asks the catalog for any table with a tenant_id column but RLS switched off:

import pytest
from django.db import connection

@pytest.mark.django_db
def test_every_tenant_table_has_rls():
    with connection.cursor() as cursor:
        cursor.execute("""
            SELECT c.relname
            FROM pg_class c
            JOIN pg_namespace n ON n.oid = c.relnamespace
            JOIN information_schema.columns col
              ON col.table_schema = n.nspname AND col.table_name = c.relname
            WHERE n.nspname = 'public'
              AND c.relkind = 'r'
              AND col.column_name = 'tenant_id'
              AND NOT c.relrowsecurity
        """)
        missing = [row[0] for row in cursor.fetchall()]
    assert missing == []

Watch for tables this test can’t see: many-to-many through tables that Django generates have no tenant_id column. Either use an explicit through model that carries tenant_id, or accept that those rows are protected only through the tables they join.

Set the tenant for each request

The application tells PostgreSQL which tenant is active by setting app.tenant_id at the start of each request:

from django.db import connection, transaction

class TenantMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        tenant = resolve_tenant(request)  # subdomain, header or session
        request.tenant = tenant
        if tenant is None:
            # No setting: tenant-scoped tables return nothing.
            return self.get_response(request)

        with transaction.atomic():
            with connection.cursor() as cursor:
                cursor.execute(
                    "SELECT set_config('app.tenant_id', %s, true)",
                    [str(tenant.id)],
                )
            response = self.get_response(request)
            if response.status_code >= 500:
                transaction.set_rollback(True)
            return response

The third argument to set_config makes the value transaction-local: it disappears when the transaction ends. That is the important design choice. With CONN_MAX_AGE, Django reuses database connections across requests, and PgBouncer in transaction pooling mode hands a server connection to a different client after each transaction. A session-level setting would outlive the request and could be picked up by the next one, which is the very leak we are trying to prevent. PgBouncer's feature table marks SET/RESET as never compatible with transaction pooling for this reason.

The trade-offs:

• Every tenant request runs in one transaction. That’s the same behaviour as ATOMIC_REQUESTS. Slow requests hold their transaction open longer, so keep external calls (payment providers, email) out of request handlers where you can.
• Errors don’t roll back on their own. Django converts exceptions raised by a view into responses before they reach the surrounding middleware, so atomic() here sees a normal return, even for a 500. The set_rollback(True) line makes sure a failed request doesn't commit half its writes.
• Streaming responses lose the tenant. The body of a StreamingHttpResponse is generated after the middleware has returned and the transaction has ended. Django's transaction docs make the same point about streaming. Any query made while streaming sees no tenant rows. Load the data first, or set the tenant again inside the generator.
• Tenant resolution needs a readable table. resolve_tenant() runs before the setting exists, so the tenants table itself must not be behind this policy.

Background jobs and commands need the same context

Celery tasks, management commands and shell sessions don’t go through middleware, so they need their own way to set the tenant:

from contextlib import contextmanager

from django.db import connection, transaction

@contextmanager
def tenant_context(tenant_id):
    with transaction.atomic():
        with connection.cursor() as cursor:
            cursor.execute("SELECT current_setting('app.tenant_id', true)")
            previous = cursor.fetchone()[0] or ""
            cursor.execute(
                "SELECT set_config('app.tenant_id', %s, true)", [str(tenant_id)]
            )
        yield
        # Releasing a savepoint does not undo a transaction-local setting,
        # so restore the outer value before leaving a nested block.
        with connection.cursor() as cursor:
            cursor.execute("SELECT set_config('app.tenant_id', %s, true)", [previous])

The restore step at the end covers a subtle case. When tenant_context runs inside an existing transaction (for example inside a request, or inside a test), atomic() creates a savepoint. PostgreSQL's SET docs say that rolling back to a savepoint cancels a local setting, but releasing one does not, so without the restore the inner tenant would stay active for the rest of the outer transaction. If the block raises, the savepoint is rolled back and the setting reverts by itself.

A Celery task then looks like this:

from celery import shared_task

@shared_task
def send_invoice_reminders(tenant_id):
    with tenant_context(tenant_id):
        for invoice in Invoice.objects.filter(status="overdue"):
            send_reminder(invoice)

Pass the tenant ID as a string, never a model instance, and have a scheduler task fan out one task per tenant. If send_reminder makes slow network calls, collect the IDs inside the context and do the slow part afterwards, so the transaction stays short.

Some work really is cross-tenant: billing runs, platform-wide reports, support tooling. Give that work its own DATABASES entry that connects as a role with BYPASSRLS, so bypassing isolation is something you do on purpose, by name:

DATABASES["cross_tenant"] = {
    **DATABASES["default"],
    "USER": env("DB_CROSS_TENANT_USER"),      # role with BYPASSRLS
    "PASSWORD": env("DB_CROSS_TENANT_PASSWORD"),
}
DATABASE_ROUTERS = ["tenancy.routers.CrossTenantRouter"]

# tenancy/routers.py
class CrossTenantRouter:
    def db_for_read(self, model, **hints):
        return None    # nothing is routed to cross_tenant implicitly

    def db_for_write(self, model, **hints):
        return None

    def allow_migrate(self, db, app_label, model_name=None, **hints):
        return False if db == "cross_tenant" else None

Code then opts in explicitly with Invoice.objects.using("cross_tenant"), which is easy to search for and review. The router's allow_migrate keeps migrate from ever running through that alias. In tests, a second alias pointing at the same database needs some thought: Django's TEST["MIRROR"] setting can redirect it to default, but it only works with TransactionTestCase, and it means the alias runs as the default role during tests.

Keep application filtering, and test the boundary

RLS is the safety net, not the main mechanism. Keep your tenant-aware managers: they make intent obvious in code, they give the query planner a direct tenant_id = ... condition, and they keep the application correct if it ever runs against a database without the policies.

Some performance and design notes:

• Index for the policy. Make tenant_id the first column of composite indexes on tenant tables, for example (tenant_id, created_at), so the policy's condition and your query's ordering use the same index.
• Keep the policy expression simple. A column compared with a setting is cheap. Subqueries or function calls in a policy run as part of every query on the table, and the PostgreSQL docs warn that policies which look at other tables can also introduce race conditions.
• Check plans as the app role. PostgreSQL evaluates policy conditions before your query’s own conditions, except for functions marked leakproof, so a plan can differ between the owner and the app role. Run EXPLAIN as app_user with a tenant set.
• Scope unique constraints per tenant. Unique and foreign-key checks bypass RLS. A global unique email lets one tenant discover that an address exists in another tenant. Use UniqueConstraint(fields=["tenant", "email"], ...). For the same reason, validate that foreign keys point to rows in the same tenant.

And test the boundary itself, against a real PostgreSQL database, connected as the restricted role:

import pytest
from django.db import DatabaseError, connection, transaction

@pytest.fixture
def as_app_role(db):
    # The test connection's user must be a member of app_user.
    with connection.cursor() as cursor:
        cursor.execute("SET LOCAL ROLE app_user")

def test_other_tenants_orders_are_invisible(as_app_role, tenant_a, tenant_b):
    with tenant_context(tenant_b.id):
        other = Order.objects.create(tenant=tenant_b, total=100)

    with tenant_context(tenant_a.id):
        # Use the unscoped manager on purpose: only RLS stands in the way.
        assert not Order.all_tenants.filter(pk=other.pk).exists()

def test_cannot_write_into_another_tenant(as_app_role, tenant_a, tenant_b):
    with tenant_context(tenant_a.id):
        with pytest.raises(DatabaseError), transaction.atomic():
            Order.all_tenants.create(tenant=tenant_b, total=100)

The first test proves that the database hides other tenants’ rows even when the application forgets to filter. The second proves that WITH CHECK blocks writes into another tenant. Run both as the app role. Run them as the owner or a superuser and they test nothing, because RLS doesn't apply.

Checklist

• The app connects as a role that is not superuser, not BYPASSRLS and not the table owner (or the tables use FORCE ROW LEVEL SECURITY).
• Migrations run as the owner role, and default privileges grant the app role access to new tables.
• Every tenant table has RLS enabled and a policy with both USING and WITH CHECK, enforced by a catalog test.
• The tenant setting is transaction-local, set in middleware and in tenant_context for jobs and commands.
• Failed requests roll back; streaming responses don’t query tenant tables after the middleware returns.
• Cross-tenant work uses a separate, explicitly named connection with a BYPASSRLS role.
• Unique constraints include tenant_id; foreign keys are validated within the tenant.
• tenant_id leads composite indexes, and plans are checked as the app role.
• Isolation tests run against PostgreSQL as the restricted role.

Conclusion

Application-level filtering is how a multi-tenant Django app is supposed to work; row-level security is what keeps it safe when it doesn’t. The setup is modest: two roles, a policy per table in a migration, a transaction-local setting per request, and an explicit path for the work that really does span tenants. In return, a forgotten filter in a raw query, a report or a new developer’s first feature produces an empty result instead of another customer’s data. It does not remove the need for tenant-aware code or tests, but it means a single mistake no longer becomes a data breach.

Lycore builds multi-tenant SaaS platforms on Django, React, Flutter and .NET. Get in touch at lycore.com if you want a review of how your platform keeps customer data apart.

For most teams, the first production agent starts life on somebody’s laptop. It runs on that developer’s API key, their GitHub token and their SaaS login, and it works, so it ships. Six months later the developer changes teams. The agent is now opening pull requests, calling models and spending money under the name of someone who no longer owns it.

That used to be an awkward bit of tech debt. Since this month it’s also out of step with how the platforms themselves treat agents. Several releases in October 2026 point the same way: an agent is a workload, and a workload needs its own identity.

In this post we’ll look at what changed, why it matters, and how to move an agent off human credentials one step at a time. The examples use Python, but the principles apply whatever your stack.

This week’s reminder

Binance is dealing with a security incident that surfaced on October 29 and that it estimates at $6.3 billion in impact. According to the exchange, its initial investigation suggests that a compromised DevOps engineer’s laptop gave an attacker access to a service container, which then submitted unauthorized withdrawal transactions. Binance says it has found no evidence of a wallet private-key leak and has promised a detailed technical post-mortem, which has not been published yet, so the full chain of events isn’t known. Even the initial account shows the pattern this post is about: one person’s machine led to a workload that was able to submit withdrawals, and they went through. That’s the case for giving every service and agent its own narrowly scoped, short-lived identity, and for requiring a separate check before any high-risk action goes through.

A signed-in AI session is an identity worth stealing too: in October, a crafted ChatGPT share link could run code as the victim and read their private chats, until OpenAI fixed it on October 26 (Parameter Labs).

What shipped this month

Four recent releases, each from its primary source:

• Agents as code. On October 20, 2026, Cursor made its Agent SDK generally available. It’s the framework Cursor uses for its own managed agents, and it treats an agent like any other software project: prompts in Markdown, tools as TypeScript functions, evals in the repo, and a single agent-sdk deploy command. Once agents live in repositories and deploy like services, the question "whose credentials does this run on?" can't be dodged.
• Workload-owned identity for Copilot agents. On October 27, 2026, GitHub announced server-to-server authentication for the Copilot SDK in public preview. A GitHub App installed on an organization can make Copilot requests with a short-lived installation token. Usage is billed to the organization, and that organization’s model policies and budgets apply. GitHub’s own description is that this lets agents operate “without tying their authentication or usage to individual users.”
• Credential binding for MCP clients. On October 20, 2026, FastMCP published advisory GHSA-vqqp-x5qm-5qp4 along with version 4.1.2. In affected versions (4.0.0 through 4.1.1), the OAuth client-credentials providers could send a client secret to whatever authorization server an MCP server advertised. The fix adds an issuer parameter, and the maintainers say plainly that upgrading alone is not enough: you have to set it.
• Tighter egress and governance controls for hosted agents. The Claude Platform release notes for October list several governance changes for Claude Managed Agents. On October 7, limited networking began applying allowed_hosts to the web search and web fetch tools, and web_fetch became restricted to URLs that have already appeared in the session. On October 20, organization-level web fetch domain restrictions started applying to agent sessions, and "user profiles" launched so platforms can attribute requests to individual end customers.

None of these is dramatic by itself, but together they mark a shift: vendors are building for agents that run unattended, with their own identities, budgets and network boundaries.

The anti-pattern: an agent wearing a human’s badge

Here’s what typically goes wrong when an agent runs on a person’s credentials.

1. Attribution is wrong. Audit logs say a human did something they never did. When an incident review asks “who merged this?”, the honest answer (“the triage bot, using Sara’s token”) isn’t in any log.
2. Offboarding breaks production. Revoke the person’s access and the agent stops working. Keep their access so the agent keeps working, and you have a security finding.
3. Blast radius is too big. A personal token usually carries everything that person can do. The agent needed read access to one repository and got write access to forty.
4. Spend is invisible. Model usage lands on someone’s personal seat or a shared key, so you can’t see what a given agent costs, and you can’t cap it.
5. Secrets never expire. A static API key sitting in a CI variable stays valid until someone remembers to rotate it, which in practice means never.

All five have the same fix: give the agent a non-human identity of its own, issue it short-lived credentials, and scope those credentials tightly.

Principle 1: one agent, one identity, per environment

Start with an inventory. For every agent you run, write down:

• what it does and which events trigger it,
• every system it calls (model APIs, Git hosting, MCP servers, internal APIs),
• the credentials it uses today, and which human they belong to.

Then create a dedicated non-human principal for each agent in each environment. On GitHub that’s a GitHub App. On the Claude API it’s a service account. On your cloud it’s an IAM role or a Kubernetes service account. Production and staging should never share an identity. If you can’t distinguish your staging agent from your production agent in an audit log, you’ll eventually be debugging the wrong one.

Anthropic’s documentation puts the distinction well: a workspace API key is a credential, whereas a service account has credentials. Credentials can be rotated and revoked. The identity, and the audit trail attached to it, stays.

Principle 2: federate instead of storing keys

A static key is a secret you have to create, distribute, rotate and eventually leak. Workload Identity Federation (WIF) removes that secret. Your workload already has an identity from its platform, such as a Kubernetes projected service-account token, a cloud metadata server or the GitHub Actions OIDC endpoint. It presents that signed JWT and exchanges it for a short-lived access token.

The Claude API has supported this since May 2026. You configure three resources: a service account, a federation issuer (your identity provider) and a federation rule that maps JWT claims to the service account. The rule also sets the token scope and lifetime: token_lifetime_seconds accepts 60 to 86,400 and defaults to 3,600. Per the Workload Identity Federation guide, the SDK handles the exchange and the refresh:

from anthropic import Anthropic, WorkloadIdentityCredentials, IdentityTokenFile

client = Anthropic(
    credentials=WorkloadIdentityCredentials(
        identity_token_provider=IdentityTokenFile(
            "/var/run/secrets/anthropic.com/token"
        ),
        federation_rule_id="fdrl_...",
        organization_id="00000000-0000-0000-0000-000000000000",
        service_account_id="svac_...",
        workspace_id="wrkspc_...",
    ),
)

In production, the docs recommend the zero-argument form. You ship one container image everywhere and inject ANTHROPIC_FEDERATION_RULE_ID, ANTHROPIC_ORGANIZATION_ID, ANTHROPIC_SERVICE_ACCOUNT_ID, ANTHROPIC_WORKSPACE_ID and ANTHROPIC_IDENTITY_TOKEN_FILE per environment. The SDK re-reads the token file on every exchange, so rotated Kubernetes tokens are picked up automatically.

The migration gotcha: a leftover key wins

The documented credential precedence puts ANTHROPIC_API_KEY above the federation variables. If an old key is still in your container environment, a CI secret or a shell profile, it silently shadows federation. Everything looks migrated, but you're still using the static key. The documented migration path is:

1. Configure federation alongside the existing key.
2. Run ant auth status inside the workload to see which credential source wins. At this stage it'll still be the API key.
3. Unset ANTHROPIC_API_KEY everywhere it's injected, then re-check ant auth status.
4. Delete the old key in the Console.

We’d add a fifth step: make a CI check fail if ANTHROPIC_API_KEY reappears in a deployment manifest for a federated workload. Regressions here are quiet.

A second detail worth knowing: by default, identity tokens with a jti claim are single-use. If your workload fetches its own JWTs, a retry loop that re-presents the same token will be rejected. Mint a fresh JWT for each exchange.

Principle 3: give the agent the smallest useful token

Short-lived isn’t the same as least-privilege. A one-hour token with admin rights is still an admin token. Every mature platform lets you narrow tokens at mint time, so use that.

The server-to-server guide for the Copilot SDK is a good template. You create a GitHub App with only the read-only Copilot Requests organization permission and install it on the organization that should pay. Then you mint an installation token that asks for exactly that permission:

import time

import httpx
import jwt  # PyJWT

def mint_copilot_token(client_id: str, private_key: str, installation_id: int) -> str:
    now = int(time.time())
    app_jwt = jwt.encode(
        {"iat": now - 60, "exp": now + 9 * 60, "iss": client_id},
        private_key,
        algorithm="RS256",
    )
    resp = httpx.post(
        f"https://api.github.com/app/installations/{installation_id}/access_tokens",
        headers={
            "Accept": "application/vnd.github+json",
            "Authorization": f"Bearer {app_jwt}",
        },
        json={"permissions": {"organization_copilot_requests": "read"}},
    )
    resp.raise_for_status()
    return resp.json()["token"]

Then pass the token explicitly and switch off fallback to whatever user credentials happen to be on the machine:

import os

from copilot import CopilotClient, RuntimeConnection

client = CopilotClient(
    connection=RuntimeConnection.for_stdio(),
    github_token=os.environ["INSTALLATION_TOKEN"],
    use_logged_in_user=False,
)

The use_logged_in_user=False line matters more than it looks. The Copilot SDK's documented auth priority falls back through environment variables, stored CLI OAuth credentials and gh auth. On a developer machine or a shared runner, that fallback can quietly put your agent back on a human's identity. Turning it off makes a missing token fail loudly instead.

Installation tokens expire after one hour, so mint a new one before expiry and restart the SDK client with it. If the agent runs inside GitHub Actions, you don’t need an App at all. The docs show granting the built-in GITHUB_TOKEN the copilot-requests: write permission in the workflow, subject to an organization policy.

The same narrowing applies to repository access. When you generate an installation access token, the repositories/repository_ids and permissions body parameters can only restrict what the installation already has, never widen it. An agent that labels issues in one repository should get a token for that repository with issues permission, and nothing else.

Principle 4: bind credentials to where they may be sent

The FastMCP advisory deserves attention beyond FastMCP users, because the bug class is general. A machine-to-machine OAuth client discovers its token endpoint from the MCP server’s metadata. If the client doesn’t check which authorization server it’s talking to, a malicious or compromised MCP server can advertise its own, and your client hands over its secret.

The advisory puts it in the same vulnerability class as separate advisories for the official Python and TypeScript MCP SDKs. With agents connecting to more and more MCP servers, some of them third-party, this is an easy mistake to repeat. The fix in FastMCP 4.1.2 is one line, but you have to write it:

import os

from fastmcp import Client
from fastmcp.client.auth import ClientCredentialsOAuthProvider

auth = ClientCredentialsOAuthProvider(
    client_id="reporting-agent",
    client_secret=os.environ["REPORTING_AGENT_SECRET"],
    scopes=["reports:read"],
    issuer="https://auth.example.com",  # the AS that issued these credentials
)

async with Client("https://mcp.example.com/mcp", auth=auth) as client:
    tools = await client.list_tools()

With issuer set, the provider only sends credentials to an authorization server whose discovered metadata names that issuer, and otherwise raises an OAuthFlowError. Leaving it out still works for compatibility but emits an MCPDeprecationWarning. We recommend turning that warning into an error in your test suite. If you ran an affected version against MCP servers you don't fully control, rotate those client secrets.

The general rule for any agent framework: every credential should know its legitimate destination, and the client should refuse to send it anywhere else.

Principle 5: constrain egress, not just ingress

Identity controls who the agent is. Network policy controls where it can reach, and for agents that read untrusted content, egress control is your main defense against data exfiltration through prompt injection.

The October Claude Managed Agents changes show the direction:

• With limited networking, the environment's allowed_hosts list now also governs web_search and web_fetch. A fetch to an unlisted host returns a url_not_allowed error to the agent.
• web_fetch now only fetches URLs that have already appeared in the session, for example in a user message or a search result. A URL that shows up only in the model's own output or in a tool's output can't be fetched. That closes off a common trick where injected instructions build an exfiltration URL.
• Organization admins can disable web fetch or set domain lists centrally, and agent sessions inherit those restrictions.

If you run your own agent loop, copy the pattern: keep an explicit egress allowlist per agent, enforce it outside the model (in the HTTP client or a proxy, not in the prompt), and log every denied request. A denied request is often the first sign that something is trying to steer your agent.

Principle 6: make every agent show up on the bill

Once each agent has its own identity, cost attribution mostly takes care of itself, and you can add caps.

• GitHub: with server-to-server auth, usage is billed to the organization that owns the App installation, under that organization’s budgets and spending controls. App-attributed usage appears in AI usage CSV reports with an app_client_id field, and owners can cut off an app by suspending or uninstalling it.
• Claude Managed Agents: since August 7, 2026, a session can carry a hard budget priced at public list rates. When the budget is reached, the session pauses with a budget_reached stop reason instead of starting new model requests. Deployments apply the budget to every session they start.
• Multi-tenant platforms: if you serve your own customers through a model API, the new Claude user profiles let you tell Anthropic which end customer each request is for. That way a safety enforcement action can apply to one customer instead of your whole organization.

Our rule of thumb: if you can’t answer “what did this agent cost last week, and what stops it costing ten times that next week?”, it isn’t ready for unattended operation.

A migration checklist

Here’s the order we’d tackle this in for an existing agent:

1. Inventory every agent, its triggers, the systems it touches and the human credentials it borrows.
2. Create a non-human identity per agent per environment: GitHub App, model-provider service account, cloud role.
3. Federate where you can. Use your platform’s OIDC token instead of a stored key, and keep token lifetimes short.
4. Remove the fallbacks. Unset leftover API keys, disable logged-in-user credential discovery, and confirm with the tool’s auth-status command.
5. Scope tokens at mint time to the permissions and repositories the agent actually needs.
6. Bind OAuth credentials to their issuer in every MCP or OAuth client, and treat binding warnings as errors.
7. Allowlist egress outside the model, and log denials.
8. Set budgets per agent or per session, and review attributed usage weekly.
9. Write the kill switch down. Know exactly which App to suspend or which service account to disable, and test that it works.

You don’t need to do all nine before the next release. Steps 2 and 4 alone fix most of the real-world risk.

Conclusion

The agent tooling released this month (agents defined as code, workload-owned Copilot identities, issuer-bound MCP credentials, egress-aware hosted agents) all assumes your agents are services, not extensions of a person. Our credentials, permissions and budgets should make the same assumption.

The work isn’t glamorous, and most of it is configuration rather than code. It is what lets you answer the questions that come up the first time an agent does something unexpected: which agent was it, what was it allowed to do, what did it cost, and how do we stop it? If you can answer those for every agent you run, you’re ahead of most teams. If you can’t, pick your busiest agent and give it its own identity this week.

By default, when a component throws while React is rendering, React removes the whole UI from the screen. The user sees a blank page, and you see nothing, because in React 19 the way errors are reported also changed. If your monitoring quietly stopped seeing some errors after an upgrade, that is why.

This post covers the pieces that decide what happens when React crashes: error boundaries, the three root error callbacks in React 19, the errors boundaries do not catch, how to reset a boundary, and how to test all of it. The code was run with React 19.3.1, react-error-boundary 6.1.6 and Vitest 4.1.12 with Testing Library in jsdom. The Sentry snippet comes from Sentry’s docs and was not run. The /client-errors endpoint is made up.

What an error boundary is

An error boundary is a component that catches errors thrown while rendering its children and shows a fallback instead. React still requires a class component for this. The docs say there is currently no way to write one as a function component, and point to react-error-boundary instead. Here is a minimal one:

import { Component } from "react";

export class ErrorBoundary extends Component {
  state = { error: null };

  static getDerivedStateFromError(error) {
    return { error }; // render the fallback on the next pass
  }

  componentDidCatch(error, info) {
    this.props.onError?.(error, info.componentStack);
  }

  reset = () => {
    this.props.onReset?.();
    this.setState({ error: null });
  };

  render() {
    if (this.state.error) {
      return this.props.fallback({
        error: this.state.error,
        reset: this.reset,
      });
    }
    return this.props.children;
  }
}

getDerivedStateFromError switches the state so the next render shows the fallback. componentDidCatch receives info.componentStack, a list of the components between the crash and the boundary. That stack is the most useful thing you can attach to a report, because the JavaScript stack alone rarely tells you which screen was involved. In development, the docs example also logs React.captureOwnerStack(), which adds the owner stack, and warns that it is not available in production.

In practice you do not need to maintain this class. The react-error-boundary package wraps it, and adds the things you would end up writing anyway: fallback, fallbackRender and FallbackComponent for the fallback UI, onError and onReset callbacks, a resetKeys prop, and a useErrorBoundary hook. Version 6 targets ES modules, and its README points projects that cannot use them to version 5.

Where to put boundaries

One boundary around the whole app is better than none, but it turns every bug into a full-page failure. The React docs suggest thinking about where it makes sense to show an error message, and give a messaging app as an example. I use three layers:

• Root: a last-resort fallback for the whole page. It should be plain and have a reload button.
• Route or page: keeps the header and navigation alive when one screen crashes. Pass the pathname as a reset key so moving to another page clears the error.
• Widget: independent panels such as a comments list, a chart or a recommendations rail. A crash there should not take the rest of the page with it.

import { ErrorBoundary, useErrorBoundary } from "react-error-boundary";
import { useTransition } from "react";
import { AppFallback, WidgetFallback } from "./Fallbacks.jsx";

export function Profile({ userId, load }) {
  const user = load(userId); // throws if the data is bad
  return <p>Profile: {user.name}</p>;
}

export function Dashboard({ userId, load }) {
  return (
    <ErrorBoundary FallbackComponent={AppFallback}>
      <h1>Dashboard</h1>
      <ErrorBoundary
        FallbackComponent={WidgetFallback}
        resetKeys={[userId]}
      >
        <Profile userId={userId} load={load} />
      </ErrorBoundary>
      <ErrorBoundary FallbackComponent={WidgetFallback}>
        <p>Recent activity</p>
      </ErrorBoundary>
    </ErrorBoundary>
  );
}

One rule: a boundary cannot catch errors thrown by itself, which the docs list explicitly. So the fallback must be dumb. No data fetching, no components that might throw. And keep a boundary above any boundary whose fallback does real work.

Also think about what you wrap. Put the boundary around the widget, not around a form with unsaved input, or a crash will throw away what the user typed.

Resetting a boundary

There are three ways a boundary leaves its error state:

1. The user clicks “Try again”. The fallback receives resetErrorBoundary, which clears the error and renders the children again.
2. resetKeys change. In the example above, resetKeys={[userId]} clears the error as soon as the user ID changes. That is the right model when the failure is tied to some input.
3. onReset runs whenever the boundary resets. Use it to clear whatever caused the failure, such as a bad cache entry or a stuck query.

Resetting renders the same children again. If the crash is deterministic, the user gets the fallback again, so do not reset automatically on a timer. Let a person decide, or tie the reset to a change in input.

What boundaries do not catch

This is where most “but I have an error boundary” bugs come from. According to the React docs, error boundaries do not catch errors in:

• event handlers
• server-side rendering
• the error boundary itself
• asynchronous code such as setTimeout or requestAnimationFrame callbacks, with one exception: errors thrown inside the function you pass to the startTransition returned by useTransition are caught

I checked the event handler case in a test. A button that throws in onClick inside a boundary does not show the fallback, and in my jsdom run the error showed up as a global error event on window. If you want the failed action to take over the UI, send the error to the boundary yourself with useErrorBoundary, or use a transition:

export function SaveButton({ save }) {
  const { showBoundary } = useErrorBoundary();
  return (
    <button
      onClick={async () => {
        try {
          await save();
        } catch (error) {
          showBoundary(error); // hand the error to the nearest boundary
        }
      }}
    >
      Save
    </button>
  );
}

export function SaveWithTransition({ save }) {
  const [pending, startTransition] = useTransition();
  return (
    <button
      disabled={pending}
      onClick={() => startTransition(async () => await save())}
    >
      Save
    </button>
  );
}

Both versions work in the tests, and both show the nearest fallback after a rejected save(). The difference is intent. A try/catch with a toast is often the better answer for a failed save, because the user keeps their screen. Reserve showBoundary for failures that leave the component unable to continue.

Server rendering has its own hooks. With renderToPipeableStream, the docs describe onError as firing on every server error, recoverable or not. By default it only calls console.error, so if you override it to send crash reports, keep calling console.error too. onShellError fires if the initial shell fails to render, before any bytes are sent, so that is where you return a fallback HTML page.

Reporting: the three root callbacks

React 19 changed what happens to errors. The upgrade guide says errors in render are no longer re-thrown. Errors that no boundary caught go to window.reportError. Errors that a boundary caught go to console.error. If your production reporting relied on re-thrown errors, you may need to update it.

The consequence is easy to miss. window.reportError() emulates an uncaught exception for global handlers, which is what a monitoring SDK listens to. An error caught by a boundary only reaches console.error, so it never gets reported unless you do it yourself. React 19 adds three options to createRoot and hydrateRoot for this. Each receives the error and an errorInfo object that contains componentStack:

• onCaughtError: an error was caught by an error boundary
• onUncaughtError: an error was thrown and no boundary caught it
• onRecoverableError: React recovered from an error on its own

import { createRoot } from "react-dom/client";
import App from "./App.jsx";
import { report } from "./report.js";

// In development, leave these out so React's default logging and your
// overlay keep working.
const errorOptions =
  process.env.NODE_ENV === "production"
    ? {
        onCaughtError: (error, info) => report(error, info, "caught"),
        onUncaughtError: (error, info) => report(error, info, "uncaught"),
        onRecoverableError: (error, info) => report(error, info, "recoverable"),
      }
    : {};

createRoot(document.getElementById("root"), errorOptions).render(<App />);

The reporting function sends the pieces you will want when triaging:

const ENDPOINT = "/client-errors";

export function report(error, info, source) {
  const body = JSON.stringify({
    source, // "caught" | "uncaught" | "recoverable"
    message: error instanceof Error ? error.message : String(error),
    stack: error instanceof Error ? error.stack : undefined,
    componentStack: info?.componentStack,
    url: window.location.href,
  });
  navigator.sendBeacon?.(ENDPOINT, body);
}

Three details from the code. First, the docs recommend leaving these options out in development, so React’s default logging and your overlay keep working. Second, sendBeacon is meant for diagnostics like this, and it exists for the case where the browser may be about to unload the page. Third, source tells you whether the user saw a fallback (caught), a blank screen (uncaught) or nothing at all (recoverable). That distinction is how you decide what to fix first. I would send recoverable errors at low severity and watch for a spike after a deploy.

If you use Sentry, its docs show the same three options wired up with Sentry.reactErrorHandler():

import * as Sentry from "@sentry/react";
import { createRoot } from "react-dom/client";

const root = createRoot(document.getElementById("root"), {
  onUncaughtError: Sentry.reactErrorHandler(),
  onCaughtError: Sentry.reactErrorHandler(),
  onRecoverableError: Sentry.reactErrorHandler(),
});

Sentry’s guide describes this as a global safety net with no UI handling, and its ErrorBoundary component as the scoped option that also renders a fallback. I did not test Sentry itself. If you use both, read its docs on how duplicate events are avoided.

Do not report twice

Reporting from a boundary’s onError and from the root onCaughtError sends every caught error twice. I confirmed that both callbacks fire once for the same error. I prefer to report from the root callbacks only, because that covers every boundary, including ones inside third-party libraries. Use a boundary's onError for local side effects only.

Whatever you use, check three things before shipping. Upload source maps, because production stacks are usually minified. Scrub the message, because errors often contain user data. And dedupe or throttle, because a component that crashes on every render can produce a flood of identical reports.

Fallback UX

A fallback is a screen in your product, so design it like one:

• Say what failed in the user’s terms (“This panel could not load”), not what the exception was. Keep stack traces out of production fallbacks.
• Offer a next step. “Try again” for a widget, a reload for the root, a link home for a route.
• Keep the layout stable. A fallback with the same height as the widget stops the page from jumping.
• Use role="alert" so assistive technology announces it, as react-error-boundary's own example does.
• Give support something to work with. If you can attach a short reference ID to the report and show it in the fallback, “it said A7F2” is a much better bug report than “it broke”.

export function WidgetFallback({ resetErrorBoundary }) {
  return (
    <div role="alert">
      <p>This panel could not load.</p>
      <button onClick={resetErrorBoundary}>Try again</button>
    </div>
  );
}

export function AppFallback() {
  return (
    <div role="alert">
      <h1>Something went wrong</h1>
      <p>Reload the page. If it keeps happening, contact support.</p>
      <button onClick={() => window.location.reload()}>Reload</button>
    </div>
  );
}

Testing error handling

Error handling that is never tested usually fails the first time it matters. These tests cover the layers from this post:

const load = (id) => {
  if (id === "bad") throw new Error("corrupt profile");
  return { name: `user ${id}` };
};

describe("placement and reset", () => {
  it("keeps the rest of the page alive when one widget crashes", () => {
    render(<Dashboard userId="bad" load={load} />);
    expect(screen.getByRole("alert").textContent).toContain("could not load");
    expect(screen.getByText("Dashboard")).toBeTruthy();
    expect(screen.getByText("Recent activity")).toBeTruthy();
  });

  it("resets on its own when resetKeys change", () => {
    const { rerender } = render(<Dashboard userId="bad" load={load} />);
    expect(screen.getByRole("alert")).toBeTruthy();
    rerender(<Dashboard userId="42" load={load} />);
    expect(screen.queryByRole("alert")).toBeNull();
    expect(screen.getByText("Profile: user 42")).toBeTruthy();
  });

They check that a crash is contained to one widget and that a changed key clears it. To check that they can fail, I removed resetKeys from the boundary, and the reset test failed.

To test the root callbacks, create the root yourself:

// No act() here: act() rethrows render errors to the test. Outside of it,
  // React behaves like it does in the browser and calls the root handlers.
  async function mount(ui, options) {
    const container = document.createElement("div");
    const root = createRoot(container, options);
    globalThis.IS_REACT_ACT_ENVIRONMENT = false;
    root.render(ui);
    await new Promise((resolve) => setTimeout(resolve, 50)); // let React commit
    return root;
  }

  it("onCaughtError fires for an error a boundary caught, with componentStack", async () => {
    const onCaughtError = vi.fn();
    const onUncaughtError = vi.fn();
    await mount(
      <ErrorBoundary fallback={<p>fallback</p>}>
        <Bomb />
      </ErrorBoundary>,
      { onCaughtError, onUncaughtError },
    );
    expect(onCaughtError).toHaveBeenCalledOnce();
    const [error, info] = onCaughtError.mock.calls[0];
    expect(error.message).toBe("render failed");
    expect(info.componentStack).toContain("Bomb");
    expect(onUncaughtError).not.toHaveBeenCalled();
  });

Four testing tips from running these:

• React 19 logs caught errors with console.error, so spy on it and keep the test output readable.
• render from Testing Library rethrows an uncaught render error, because act does. I confirmed that expect(() => render(<Bomb />)).toThrow() passes. When you want React to call onUncaughtError the way it does in a browser, call createRoot directly and skip act.
• Assert on componentStack, not just the message. A report without it is much harder to act on.
• Test the reset path as well as the failure path, since that is the one users click.

Checklist

• A root boundary with a plain fallback, plus boundaries at route and widget level.
• resetKeys on anything tied to an ID or route, and a "Try again" button everywhere else.
• Event handlers and async code handle their own errors, or call showBoundary.
• onCaughtError, onUncaughtError and onRecoverableError report to one place, only in production.
• Reports carry componentStack, route, release and a source label, with source maps uploaded and PII scrubbed.
• Server rendering has onError and onShellError set.
• Tests cover containment, reset and the root callbacks.

Conclusion

An error boundary does one job: it decides how much of the screen a crash takes with it. Everything else is on you. Put boundaries where a partial page is still useful, report from the root callbacks so nothing slips past, handle event and async errors where they happen, and test the fallback like any other screen. The moment to find out your reporting has a gap is a Tuesday afternoon in a test run, not the morning after a release.

Lycore builds and maintains production web and mobile apps in React, Flutter, Django and .NET.

A developer changes how orders are saved in an admin action and, a day later, finance asks why three hundred customers received a second “your order has shipped” email. Nobody touched the email code. It turns out a post_save receiver on Order sends that email whenever the status field is “shipped”, and the new admin action saved every selected order twice.

Django signals are appealing early in a project. They let you react to model changes without editing the code that makes them, and the first few receivers feel clean and decoupled. Over time they become the place where important business logic hides: invisible from the call site, triggered by saves from all over the codebase, and easy to break without knowing.

This post covers where signals cause trouble, what we use instead for business side effects, how to migrate without a rewrite, and the cases where signals are still the right tool.

Photo by Sasun Bughdaryan on Unsplash

The receiver behind the duplicate emails

It helps to look at the code that caused the incident. It’s the kind of receiver most Django projects have somewhere:

# orders/signals.py
from django.db.models.signals import post_save
from django.dispatch import receiver

from notifications.tasks import send_shipping_email
from .models import Order

@receiver(post_save, sender=Order)
def notify_customer_on_ship(sender, instance, created, **kwargs):
    if instance.status == Order.Status.SHIPPED:
        send_shipping_email.delay(instance.pk)

It reads fine in isolation. The problem is what it actually checks: not “the order has just shipped” but “the order is saved while its status is shipped”. Those are the same thing only as long as every code path saves a shipped order exactly once. Fixing a typo in the address of a shipped order sends the email again. So does an admin action that saves twice.

The usual patch is to remember the previous status in pre_save or __init__ and compare. That works until someone writes with update(), which skips both signals, or reloads the instance between the two saves. Each patch makes the receiver more clever, and none of it is visible from the code that triggers the email.

Why signals get harder to live with

The problems all come from the same property: the code that triggers the behaviour doesn’t mention it. Django’s own signals documentation now carries a warning to that effect:

Signals give the appearance of loose coupling, but they can quickly lead to code that is hard to understand, adjust and debug.

In practice, that shows up in a few specific ways:

• The call site tells you nothing. Reading order.save() doesn’t reveal the emails, stock updates and webhooks that follow. You have to search the codebase for receivers to find out.
• Saves fire them regardless of intent. Admin edits, management commands, scripts and the shell all trigger them. Fixtures loaded with loaddata do too; receivers get raw=True in that case, and most receivers never check it.
• Data migrations are inconsistent. RunPython code uses historical models from apps.get_model(), which aren’t the same class as your imported Order, so a receiver registered with sender=Order doesn’t fire there. One more way the same business event behaves differently depending on where the row was written.
• Bulk operations skip them. The QuerySet API reference is explicit that update() and bulk_create() don’t call save() and don’t send pre_save or post_save; bulk_update() doesn’t either. QuerySet.delete(), on the other hand, does send pre_delete and post_delete for each object. Remembering which is which is a job nobody should have.
• post_save can run before the data is committed. The signal is sent from save(), so when that save happens inside a transaction (an atomic() block, ATOMIC_REQUESTS, or a service that wraps several writes), the receiver runs before the commit. A receiver that enqueues a Celery task can have the worker pick it up and read the old row, or act on a change that is then rolled back.
• Order is an undocumented dependency. Receivers run in the order they were connected, which in practice means app-loading and import order. When one receiver depends on another’s side effect, reordering INSTALLED_APPS can change behaviour.

None of this is a bug in Django. Signals are a low-level notification mechanism. They simply aren’t a good home for business rules.

A service function makes the use case explicit

For anything that represents a business event (an order ships, a subscription is cancelled, an invoice is issued), we write a plain function that does the whole thing in one visible place:

# orders/services.py
from django.db import transaction
from django.utils import timezone

from notifications.tasks import send_shipping_email
from .models import Order, OrderEvent

class InvalidTransition(Exception):
    pass

def mark_order_shipped(order_id, tracking_number, *, actor):
    with transaction.atomic():
        order = Order.objects.select_for_update().get(pk=order_id)
        if order.status != Order.Status.PAID:
            raise InvalidTransition(f"Cannot ship an order in status {order.status}")

        order.status = Order.Status.SHIPPED
        order.tracking_number = tracking_number
        order.shipped_at = timezone.now()
        order.save(update_fields=["status", "tracking_number", "shipped_at"])

        OrderEvent.objects.create(order=order, kind="shipped", actor=actor)
        transaction.on_commit(lambda: send_shipping_email.delay(order.pk))

    return order

Everything the use case does is visible from the function body: the state check, the update, the audit record and the email. A few details carry most of the weight:

• The row lock. select_for_update() makes a second concurrent request wait until the first commits. It then sees the order as shipped, fails the status check and raises InvalidTransition. The duplicate admin save from the opening story would raise instead of sending a second email. select_for_update() has to run inside a transaction; Django raises TransactionManagementError if you evaluate it in autocommit mode.
• The transition, not the state. The email is tied to the move from paid to shipped, which can happen once, rather than to “saved while shipped”, which can happen any number of times.
• The email after commit. transaction.on_commit() runs the callback only once the transaction commits, and discards it if the transaction rolls back. The worker always sees the new state, and a failed transaction sends nothing.
• A primary key, not an object. The task receives order.pk and loads a fresh row, rather than a serialised copy of an instance that may be stale by the time the worker runs.

If you’re on Celery 5.4 or later with its Django task class, send_shipping_email.delay_on_commit(order.pk) does the same wrapping for you. The Celery Django guide covers it, including the trade-off that delay_on_commit() doesn’t return a task ID.

Calling it from everywhere that ships orders

Views, admin actions, API endpoints and management commands all call the same function, and handle the failure in the way that suits them:

# orders/views.py
from django.contrib import messages
from django.contrib.auth.decorators import permission_required
from django.shortcuts import redirect
from django.views.decorators.http import require_POST

from .forms import ShipOrderForm
from .services import InvalidTransition, mark_order_shipped

@require_POST
@permission_required("orders.change_order")
def ship_order(request, order_id):
    form = ShipOrderForm(request.POST)
    if form.is_valid():
        try:
            mark_order_shipped(
                order_id, form.cleaned_data["tracking_number"], actor=request.user
            )
        except InvalidTransition as exc:
            messages.error(request, str(exc))
    return redirect("orders:detail", order_id)

Two details are worth knowing. If you use ATOMIC_REQUESTS, the service’s atomic() block becomes a savepoint inside the request transaction, and Django runs on_commit callbacks when the outermost transaction commits, so the email still waits for the real commit. And if you call on_commit() when no transaction is open, Django runs the callback immediately. That’s why the service registers it inside its own atomic() block.

A data migration that bulk-updates historic orders doesn’t call the service, and therefore doesn’t email anyone, which is exactly what you want. If you ever do need the side effects for a backfill, you call the service in a loop and the decision is visible in the code.

Make the side effect safe to repeat

Moving the trigger into a service fixes “sent when it shouldn’t be”. It doesn’t make the task itself safe to retry. Celery tasks can run more than once, for example when a worker retries after a temporary mail server error. A small guard on the task keeps retries from turning into duplicates:

# notifications/tasks.py
from smtplib import SMTPException

from celery import shared_task
from django.core.mail import send_mail
from django.utils import timezone

from orders.models import Order

@shared_task(autoretry_for=(SMTPException,), retry_backoff=True, max_retries=5)
def send_shipping_email(order_id):
    order = Order.objects.select_related("customer").get(pk=order_id)
    if order.shipping_email_sent_at is not None:
        return

    send_mail(
        subject="Your order has shipped",
        message=f"Tracking number: {order.tracking_number}",
        from_email=None,
        recipient_list=[order.customer.email],
    )
    Order.objects.filter(pk=order_id, shipping_email_sent_at__isnull=True).update(
        shipping_email_sent_at=timezone.now()
    )

This doesn’t give you exactly-once delivery. A crash between sending and recording can still lead to a second email. But it turns duplicates from a routine event into a rare one, and the timestamp gives support a straight answer to “did we email this customer?”.

Testing gets simpler too

A service function is tested like any other function: call it and assert on the result. There’s no need to disconnect receivers in tests or work out which signals fired:

@pytest.mark.django_db
def test_shipping_queues_one_email(paid_order, staff_user, django_capture_on_commit_callbacks, mocker):
    send = mocker.patch("orders.services.send_shipping_email.delay")

    with django_capture_on_commit_callbacks(execute=True):
        mark_order_shipped(paid_order.pk, "TRK123", actor=staff_user)

    send.assert_called_once_with(paid_order.pk)

A normal django_db test runs inside a transaction that is rolled back, so on_commit callbacks would never fire on their own. pytest-django’s django_capture_on_commit_callbacks fixture captures them, and with execute=True runs them as the block exits, emulating a commit. (mocker comes from the pytest-mock plugin.)

The failure path is just as direct, and it’s the test that would have caught the duplicate emails:

@pytest.mark.django_db
def test_shipping_twice_is_rejected(paid_order, staff_user, django_capture_on_commit_callbacks, mocker):
    send = mocker.patch("orders.services.send_shipping_email.delay")

    with django_capture_on_commit_callbacks(execute=True):
        mark_order_shipped(paid_order.pk, "TRK123", actor=staff_user)

    with pytest.raises(InvalidTransition):
        mark_order_shipped(paid_order.pk, "TRK123", actor=staff_user)

    assert send.call_count == 1

Where signals are still the right tool

Signals remain useful when the code that sends them can’t know about the code that receives them:

• Reacting to events from Django itself or third-party apps, such as user_logged_in, request_finished or a payments package’s webhook signal. You can’t edit that code, so a signal is the intended extension point.
• Cross-cutting technical concerns that should apply to every save, such as cache invalidation or search indexing, as long as they are idempotent and tolerate the gaps around bulk updates.
• Letting an optional app hook into a core app without the core app importing it.

When you do keep a receiver, a few habits prevent most surprises. Connect it in the app’s AppConfig.ready(), and pass dispatch_uid if the connecting code could run more than once. Return early when raw is true so fixture loading doesn’t trigger it. And wrap anything that talks to the outside world in transaction.on_commit().

The distinction is between technical plumbing, which signals handle well, and business behaviour, which should be called explicitly.

Migrating away from existing receivers

You don’t need to remove every signal in one go. Start with the receivers that cause incidents:

1. Make an inventory. grep -rn "@receiver\|\.connect(" --include="*.py" . usually finds them all. For each, note the signal, the sender and what it does.
2. Sort them. Mark each one as business rule (emails, payments, stock, state changes) or technical plumbing (cache, search index, metrics).
3. Find every trigger. For each business receiver, list the places that save that model: views, admin, API, tasks, commands. Those are the call sites for the new service function.
4. Move one receiver per change. Write the service function, call it from the places that should trigger the behaviour, and delete the receiver in the same change. Shipping both at once is how the duplicate emails come back.
5. Check the bulk paths. Look for update() and bulk_create() on the same model. They never fired the receiver, so decide explicitly whether they should call the service now.
6. Keep the plumbing. Leave the remaining technical receivers in place, but make sure they check raw and defer external calls with on_commit().

A quick checklist

• Every business event has one service function, and every entry point calls it
• State changes are checked as transitions, under select_for_update() inside atomic()
• External side effects are queued with on_commit() (or delay_on_commit())
• Tasks receive primary keys and are safe to run twice
• Tests use django_capture_on_commit_callbacks and cover the failure path
• Remaining receivers are technical, idempotent, check raw and live in ready()

Conclusion

Signals hide behaviour from the code that triggers it, fire for saves regardless of intent, skip bulk writes, and can run before the data is committed. For business events, a service function that does the work explicitly, inside a transaction, with side effects queued after commit, is easier to read, test and change. Keep signals for framework hooks and genuinely cross-cutting plumbing.

Lycore builds and maintains production systems on Django, React, Flutter and .NET. Get in touch at lycore.com if your Django codebase has started doing things nobody can explain.

Python 3.15.0 shipped on 9 October 2026. Two weeks later, on 25 October, the core team pushed out 3.15.2. It was an expedited security release that blocks the compromised eMudhra intermediate CA keys in the ssl module and fixes a tarfile path-handling issue. If your team runs Django apps, Celery workers, or Python services that call LLM APIs, now is the time to plan the upgrade instead of letting it slip into next year.

We have read through the official “What’s new in Python 3.15” document and ran the examples below on a 3.15 interpreter. This post skips the full changelog. It covers what matters for teams shipping web and AI backends: what can break, which new features are worth adopting, and the order we would roll it out in.

Don’t standardize on 3.15.0. Target the latest 3.15 patch release, which is 3.15.2 at the time of writing. Python 3.15.1 (October 21) was the first maintenance release, with around 330 bug fixes, and an expedited security release: its bundled pip includes an updated certifi to deal with the eMudhra / emSign CA certificate incident. Four days later, Python 3.15.2 followed with around 130 more fixes and made the ssl module itself block the compromised eMudhra intermediate CA keys (CVE-2026–115557), along with a tarfile filter fix (CVE-2026–114583). Pin your Docker images and CI to 3.15.2 or newer. In existing virtual environments, upgrade pip (and uv, if you use it), because those environments don’t pick up the new bundled pip on their own.

Why this release matters more than usual

Most Python minor releases are easy to adopt. You bump the version, run the tests, fix a deprecation or two, and move on. Python 3.15 is mostly like that, with three exceptions that make it worth a deliberate plan:

1. A default changed. Python now uses UTF-8 as the default text encoding, whatever the system locale says (PEP 686).
2. Language syntax changed. There is a new lazy soft keyword for imports (PEP 810), plus unpacking inside comprehensions (PEP 798).
3. Production tooling got much better. A new statistical sampling profiler called Tachyon ships in the standard library as profiling.sampling. It can attach to a running process by PID.

There is also a timing detail for Django teams. According to the Django installation FAQ, Python 3.15 support arrives in Django 6.1.3. Django 6.0 supports Python 3.12 to 3.14 only. The Django 6.1.3 release notes list it as expected on 3 November 2026, with “compatibility with Python 3.15”. So the order is: get on Django 6.1 first, then move the interpreter.

Part 1: What can break

The UTF-8 default

This is the change most likely to bite, and it does so quietly. Any open() call without an encoding argument now uses UTF-8 instead of the locale's preferred encoding:

# Before 3.15: encoding depended on the OS locale (often cp1252 on Windows)
# From 3.15: always UTF-8 unless you say otherwise
with open("export.csv") as f:
    rows = f.read()

On Linux containers with a UTF-8 locale, you probably won’t notice anything. The risk sits in places like these:

• Windows developer machines or Windows-hosted jobs that read files produced by legacy tools
• CSV exports that customers open in Excel and send back to you
• Old fixtures or data files that were saved in a regional encoding years ago

The fix is the same advice that has applied for years: always pass encoding explicitly. To find the calls that rely on the default, turn on the opt-in encoding warning, which has existed since Python 3.10:

python -X warn_default_encoding -m pytest
# or
PYTHONWARNDEFAULTENCODING=1 python -m pytest

On 3.15, every implicit-encoding open() emits an EncodingWarning that points to the exact line. If you need the old behaviour as a stopgap, the docs say you can turn UTF-8 mode off with PYTHONUTF8=0 or -X utf8=0. Treat that as a temporary bridge, not a fix.

Smaller API changes worth grepping for

The “Porting to Python 3.15” section is short, but a few items show up in real codebases:

• sqlite3.connect(): every parameter except database is now keyword-only. sqlite3.connect("db.sqlite3", 5.0) now raises TypeError. Write timeout=5.0 instead. Scripts and test helpers often use the positional form.
• argparse: with add_argument('-f', '-foo'), dest is now inferred from the single-dash long option, so you get foo instead of f. Pass dest explicitly if any code reads args.f.
• base64.urlsafe_b64decode() no longer requires padding. If you relied on it rejecting unpadded input (for example, as part of validating tokens), pass padded=True.
• unittest: assertWarns() and assertWarnsRegex() no longer swallow warnings that don't match. Your test output may get noisier, and suites that treat warnings as errors may start failing.
• profile module: deprecated and scheduled for removal in 3.17. cProfile stays as an alias for the new profiling.tracing.

None of these is hard to fix. They are just easier to fix before a deploy than during one.

Dependencies are the real gate

Your own code is rarely what blocks a Python upgrade. Your dependencies are. Anything with compiled extensions, such as database drivers, crypto libraries, numeric and ML stacks, and some tokenizer packages, needs wheels built for 3.15. Before you schedule production work, check that each critical package has published 3.15 wheels or at least installs cleanly from source in your build image.

Part 2: Features worth adopting

Lazy imports: faster startup without scattering imports

AI-heavy Python services often have a startup problem. Importing an LLM SDK, a vector store client, pandas, and a few cloud SDKs at module level can make every management command, every worker boot, and every test run pay that cost, even when the code path never uses them.

The usual workaround was to move imports inside functions. It works, but imports end up scattered across the codebase. Python 3.15 adds an explicit alternative:

# services/summarize.py
lazy import json
lazy from openai import OpenAI      # not loaded until first use

def summarize(text: str) -> str:
    client = OpenAI()                # the module actually loads here
    ...

We checked the behaviour on 3.15: after lazy import json, "json" in sys.modules is False until the first attribute access, and True right after it.

Know the rules before you sprinkle lazy around:

• It is only allowed at module scope. Using it inside a function, a class body, or a try/except block is a SyntaxError.
• lazy from x import * and lazy __future__ imports are not allowed.
• If the module fails to load, the error is raised at first use, not at import time. The traceback shows both locations, but in a web app this can move a startup crash into the middle of a request. For anything critical, a smoke test that touches the lazy names at boot is a cheap safeguard.

If you need to support older Pythons, list the module names in __lazy_modules__. On 3.15, regular imports of those modules become lazy, and older versions simply ignore the variable:

__lazy_modules__ = ["pandas", "openai"]
import pandas   # lazy on 3.15+, eager on older versions
import openai

There is also a global mode (-X lazy_imports=all or PYTHON_LAZY_IMPORTS=all) and a filter hook, sys.set_lazy_imports_filter(). The What's New document shows the filter being used to make only your own application's modules lazy while third-party packages stay eager. That is a sensible middle ground for a large Django project.

Measure before and after. python -X importtime manage.py check already shows where import time goes, so you can confirm whether lazy imports actually help your startup path rather than guessing.

frozendict: immutable config you can hash

frozendict is now a built-in. It is immutable, preserves insertion order, ignores order when comparing, and is hashable when its keys and values are hashable:

MODEL_DEFAULTS = frozendict(model="small", temperature=0.2, max_tokens=800)

MODEL_DEFAULTS["temperature"] = 0.9
# TypeError: 'frozendict' object does not support item assignment

# Hashable, so it can be a cache key
cache = {MODEL_DEFAULTS: "cached-response"}

That fits module-level settings, feature-flag snapshots, and prompt or model parameter sets that should never change at runtime. json.dumps() handles it directly, which we confirmed. One thing to watch: frozendict is not a subclass of dict, so isinstance(x, dict) returns False. If you have validation code that checks for dict, update it to isinstance(x, (dict, frozendict)) or check against collections.abc.Mapping.

sentinel: a better "missing" value

Many codebases have something like _MISSING = object() to tell "not passed" apart from None. The new built-in sentinel does the same job with a readable repr, identity that survives copying and pickling (when defined at module or class scope), and support in type hints:

MISSING = sentinel("MISSING")

def update_profile(user_id: int, bio: str | None | MISSING = MISSING):
    if bio is MISSING:
        return            # field not sent: leave it alone
    save_bio(user_id, bio)  # None means "clear it"

PATCH-style API endpoints, where “absent” and “explicitly null” mean different things, are exactly where this helps.

Unpacking in comprehensions

A small but welcome change: * and ** now work inside comprehensions.

batches = [[1, 2], [3], [4, 5]]
flat = [*b for b in batches]            # [1, 2, 3, 4, 5]

layers = [{"model": "small", "temperature": 0.2}, {"temperature": 0.7}]
merged = {**d for d in layers}          # later dicts win
# {'model': 'small', 'temperature': 0.7}

It reads better than nested comprehensions or itertools.chain.from_iterable() for merging config layers or flattening batched results.

Part 3: Tachyon, the profiler you can use in production

For teams running real traffic, this may be the most useful part of the release. profiling.sampling (Tachyon) samples stack traces from a running process instead of instrumenting every call. The docs describe it as low overhead and suited to debugging in production, with no code changes or restarts.

# Profile a running Gunicorn or Celery worker for 30 seconds into a flame graph
python -m profiling.sampling attach --flamegraph -d 30 -o profile.html 12345

# One-shot stack dump of a stuck process (like a traceback for every thread)
python -m profiling.sampling dump 12345

# Live, top-like view
python -m profiling.sampling attach --live 12345

Three features stand out for web and AI workloads:

• Modes. --mode wall (the default) includes time spent waiting on I/O. --mode cpu counts only on-CPU time. For a slow endpoint that calls an LLM, wall mode shows you the network wait, while CPU mode shows whether your own parsing or serialization is the real problem.
• --async-aware rebuilds stacks across await boundaries. That matters a lot for async Django views and asyncio-based agent loops.
• dump answers the 2 a.m. question: what is this hung worker doing right now?

Two practical caveats from the profiling.sampling documentation. First, attaching to another process needs elevated permissions: root, CAP_SYS_PTRACE, or a relaxed Yama ptrace_scope on Linux. In containers, that usually means a deliberate debug setup instead of something you can just run. Second, the profiler and the target must run the same Python minor version, so you can't attach from 3.14 to a 3.15 process. Plan your debug sidecar or image to match.

Part 4: The JIT, with realistic expectations

The experimental JIT got a big upgrade in 3.15. The release notes report a 7–8% geometric mean improvement over the standard interpreter on x86–64 Linux, and 11–12% over the tail-calling interpreter on AArch64 macOS, measured with pyperformance. The same section notes that individual benchmarks range from roughly a 15% slowdown to more than a 100% speedup.

That range is the key point. A geometric mean across a benchmark suite tells you little about a Django app that spends most of its time waiting on PostgreSQL and an LLM API. The JIT is still experimental. Update: the JIT’s future in CPython is uncertain. According to InfoWorld, the Python Steering Council had paused JIT work until a supported-JIT proposal was approved, and on October 19 the authors withdrew PEP 836, so the experimental JIT is being removed from the mainline interpreter. Treat it as a curiosity, not a foundation for production plans. Whether it is available depends on how your interpreter was built (--enable-experimental-jit), and on builds that include it, PYTHON_JIT=1 or PYTHON_JIT=0 controls it at startup. Our advice: leave it off in production and try it on a CPU-heavy batch job or a benchmark of your own hot path. Ship it only if your numbers, not the release notes, justify it.

Our upgrade order

Here is the sequence we would follow for a typical Django plus Celery plus AI-integration codebase:

1. Get green on 3.14 with warnings as errors. Run python -W error::DeprecationWarning -m pytest. Anything deprecated earlier and removed in 3.15 shows up here first.
2. Fix implicit encodings. Run the suite with -X warn_default_encoding and add encoding="utf-8" (or the correct legacy encoding) wherever it is flagged.
3. Upgrade Django to 6.1.x. Django’s 3.15 support lands in 6.1.3, so this step comes before the interpreter change.
4. Add 3.15 to CI as a non-blocking job. Watch for wheel availability, the sqlite3 and argparse changes, and new test warnings. Make it blocking once it stays green.
5. Use the latest patch release. Start from 3.15.2 or later, since it contains the security fixes and a fix for an import deadlock regression introduced in 3.15.0.
6. Roll out by workload. Start with background workers and internal tools, then the web tier. Keep the previous image available for a quick rollback.
7. Adopt new features one at a time. frozendict for config, sentinel for PATCH semantics, and lazy imports where -X importtime shows a real startup cost. Avoid a big-bang refactor.

If you are also reviewing your deploy pipeline as part of this work, we covered the wider picture of deploying and scaling a production Django app separately. Most of that checklist applies here too.

Conclusion

Python 3.15 is a solid release with a few sharp edges. The UTF-8 default is the change most likely to cause a quiet production bug, and you can find every affected line today with one interpreter flag. Lazy imports, frozendict, and sentinel are useful additions that don't require a big rewrite. Tachyon gives Python teams a sampling profiler in the standard library that can attach to a live process, which is valuable for anyone debugging slow AI endpoints or stuck workers.

Don’t rush it, and don’t put it off. Start the CI job this week, wait for Django 6.1.3 and your dependencies to catch up, and roll out in stages. The release notes describe the changes, but only your test suite can tell you how they affect your code.

Sam Altman has a point about RAM. He also has a business that benefits from you agreeing with him. Both things can be true, and if you build software for a living, the useful question is not who wins the argument online. It is where your own inference workloads should run, given a memory market that is tight and AI vendors whose futures are still being written.

What Altman actually said

On the Cheeky Pint podcast with Stripe cofounder John Collison, Altman was asked about supply constraints. As quoted by The Verge and Tom’s Hardware, he first described the hope many people have for local inference: that open-source models running on their own computers mean “we’re not going to need all the stuff in the data center.” Then he said:

“From a global allocation perspective, actually, it’s quite terrible to have everybody have a bunch of the world’s limited supply of DRAM sitting in their own computer and being used 5% of the time. You want maximum utilization in the cloud if you’re trying to deal with this very supply-constrained environment. And somehow many people’s intuition is the opposite of that.”

Note what he did not say. He did not say PCs are terrible, or that you should not own a computer. His claim is about allocation: a scarce resource sitting idle most of the time is wasteful compared with the same resource kept busy in a shared data center. As an economic argument about utilization, that is hard to dispute.

The context: memory is expensive right now

The comment landed during a painful stretch for anyone buying memory. TweakTown reports that the cheapest 32GB DDR5–6000 kit it tracked had climbed to $449, that a Corsair 32GB DDR5–4800 kit that once sold for $80 now costs $399, and that TrendForce forecasts conventional DRAM contract prices to rise another 10% to 15% in the fourth quarter of 2026 as suppliers prioritize AI server memory. The Verge adds that Samsung and SK Hynix have long-term deals with cloud operators and projects like OpenAI’s Stargate that have reserved much of the world’s RAM output over the next few years.

So the person arguing that DRAM belongs in the data center runs one of the companies whose demand is helping make DRAM scarce. That does not make the argument wrong. It does mean you should check it against your own numbers rather than take it on authority.

A counterpoint from Elon Musk

Musk took the memory debate in a different direction. In a reply on X reported by PC Gamer, defending Counterpoint research on robot memory demand, he wrote:

“Robot RAM needs will exceed everything else on Earth, as you need the robots to be on physical substates in order to shape atoms. Super intelligence needs will vastly exceed that, but ~100% of SI will be deep space compute. Obviously.”

PC Gamer points out the catch in its own headline: the claim only works if the data centers move to space. Still, the underlying idea is relevant here. Some compute has to sit next to the physical world, close to sensors and actuators, because the round trip to a distant data center is too slow or too fragile. That is an argument for local inference, just at the robot end of the spectrum rather than the laptop end.

A note on vendor risk

Short context paragraph, not investment advice. On October 30, TechCrunch reported, citing the Financial Times, that Blackstone told investors in a chip financing deal that it now expects Anthropic to receive a double-B range rating, the top tier of speculative “junk” grade, after its IPO. That is a forecast by Blackstone, not a rating issued by any credit agency, and both Anthropic and Blackstone declined to comment to TechCrunch. We draw no conclusion about Anthropic from it. The practical lesson is general: even the strongest AI vendors are businesses with large capital needs, changing prices and changing terms. If your product depends on one hosted model, you are carrying that vendor’s risk whether or not you think about it.

Opinion: the case for local inference anyway

Everything from here on is our opinion, based on the reasoning laid out below.

Altman is right that idle memory is wasteful at a global level. But engineering teams do not optimize global DRAM allocation. They optimize for their users, their data and their risk. On those terms, local or on-prem inference is often the better choice, and in other cases renting cloud capacity clearly wins. The trick is knowing which case you are in.

When local or on-prem inference makes sense

• Privacy and data residency. If prompts contain health records, legal documents, source code or customer data you are not allowed to send to a third party, keeping the model inside your boundary removes a whole category of risk and paperwork.
• Latency and offline use. Autocomplete, on-device assistants, field apps and anything on a factory floor benefit from not waiting on a network. This is the same logic as Musk’s robots, at a smaller scale.
• Steady, predictable utilization. Altman’s own argument cuts both ways. Hardware idling at 5% is wasteful, but hardware you keep busy around the clock with a steady internal workload, like classification, embedding or document extraction, is not idle. High utilization is the point of owning it.
• Vendor risk. An open-weight model you run yourself cannot be deprecated, repriced or rate-limited out from under you. It can be a fallback even if it is not your primary path.
• Small, specialized models. Many production tasks do not need a frontier model. A smaller model tuned for one job may run comfortably on modest hardware.

When renting cloud capacity is better

• Spiky or unpredictable demand. If traffic swings wildly, paying per request is usually more sensible than buying for the peak and idling the rest of the time. This is exactly the utilization argument Altman made, and it is correct here.
• You need frontier capability. The largest hosted models are not something most teams can run themselves, and for hard reasoning or broad knowledge tasks the quality gap can matter more than cost.
• Early-stage products. Before you know your usage pattern, committing to hardware is guessing. Rent first, measure, then decide.
• No one to run it. Self-hosted inference is infrastructure. It needs patching, monitoring, capacity planning and someone on call. If you do not have that, the cloud is buying you an operations team.

A simple decision framework

For each AI workload, not for the company as a whole, answer five questions in order:

1. Can this data leave our boundary? If no, the answer is local or on-prem, or a hosted option with contractual and technical controls you have actually verified. Stop here.
2. Does the task need a frontier model? Test a smaller open-weight model against real examples. If it fails your quality bar, rent the frontier model and revisit later.
3. Is the load steady or spiky? Steady and high favors owning. Spiky and low favors renting.
4. Is latency or offline operation a product requirement? If yes, push inference toward the edge.
5. What happens if our vendor changes price, terms or availability next quarter? If the honest answer is “the product stops,” you need a second path, either another vendor or a local fallback.

Most teams will end up with a mix: a hosted frontier model for the hardest requests, a cheaper hosted or local model for routine ones, and an abstraction layer in the code so swapping between them is a configuration change rather than a rewrite.

A cost-thinking checklist

We are deliberately not quoting prices here, because they are moving fast and depend heavily on your volumes and region. Instead, make sure your comparison includes all of these:

• Utilization. Estimate how many hours a day the hardware would actually be busy. Be honest. This is the number Altman’s argument rests on.
• Full hardware cost. Not just GPUs, but memory, which TweakTown’s figures show is not cheap right now, plus power, cooling, hosting and replacement cycles.
• People cost. Engineering time to deploy, monitor, update models and handle incidents.
• Per-request cloud cost at realistic volume. Include input and output tokens, retries, long contexts and any premium for higher rate limits.
• Quality cost. A cheaper model that needs more retries, more human review or produces more support tickets is not cheaper.
• Compliance cost. Reviews, contracts and audits for sending data to a third party, versus the controls required to run models yourself.
• Switching cost. How much code assumes one vendor’s API, prompt format or tool-calling style.
• Risk cost. What an outage, price change or deprecation would cost you, multiplied by how likely you think it is.

What we would do this quarter

Put a thin provider interface between your application and every model call. Keep at least one open-weight model running somewhere, even at small scale, and run your evaluation set against it regularly so you know what it can and cannot do. Log usage per workload so the “steady or spiky” question has data behind it. Then revisit the decision every quarter, because memory prices, model quality and vendor terms are all moving.

If you are planning this kind of setup, it is the sort of work our AI development team does with clients, including projects where data cannot leave their infrastructure.

The bottom line

Altman is right that a world where everyone’s RAM sits idle is inefficient. But your architecture is not a global allocation problem. It is a set of specific workloads with specific privacy, latency, quality and risk requirements. For some of them, renting the busiest, biggest model in someone else’s data center is the right call. For others, the memory sitting on your own hardware, kept busy, is not wasted at all. It is insurance.

Sources

• The Verge: Sam Altman says it’s better for RAM to be in a data center than on your desk (Oct 30, 2026)
• Tom’s Hardware on Altman’s Cheeky Pint comments (Oct 30, 2026)
• TweakTown on memory prices and the TrendForce forecast (Oct 30, 2026)
• PC Gamer on Elon Musk’s robot RAM comment (Oct 28, 2026)
• TechCrunch on the FT report about Blackstone’s rating expectation (Oct 30, 2026)

For about eleven hours on October 29, developers who installed or auto-updated the Claude Code extension from the Cursor VS Code Extension Marketplace on Windows or macOS may have received something extra: a backdoor that tried to give a stranger SSH access to their machine. Anthropic did not build it. Anthropic’s own systems were not compromised. But the incident is a clean, well-documented example of a risk most teams still underrate: the code editor is now one of the most privileged pieces of software in the company, and it installs third-party code with very little friction.

This post walks through what is confirmed, what the malicious code did, how to check your own machines, and the controls we think every engineering team should have in place before the next one.

What happened: the timeline

According to Cursor’s security notice, maliciously modified builds of the extension Anthropic.claude-code, versions 2.1.312 and 2.1.313 for Windows and macOS, were available in the Cursor VS Code Extension Marketplace between about 2:27 AM PT and 1:35 PM PT on October 29, 2026. Cursor states plainly that these versions "were not developed by Anthropic."

Cursor says it removed the malicious version as soon as it was detected, then shut down the whole marketplace “out of an abundance of caution,” audited access, confirmed no other extensions were tampered with, rotated its keys, rebuilt the marketplace and brought it back online. It also worked with the hosting provider to take the attacker’s server offline. Cursor updated the notice on October 30 to refine the indicators and clarify the affected versions.

On October 30, Microsoft Security Research published its own analysis. Microsoft says it confirmed the malicious packages were never published or distributed from the Visual Studio Marketplace or from npm, that Anthropic confirmed to Microsoft its systems were not compromised, and that “all evidence suggests that the incident is limited to Cursor marketplace.”

The endpoint security vendor Phorion published a detailed technical advisory focused mostly on macOS. Phorion’s write-up is a single vendor’s analysis, so we cite it for the technical detail it adds rather than for any broader claims. Phorion itself updated its advisory on October 31 to match Cursor’s list of affected versions.

What has not been published by these sources: who did it, and how the malicious build got into the marketplace. We are not going to guess.

What the malicious code did

All three sources describe the same basic chain. Microsoft notes that the sequence “reflects code execution order; it does not prove that every step succeeded on every host.” With that caveat, the injected code attempted to:

1. Collect host details. Cursor lists hostname, username, OS, architecture, home directory path, environment variable names, working directories, configured Git email and public IP. Microsoft and Phorion both stress that it collected the names of environment variables matching patterns like KEY, TOKEN, SECRET, PASS, AWS or GCP, up to forty of them, not their values. In Phorion’s words, it gives the operator “a quick inventory” of which credentials live on the machine.
2. Plant an SSH key. It appended an attacker-controlled public key, labeled svc-telemetry, to the user's ~/.ssh/authorized_keys.
3. Try to turn on SSH and open a reverse tunnel. It attempted to enable the local SSH service and open an outbound reverse SSH tunnel to the attacker’s server, so the attacker could reach the machine’s SSH port from outside.
4. Install persistence. On macOS, a user LaunchAgent labeled com.apple.sysmond. On Windows, a scheduled task named SysMon that tries to run as SYSTEM, with a fallback script in the user's Startup folder. Background scripts retry the tunnel roughly every 60 seconds, according to Cursor.
5. Beacon home. Send the collected host details to the attacker’s server.

The names are worth noticing. svc-telemetry, SysMon and com.apple.sysmond are all chosen to look like boring system or monitoring components. Microsoft also notes that the loader "suppresses command output and silently catches errors" while the extension keeps working normally, so users had no visible sign anything was wrong.

The part that partly failed

Cursor says that on most macOS and Windows hosts the step that turns on the local SSH service fails, because it needs admin rights the extension does not have, and because of errors in the malicious code. Phorion documents one of those bugs in detail: on macOS the script checks the Remote Login state without elevation, gets an “administrator access” error instead of “Off,” and so never even attempts to enable SSH.

That is good news, but it is not the end of the story. Cursor is explicit: “The extension added the attacker’s SSH key, installed persistence, and sent host details to the attacker. Follow the remediation steps below regardless.” Microsoft makes the same point: the loader “may still write keys and persistence artifacts even when tunnel access fails.” If a machine already had Remote Login or an SSH server enabled, or someone turns it on later, the planted key becomes a working door. And because the persistence lives outside the extension folder, uninstalling the extension is not enough.

How to check your own machines

If anyone on your team installed or updated Anthropic.claude-code 2.1.312 or 2.1.313 from the Cursor marketplace on macOS or Windows during the window on October 29, start with Cursor's notice. It publishes a one-line check for the svc-telemetry key on macOS and Windows, downloadable remediation scripts for each platform, and the full list of indicators of compromise, including file hashes, the attacker IP and the key fingerprint. Use the official check and scripts rather than copies circulating on social media, and, as Microsoft advises, review the scripts before running them.

Microsoft’s guidance adds two useful points. First, preserve evidence before cleanup if you have a security team that will investigate. Second, do not rely on the comment string alone: confirm the key against the published fingerprint, because a missing marker “should not replace review of the installation, process, network, and persistence evidence.”

Separately from this incident, these are generic hygiene checks worth doing on any developer machine from time to time:

• Read your ~/.ssh/authorized_keys and make sure you recognize every line. Many laptops should have no entries at all.
• On macOS, look through ~/Library/LaunchAgents for anything you did not install. A name that looks like an Apple component is not proof it is one.
• On Windows, review Task Scheduler and your Startup folder for entries you cannot explain.
• Check whether Remote Login (macOS) or OpenSSH Server (Windows) is on, and turn it off if you do not need it.

If you find the indicators, the published guidance from both Cursor and Microsoft is consistent: isolate the host, remove the key, persistence and tunnel processes, consider rebuilding the machine, and rotate credentials and sessions that were reachable from it, from a different, trusted machine. Cursor notes that the limited environment variable collection “does not rule out later credential access over the backdoor.”

Team controls that would have reduced the damage

An eleven-hour window is short. Most of the controls below work by making sure your team is not among the first people to install a brand new version of anything.

1. Approved publishers and an install cooldown

Microsoft’s top recommendation for organizations using Cursor is to apply Cursor’s security-hardening guidance, “including restricting extensions to approved publishers and configuring an extension installation cooldown to delay installation of newly published versions.” A cooldown of even a day or two would have kept most machines out of this window entirely, because the bad versions were pulled within hours.

2. An extension allowlist

Treat IDE extensions like dependencies. Keep a short, reviewed list of extensions your team actually needs, block the rest through managed settings where your editor supports it, and review additions the way you would review a new package in production code.

3. Least privilege on developer machines

This incident is a direct case study for not running daily work as an administrator. The SSH-enable step failed on most machines precisely because the extension did not have admin rights. Cursor notes that the exception was “highly unusual circumstances where Cursor was started with administrator privileges.” Keep it that way. Also keep SSH servers off on laptops unless there is a clear reason.

4. Short-lived credentials, rotated from a trusted machine

The beacon collected environment variable names because long-lived secrets sitting in a shell environment are valuable. Prefer short-lived tokens from an identity provider or secrets manager over static keys in .env files and shell profiles. When something like this happens, rotate from a machine you trust, not from the one you are worried about.

5. Monitoring that covers workstations

Microsoft and Phorion both describe detections built on endpoint telemetry: new entries in authorized_keys, new LaunchAgents and scheduled tasks, unexpected outbound SSH. If your monitoring stops at servers and cloud accounts, your developers' laptops are a blind spot. At minimum, track which extensions and versions are installed across the team so you can answer "were we exposed?" in minutes, not days.

6. A short incident playbook for supply-chain alerts

Decide in advance who checks the inventory, who runs the official remediation, and who rotates which credentials. When the advisory lands, you want a checklist, not a meeting.

A pattern, not a one-off

Earlier this week we wrote about the Binance hack, which reportedly started with one compromised laptop. The two incidents are very different, and nothing published links them. But they point at the same place: the individual engineer’s machine, where source code, cloud credentials, SSH keys and AI coding agents all live together.

AI coding tools make this more pressing, not less. They are popular, they update often, and teams install them quickly because they are useful. That is exactly the profile an attacker wants to imitate. If you are weighing which assistant to standardize on, we compared a couple of the main options in Claude Code vs GitHub Copilot. Whatever you choose, how you install and update it matters as much as which one you pick.

Opinion: workstations are the new perimeter

This section is our opinion.

For years, security budgets have focused on production: servers, clusters, cloud accounts. The developer laptop got antivirus and disk encryption and was otherwise trusted. We think that model is out of date. A modern developer workstation holds source code, deploy keys, cloud sessions, and now AI agents that can run commands on the developer’s behalf. Anything that runs inside the IDE inherits all of that. In practice, your IDE is production.

That does not mean locking developers down until they cannot work. It means applying the boring controls we already use for production, like allowlists, delayed rollouts, least privilege, short-lived credentials and monitoring, to the place where production access actually starts. The teams that do this will treat the next malicious extension as a fifteen-minute inventory check. The teams that do not will spend a week wondering what an attacker could have reached.

Sources

• Cursor: Security notice on Anthropic.claude-code 2.1.312 and 2.1.313 (Oct 29, updated Oct 30, 2026)
• Microsoft Security Blog: Inside a malicious Claude Code-branded extension and its persistent SSH backdoor (Oct 30, 2026)
• Phorion: Official Claude Code Extension Compromise advisory (Oct 30, updated Oct 31, 2026)

It usually starts with one bad afternoon. Someone on the support team deletes the wrong customer, and the only way back is a restore from last night’s backup, followed by hours of picking that customer’s rows out of the restored copy and putting them back in production. Once it’s over, the team agrees it must never happen again, and someone adds a deleted_at column to every model.

For a while it works. Then, months later, the problems start. Deleted customers show up in a finance export, because the export query was written before the change and nobody updated it. A user who closed their account can’t sign up again: their old, deleted account still holds the email address, and the unique constraint doesn’t know the difference. Then the order history page crashes with “Customer matching query does not exist” for an order whose customer was soft-deleted.

Soft deletes are a reasonable tool. The trouble is that a deleted_at column quietly changes the meaning of every query, constraint and relation that touches the table, and most teams find that out one bug at a time. This post covers those costs, a pattern that keeps them in one place, and the parts that still need care.

What soft deletes change

When a row can be “deleted” but still exist, a few things stop being true:

• Every query has to exclude deleted rows. Every list view, search, report, export and count now needs deleted_at IS NULL. Miss it once and deleted data reappears, and that kind of bug rarely fails a test because the test data has no deleted rows in it.
• Unique constraints break. A plain unique constraint on email counts deleted rows too, so a deleted account blocks a new sign-up with the same address.
• Related rows don’t cascade. With a real delete, Django’s on_delete rules decide what happens to orders, addresses and tokens. A soft delete is just an UPDATE, so nothing happens to related rows unless you write the code for it.
• Erasure obligations remain. If you are subject to GDPR or a similar law, a soft-deleted customer’s personal data is still personal data you hold. Hiding a row is not erasing it.
• Tables and indexes grow. Deleted rows stay in the table and in every index, and most queries filter them out anyway. On tables with a lot of churn, that dead weight adds up.

A pattern that contains the costs

The aim is to make the safe behaviour the default, so a developer who writes Customer.objects.filter(...) gets active rows without having to remember anything, while deleted rows are still reachable when you ask for them explicitly.

from django.db import models
from django.utils import timezone

class SoftDeleteQuerySet(models.QuerySet):
    def delete(self):
        # Bulk soft delete: one UPDATE, no per-object delete() calls.
        return self.update(deleted_at=timezone.now())

    def hard_delete(self):
        return super().delete()

    def alive(self):
        return self.filter(deleted_at__isnull=True)

    def dead(self):
        return self.filter(deleted_at__isnull=False)

class ActiveManager(models.Manager.from_queryset(SoftDeleteQuerySet)):
    def get_queryset(self):
        return super().get_queryset().filter(deleted_at__isnull=True)

class SoftDeleteModel(models.Model):
    deleted_at = models.DateTimeField(null=True, blank=True, editable=False)

    objects = ActiveManager()                       # first, so it is the default manager
    all_objects = SoftDeleteQuerySet.as_manager()   # everything, including deleted rows

    class Meta:
        abstract = True

    def delete(self, using=None, keep_parents=False):
        self.deleted_at = timezone.now()
        self.save(update_fields=["deleted_at"])

    def hard_delete(self, using=None, keep_parents=False):
        return super().delete(using=using, keep_parents=keep_parents)

    def restore(self):
        self.deleted_at = None
        self.save(update_fields=["deleted_at"])

Order matters here. Django treats the first manager declared on a model as its default manager, and several parts of the framework use the default manager, including the admin’s change list and dumpdata. Declaring objects = ActiveManager() first means those all see active rows only. all_objects is the explicit escape hatch for the places that really need everything: support tools, restore flows and purge jobs.

Two details are easy to miss:

• QuerySet.delete() is a bulk operation and never calls delete() on your model instances. That is why the queryset needs its own override. Without it, Customer.objects.filter(...).delete() would skip Customer.delete() and permanently remove the rows.
• Our queryset delete() returns the number of rows updated, while Django's normally returns a tuple of the total count and a per-model breakdown. Code that unpacks the result of .delete() needs adjusting.

Also bear in mind that update() does not call save() and does not send pre_save or post_save signals. If something in your project listens for those signals (search indexing, cache invalidation, audit logging), a bulk soft delete won't trigger it. Either loop over the objects and call delete() on each one, or have the queryset send whatever event your listeners need.

dumpdata deserves a note of its own. Because it uses the default manager, a fixture or backup made with plain dumpdata will leave out every soft-deleted row. If you use it for anything that needs to be complete, pass --all, which makes it use the base manager instead.

Fix uniqueness with partial constraints

The sign-up bug has a clean fix at the database level: enforce uniqueness only among active rows.

from django.db import models
from django.db.models import Q

class Customer(SoftDeleteModel):
    email = models.EmailField()
    name = models.CharField(max_length=200)

    class Meta:
        constraints = [
            models.UniqueConstraint(
                fields=["email"],
                condition=Q(deleted_at__isnull=True),
                name="uniq_active_customer_email",
            ),
        ]

On PostgreSQL this becomes a partial unique index, CREATE UNIQUE INDEX ... WHERE deleted_at IS NULL. Any number of deleted rows can share an email address, but only one active row can hold it. Not every backend supports conditional constraints, though. Django documents that MySQL and MariaDB ignore the condition, and Oracle doesn't support partial indexes, so check your database before you rely on this.

A few refinements are worth considering:

• Case-insensitive emails. If Alice@Example.com and alice@example.com should count as the same person, use an expression instead of a field: models.UniqueConstraint(Lower("email"), condition=Q(deleted_at__isnull=True), name="uniq_active_customer_email_ci"), with Lower imported from django.db.models.functions.
• Constraints on the abstract base class. Constraint names must be unique, and anything in Meta on an abstract model is inherited unchanged by every subclass. If you put a shared constraint on SoftDeleteModel, include %(app_label)s_%(class)s in the name so each concrete model gets its own.
• Form validation. Django checks constraints during model validation, so a ModelForm reports the duplicate email as a normal form error, and the user doesn't get a 500 from the database.

Restoring now has a sharp edge, and that’s a good thing. If a deleted customer’s email has since been taken by a new active account, restore() fails with an IntegrityError. The database is refusing to let two active customers share an email, which is exactly the rule you wanted. Turn that into a clear message rather than a crash:

from django.db import IntegrityError, transaction

class RestoreConflict(Exception):
    pass

def restore_customer(customer):
    try:
        with transaction.atomic():
            customer.restore()
    except IntegrityError as exc:
        raise RestoreConflict(
            f"{customer.email} now belongs to another active account."
        ) from exc

The atomic() block matters on PostgreSQL. After an error, the current transaction can't be used until it's rolled back, and the savepoint keeps the failure contained.

Know where the manager doesn’t apply

The default manager covers queries that start from Customer.objects. Several common paths never touch it:

• Forward relations. order.customer is loaded through the model's base manager, not the default manager. Django does this on purpose so that a related object can always be fetched. The order page in our story crashed because a view looked the customer up with Customer.objects.get(pk=order.customer_id), which goes through the filtered manager and raises DoesNotExist. Decide what a page should show for a deleted customer (often their name plus a "deleted" label), and don't try to "fix" this by pointing Meta.base_manager_name at a filtering manager. Django's documentation says explicitly not to filter in a base manager.
• Filters across relations. Order.objects.filter(customer__country="PK") joins to the customer table directly, and the join does not apply Customer's default manager. Orders of deleted customers are included unless you add customer__deleted_at__isnull=True yourself.
• Aggregations through relations. Customer.objects.annotate(order_count=Count("orders")) counts soft-deleted orders too. If orders can be soft-deleted, filter inside the aggregate:

from django.db.models import Count, Q

Customer.objects.annotate(
    order_count=Count("orders", filter=Q(orders__deleted_at__isnull=True))
)

• Raw SQL, database views and BI tools. Anything that reads the table without going through the ORM sees every row. For reporting, a database view such as active_customers that applies WHERE deleted_at IS NULL gives analysts a safe default, and you only have to write the filter once.

Write a test for each case. These are exactly where a refactor quietly reintroduces deleted data.

Handle related rows explicitly

Since a soft delete doesn’t cascade, decide per relation what should happen and write it down in code. A useful trick is to give the parent and its children the same timestamp, so a restore can bring back exactly the rows that went with it, and leave alone the ones that were deleted separately before.

from django.db import transaction
from django.utils import timezone

class Customer(SoftDeleteModel):
    # fields and Meta as above

    def delete(self, using=None, keep_parents=False):
        now = timezone.now()
        with transaction.atomic():
            # Reverse relations use the default manager, so rows that
            # were already deleted keep their original timestamp.
            self.addresses.all().update(deleted_at=now)
            self.api_tokens.all().update(deleted_at=now)
            self.deleted_at = now
            self.save(update_fields=["deleted_at"])

    def restore(self):
        with transaction.atomic():
            Address.all_objects.filter(
                customer=self, deleted_at=self.deleted_at
            ).update(deleted_at=None)
            ApiToken.all_objects.filter(
                customer=self, deleted_at=self.deleted_at
            ).update(deleted_at=None)
            self.deleted_at = None
            self.save(update_fields=["deleted_at"])

Not every relation should follow the parent. Orders and invoices are usually business records that need to outlive the customer, so they keep pointing at the soft-deleted row, and the pages that display them handle that state. Access tokens are the opposite: you may want to hard-delete or revoke them straight away, because a soft-deleted token that some code path still accepts is a security bug.

Give staff a view of deleted rows in the admin

The admin builds its change list from the default manager, so out of the box staff can’t see deleted rows at all. Support staff, who get asked to bring customers back, need to see them. A small ModelAdmin covers it:

from django.contrib import admin, messages

class DeletedFilter(admin.SimpleListFilter):
    title = "status"
    parameter_name = "deleted"

    def lookups(self, request, model_admin):
        return [("no", "Active"), ("yes", "Deleted")]

    def queryset(self, request, queryset):
        if self.value() == "yes":
            return queryset.dead()
        if self.value() == "no":
            return queryset.alive()
        return queryset

@admin.register(Customer)
class CustomerAdmin(admin.ModelAdmin):
    list_display = ["email", "name", "deleted_at"]
    list_filter = [DeletedFilter]
    actions = ["restore_selected"]

    def get_queryset(self, request):
        qs = Customer.all_objects.all()
        ordering = self.get_ordering(request)
        return qs.order_by(*ordering) if ordering else qs

    @admin.action(description="Restore selected customers")
    def restore_selected(self, request, queryset):
        restored, conflicts = 0, []
        for customer in queryset.dead():
            try:
                restore_customer(customer)
                restored += 1
            except RestoreConflict as exc:
                conflicts.append(str(exc))
        self.message_user(request, f"Restored {restored} customer(s).")
        for conflict in conflicts:
            self.message_user(request, conflict, level=messages.WARNING)

Because the admin’s built-in “Delete selected” action calls queryset.delete(), it will soft-delete through our queryset. The confirmation page, though, is built by Django's deletion collector, so it may list related objects as about to be deleted when our soft delete won't touch them. If that would confuse your staff, override delete_queryset() and delete_model() to call your explicit soft-delete method, and consider removing the built-in action for this model.

Test the boundaries

The tests worth having are the ones that pin down each edge from the sections above. With pytest and pytest-django:

import pytest
from django.db import IntegrityError
from django.db.models import Count, Q

pytestmark = pytest.mark.django_db

def test_default_manager_hides_deleted():
    customer = Customer.objects.create(email="a@example.com", name="A")
    customer.delete()
    assert not Customer.objects.filter(pk=customer.pk).exists()
    assert Customer.all_objects.filter(pk=customer.pk).exists()

def test_email_can_be_reused_after_delete():
    Customer.objects.create(email="a@example.com", name="Old").delete()
    Customer.objects.create(email="a@example.com", name="New")  # must not raise

def test_restore_conflict_is_rejected():
    old = Customer.objects.create(email="a@example.com", name="Old")
    old.delete()
    Customer.objects.create(email="a@example.com", name="New")
    with pytest.raises(RestoreConflict):
        restore_customer(Customer.all_objects.get(pk=old.pk))

def test_order_still_loads_its_deleted_customer():
    customer = Customer.objects.create(email="a@example.com", name="A")
    order = Order.objects.create(customer=customer)
    customer.delete()
    order = Order.objects.get(pk=order.pk)
    assert order.customer.deleted_at is not None

def test_order_count_ignores_deleted_orders():
    customer = Customer.objects.create(email="a@example.com", name="A")
    Order.objects.create(customer=customer)
    Order.objects.create(customer=customer).delete()
    row = Customer.objects.annotate(
        n=Count("orders", filter=Q(orders__deleted_at__isnull=True))
    ).get(pk=customer.pk)
    assert row.n == 1

Run these against the same database engine you use in production. A conditional constraint that SQLite enforces tells you nothing about MySQL, and the reverse is also true.

Adding soft deletes to an existing table

Retrofitting a live table is mostly about ordering the steps so nothing breaks in between.

Step 1: Add the column. In PostgreSQL, adding a nullable column with no default doesn’t rewrite the table, so this is fast even on large tables.

Step 2: Add the partial unique index before dropping the old constraint. If the table is large and busy, remember that a normal index build blocks writes until it finishes. CREATE INDEX CONCURRENTLY avoids that but can't run inside a transaction, so the migration has to be non-atomic. Django's SeparateDatabaseAndState lets you run the concurrent SQL while still recording the constraint in Django's migration state:

from django.db import migrations, models

class Migration(migrations.Migration):
    atomic = False
    dependencies = [("customers", "0012_customer_deleted_at")]

    operations = [
        migrations.SeparateDatabaseAndState(
            database_operations=[
                migrations.RunSQL(
                    sql=(
                        "CREATE UNIQUE INDEX CONCURRENTLY uniq_active_customer_email "
                        "ON customers_customer (email) WHERE deleted_at IS NULL;"
                    ),
                    reverse_sql=(
                        "DROP INDEX CONCURRENTLY IF EXISTS uniq_active_customer_email;"
                    ),
                ),
            ],
            state_operations=[
                migrations.AddConstraint(
                    model_name="customer",
                    constraint=models.UniqueConstraint(
                        fields=["email"],
                        condition=models.Q(deleted_at__isnull=True),
                        name="uniq_active_customer_email",
                    ),
                ),
            ],
        ),
    ]

Step 3: Drop the old unique=True in a following migration, once the partial index exists. Until then, the old constraint still blocks reused emails, which is safe, just strict.

Step 4: Audit the callers before switching the managers on. Search for .delete(, raw SQL, annotate( with counts across relations, exports, and anything that unpacks the return value of delete(). Each one is a decision: active rows, all rows, or a real delete.

Step 5: Only then deploy the model change that makes objects the filtered manager.

Consider the alternatives first

Soft deletes are often used where a different tool would fit better:

• A status field for business states. A cancelled subscription or a suspended account isn’t deleted. status = "suspended" says what actually happened, and queries can be explicit about which states they want.
• Audit history for the “undo” requirement. A package like django-simple-history records every version of a row, including the last one before a delete, so you can do a real delete and still reconstruct the record.
• An archive table. Move deleted rows into a separate table with the same shape. The main table stays clean, constraints keep working unchanged, and restoring is an explicit copy back.

Whichever you choose, keep a real deletion path. hard_delete() must exist, and something must call it. A scheduled purge that permanently removes rows once their retention period has passed is what turns a soft delete into a recycle bin rather than a slow leak:

# customers/management/commands/purge_deleted_customers.py
from datetime import timedelta

from django.core.management.base import BaseCommand
from django.db import transaction
from django.utils import timezone

from customers.models import Customer

class Command(BaseCommand):
    help = "Permanently delete customers soft-deleted before the retention cutoff."

    def add_arguments(self, parser):
        parser.add_argument("--days", type=int, default=30)
        parser.add_argument("--batch-size", type=int, default=500)
        parser.add_argument("--dry-run", action="store_true")

    def handle(self, *args, days, batch_size, dry_run, **options):
        cutoff = timezone.now() - timedelta(days=days)
        expired = Customer.all_objects.filter(deleted_at__lt=cutoff).order_by("pk")

        if dry_run:
            self.stdout.write(f"{expired.count()} customer(s) would be purged.")
            return

        total = 0
        while True:
            pks = list(expired.values_list("pk", flat=True)[:batch_size])
            if not pks:
                break
            with transaction.atomic():
                Customer.all_objects.filter(pk__in=pks).hard_delete()
            total += len(pks)

        self.stdout.write(self.style.SUCCESS(f"Purged {total} customer(s)."))

The hard delete goes through Django’s normal deletion collector, which finds related rows through the base manager, so soft-deleted children are removed along with the parent according to their on_delete rules. If orders point at customers with on_delete=PROTECT, the purge will fail with ProtectedError for customers who have orders. That is a decision to make on purpose: for those customers, the usual answer is to anonymise the personal fields and keep the row, so the financial records stay intact and the personal data goes. Batches keep each transaction short, and --dry-run gives you a number to check before the first real run.

A short checklist

• The filtered manager is declared first, and all_objects exists for explicit use.
• Bulk QuerySet.delete() is overridden, and signal listeners are accounted for.
• Unique fields use conditional constraints on a backend that supports them.
• Restore handles IntegrityError with a clear message.
• Pages that follow order.customer handle a deleted customer on purpose.
• Cross-relation filters and aggregates exclude deleted rows explicitly.
• Reports and BI tools read from a view that hides deleted rows.
• Each related model has a decided behaviour: cascade, keep, or hard-delete.
• dumpdata backups use --all.
• A scheduled purge permanently deletes rows past retention, with a plan for protected relations.
• Each of the above has a test running on your production database engine.

Conclusion

Soft deletes look like a one-column change, but they really change what “exists” means across your whole data model. The pattern above doesn’t make that cost disappear. It gathers it into a small number of places you can see and test: a default manager that hides deleted rows, an explicit manager for when you need everything, constraints that understand deletion, deliberate decisions about related rows, and a purge job that actually deletes. If you can’t name who needs deleted rows back and for how long, a status field, audit history or archive table is likely the simpler answer. If you can, build it this way, and you’ll get the safety net without deleted customers turning up where they don’t belong.

Lycore builds and maintains production systems on Django, React, Flutter and .NET. Get in touch at lycore.com if deleted data keeps reappearing where it shouldn’t.