Stack database backups and restore

Part of Docker Compose.

Going back to the previous containers puts the previous code back, never the data: a migration the failed attempt ran stays in the database. Before 3.2 the copy an update took held, at best, a tarball of a live database's data directory, which nobody can restore. Now it holds a dump.

What is detected. The services whose image is the official postgres, mysql, mariadb or mongo (any tag, -alpine included, from Docker Hub). The user and database come from the service's environment as Compose resolves it (docker compose config --format json), so a ${DB_USER} filled from .env is what the container really has: POSTGRES_USER and POSTGRES_DB, MYSQL_* or MARIADB_*, MONGO_INITDB_*. Other images (bitnami/postgresql, postgis/postgis) are not detected; declare them:

# noust.yaml
backup:
  databases:
    - service: db
      engine: postgres        # postgres | mysql | mariadb | mongo
      database: app           # optional: read from the service's environment
      user: app               # optional: the same

A list replaces detection. A user or database name has to be plain (letters, digits and _ . @ $ -, starting with a letter or digit); for a database that is not one of the four engines, back it up with a pre_deploy hook of your own.

How it is dumped. Through the engine's own client, inside the service's container, with docker compose exec -T:

The password is read inside the container, from its own environment (POSTGRES_PASSWORD, MYSQL_ROOT_PASSWORD, MONGO_INITDB_ROOT_PASSWORD...) or from the file a *_FILE variable names, and handed to the client the way it takes a secret outside a command line. It is in no argument on the host or in the container. The dump goes straight to a compressed file, mode 0600, named stack-<service>-<engine>-<database>.<dump|sql|archive>.gz, under noust-backup/databases/ in the backup's archive, with a manifest entry that says how to put it back.

When. In the backup every update of a Compose stack takes first, before the pre_deploy hooks run and before anything is recreated. It is on by default for a stack that has databases. noust backup create DOMAIN --include-databases includes them in a backup of your own, and that is not switched off by backup.databases: off, which is about the automatic copy.

If the copy fails, the update stops. Without it there is no way back from a migration. The error has the engine's own output and the command that carries on without it:

To update shop.example.com without a copy of its databases, run: noust app backup-before-update
shop.example.com off. To keep the copy, fix what the output above says and run the update again.

Turning it off, when the database is backed up another way or cannot be dumped from its container:

noust app backup-before-update shop.example.com          # shows it
noust app backup-before-update shop.example.com off      # or on

or backup.databases: off in noust.yaml, or PATCH /api/apps/{domain}/backup-before-update with {"enabled": false} (apps.manage, sudo mode), or the application's Settings > Automation in the console. GET /api/apps/{domain} shows backup_before_update.

Restoring. With the stack's service reachable through its compose file:

noust backup list shop.example.com
noust backup restore BACKUP_ID --databases-only     # put back only the databases; every file stays
noust backup restore BACKUP_ID                      # files and databases, as the backup was taken

The application is stopped while the dumps go back and started again if it was running. A database's container is started alone when it is not running and stopped again afterwards. The restore replaces what the database holds (--clean --if-exists, --drop), and Postgres does it in one transaction, so a dump it cannot read whole leaves the database as it was. Do this after going back past a migration, as Rolling back past a migration says.

Volumes. noust backup create --include-docker-volumes now copies the volumes Docker really has: <project>_<key> unless the volume names itself with name: (or is external), read from docker compose config. Before, it mounted the key, which is a new, empty volume.

Every night, and tried. A recipe installed on Compose keeps its database in the stack, so the daily policy that protects the databases in Noust's store never reached it. The recipe's default backups now make the application's own schedule, which dumps each of those databases through its engine's client, and the restore drill loads every dump into a throwaway container of the same image the service ran (the manifest records the image) and counts what it holds. The container has no network, a name and a label of its own, and is removed by name; a dump that does not load fails the drill with the engine's own words.