Back up applications and databases

Noust backs up three different things, each with its own command, schedule and retention. Together they are enough to bring a server back from nothing.

Every destination (rclone) is the same one for the four of them: configure it once with noust backup destination.

The rest of this guide is on pages of its own:

• Where backups go: destinations, encrypted
• Bring a whole server back, or move to another

Databases

PostgreSQL, MySQL, MariaDB and MongoDB are covered; a SQLite file inside an application is protected by the application's own backup, which takes it with SQLite's online backup so it is consistent while the application writes. noust db sqlite list shows which files an application keeps, check asks SQLite whether one is sound, and copy takes a consistent copy of one before a risky change (see databases).

noust db backup-schedule set app_db -e postgresql --schedule daily --keep 7
noust db backup-schedule set --all -e postgresql --schedule daily       # every database on the engine
noust db backup-schedule set --all -e postgresql --exclude-database scratch
noust db backup-schedule set app_db -e postgresql --exclude-schema audit --exclude-table logs
noust db backup-schedule set app_db -e mongodb --exclude-table sessions   # a MongoDB collection
noust db backup-run app_db -e postgresql                                 # now, the way the timer does it

An all-databases policy skips the databases that have a policy of their own and the ones it excludes. When one database fails the others are still dumped, and the run reports the failure at the end. ClickHouse is not covered: Noust has no ClickHouse engine manager yet.

Importing a dump

noust db import app_db /srv/dumps/app.sql.gz -e postgresql --inspect   # what is it, can the engine load it
noust db import app_db /srv/dumps/app.sql.gz -e postgresql             # load it
noust db import app_db /srv/dumps/app.sql.gz -e postgresql --replace   # drop the database first

A PostgreSQL dump from outside is loaded as the database's owner, not as the server's superuser, so a dump that runs COPY ... FROM PROGRAM is refused by the server itself; --as-superuser is the explicit way for a dump you trust, and noust db import-all brings a whole pg_dumpall (details).

The format is read from the file's bytes (SQL, custom and tar PostgreSQL dumps, MongoDB archives, gzip, bzip2, xz, zip), never from its name. A server path must be under <backup directory>/imports or a directory listed in databases.import_directories. The database is dumped first and put back if the load fails; a database the import created is dropped again. PostgreSQL owners and privileges in the dump are dropped unless --keep-owners says otherwise, because they are the roles of the server it came from.

The console uploads a file from the browser in chunks (/api/databases/imports/uploads), resumable at the offset the server reports, up to databases.import_max_bytes (10 GiB by default). Upload files are private to Noust (mode 0600) and swept after 24 hours.

The roles of a PostgreSQL server

A dump names its owners and the roles it grants to and carries none of them, so one restored on a new server lost its owners or failed on the first ALTER ... OWNER TO. An application backup now also holds pg_dumpall --globals-only of each PostgreSQL server its dumps came from (once per server), and a restore creates the roles the server lacks before it loads a dump. Only what is missing is created: nothing that exists is altered, the superuser's password included, and the file is read statement by statement, so a psql meta-command or anything that is not a role statement is never run. What cannot be put back (a tablespace's directory, a setting for a database that is not there) is listed. Noust's own state carries the same file, so noust restore-server has the roles before the databases.

noust db globals dump [-o FILE]            # take them now (root-only file: it holds password hashes)
noust db globals list                      # the files this server has dumped
noust db globals restore FILE              # create the roles a file has and this server lacks

Dumps that are really dumps

A dump that exits cleanly and wrote nothing (a client killed mid-pipeline, a connection to the wrong place) is refused and its file removed, not left in the backup directory with a valid name: an empty file, or a gzip of nothing, fails where every engine's dump passes. A dump that fails or runs past its deadline leaves no file, and dump | gzip does not hide a failing producer.

Keeping by size

A policy keeps by number and age, and by size too: noust db backup-schedule set app_db -e postgresql --keep 14 --keep-days 30 --keep-gb 20, and --to nas:14:30:50 for a destination (name, count, days, gigabytes). The newest dumps that fit are kept, counting from the newest back; one that does not fit ends the list (a window of time, not a pile of whatever is small); and the newest always stays, whatever its size, so a limit never leaves a database with no dump. Dumps taken by hand and the safety copy of a restore are never retention's. Application, data and Noust-state backups keep by number and age only.

When and how a backup runs

Every schedule (a database's, an application's, an application's data, Noust's own state) takes the same options.

noust db backup-schedule set app_db -e postgresql --schedule daily --max-runtime 2h --timezone Europe/Madrid
noust backup schedule update shop.example.com --max-runtime 90m
noust backup priority show
noust backup priority set --nice 15 --io-class idle --cpu-quota 50
noust backup priority apply                # bring the units an earlier version wrote along
noust backup sweep                         # apply every policy's retention now

• A time limit (--max-runtime, max_runtime_seconds in the API; one minute to seven days) becomes TimeoutStartSec= in the service the schedule runs in: systemd stops what runs past it, the unit fails and the monitor announces it. It is not RuntimeMaxSec=, which systemd ignores in a Type=oneshot service and says so.
• A time zone (--timezone, timezone in the API) becomes the suffix of OnCalendar= (*-*-* 02:00:00 Europe/Madrid). It is checked against this server's own zone database. An update that names neither keeps what the timer had, as it does retention.
• A priority. A backup runs behind the applications: Nice=10 and best-effort disk priority 7 by default (backup.nice, backup.io_class best-effort or idle, backup.io_priority, backup.cpu_quota as a percentage of one CPU), written between two marker comments in the one place units are written, so a second pass changes nothing and a unit that is not Noust's is never touched.
• Retention that does not wait. A policy used to prune only after a run that went well, so a run that failed (a full disk, usually) pruned nothing. It now prunes after a run that could not take its copy, and a daily noust-backup-sweep timer applies every database, data and application policy without a run, reading the policies and the files and never the engines. The newest copy is kept, and when it failed its check the newest good one is kept too. A manual backup and a restore's safety copy are never a schedule's to delete. Destinations are pruned when a run sends to them.
• A failed backup says all it was told. The notice carries the end of the failure, bounded to what a chat message holds. The whole of what the tool said is kept (up to 512 KiB, scrubbed of the credentials Noust knows, the newest 200 failures) and the notice names it: noust backup failures show 42 (--raw prints nothing but the output) and GET /api/backups/failures/42/output, which the notice links to when the console has a public address.

Application data

The persistent paths of an application on releases, the named volumes of a Compose stack and a recipe's data directories are backed up as one archive each:

noust backup volumes targets shop.example.com
noust backup volumes schedule set shop.example.com --schedule daily --keep 7 --to nas:14 --consistency pause
noust backup volumes run shop.example.com        # now, the way the timer does it
noust backup volumes verify shop.example.com <id> --deep
noust backup volumes restore shop.example.com <id>

--consistency is live (copy while it runs; SQLite files inside are snapshotted), pause (freeze the application for the copy) or stop (stop it, copy, start it again). An application stopped for a backup is always started again, also when the copy fails. A SQLite database an application is writing is taken with SQLite's own backup API and put in the archive under the file's name, with its -wal, -shm and -journal files left out, so a copy never holds a transaction half-way; a file that only looks like a database is archived as it was. An application stopped for a copy is started again also when the process is told to stop (SIGTERM, what systemd sends to a timer's unit), and if it still cannot be, the error says which unit and the command to start it. A Docker volume archived while its application runs (--consistency live) that turns out to hold an engine's data directory (PostgreSQL's PG_VERSION, InnoDB's ibdata1, MongoDB's WiredTiger.wt, Redis's appendonlydir) is flagged in the log and the audit record as a risk, with the two ways out: --consistency stop, or the database's own dumps.

A restore takes a safety copy of what is there, replaces the data (it does not merge), starts the application through the health gate and puts the old data back when the gate fails.

Noust's own state

noust self-backup create                         # asks for a passphrase
noust self-backup list
noust self-backup verify noust-self-<host>-<time>.tar.gz.enc
noust self-backup schedule set --schedule daily --keep 7 --to nas:30
noust self-backup restore noust-self-<host>-<time>.tar.gz.enc

A bundle is one file, sealed with a passphrase (scrypt, then AES-256-CBC and HMAC-SHA256 through openssl, the same envelope as a central's sealed secrets). It holds the store (taken with SQLite's online backup, so it is consistent), the configuration, the secrets and keys (fleet, SSH, sealing), the certificates, and the unit files and drop-ins Noust generated. It also holds the web server's configuration (/etc/nginx and Apache's directory, each file with its mode and its links, kept relative so Debian's sites-enabled links survive a different layout), so a site edited by hand or the site of an adopted application comes back after a disaster, and the roles of each PostgreSQL server. It does not hold the metrics and observation databases (they regenerate) or application backups.

The passphrase is never an argument. Give it at the prompt, in a file only its owner can read (--passphrase-file), in NOUST_SELF_BACKUP_PASSPHRASE or as one line of standard input. Keep it outside this server: without it the bundle cannot be read, and it is the only thing the bundle does not carry. A scheduled backup needs the passphrase unattended, so schedule set stores it in Noust's secret store on this server, which a bundle sent elsewhere does not share a disk with.

verify proves a bundle restorable without restoring it: the envelope, every file's digest, the store's integrity check and that this Noust can migrate its schema. restore first checks every path in the bundle against this server's own directories (a bundle never decides where a file goes), refuses to run over a server that already holds state without --force (and then takes a bundle of what was there first), stops Noust's services, puts the files back, runs the store's migrations and starts everything again.

Backups sealed on this server's disk

A backup on the disk that holds it is readable by whoever gets the disk, a snapshot of it, a copy of /var/backups or a retired drive. Destinations can already be encrypted (Where backups go); this seals the copies that stay on the server:

noust backup encryption enable            # asks; shows the key once. Keep it somewhere else.
noust backup encryption status            # sealed or in clear, which key, how many
noust backup encryption show-key [ID]     # show a key again
noust backup encryption rotate            # a new key for what is made from now on
noust backup encryption import-key ID --passphrase-file FILE   # put back a key that was lost
noust backup encryption disable           # stop sealing; every key is kept
noust backup encryption seal-existing     # seal what was made before (below)

The same operations are /api/backups/encryption: GET (status), POST /enable, /rotate and /disable (sudo mode), GET /keys/{id} (sudo mode and secrets.reveal; the value is shown, the read is recorded without it), POST /keys (import; never echoed) and /seal-existing (a job).

From then on every backup this server makes is written as an encrypted, authenticated file (scrypt, AES-256-CBC through openssl, HMAC-SHA256 over the whole file: the construction of Noust's own state) under the same name: an application's archive, an application's data, a database dump (whichever engine, a safety copy included) and the base backups of point-in-time recovery. Sidecars stay readable: they hold ids, sizes and digests, never the data.

• One door. Every reader of a backup file (verification, restore, the restore test, the drill, the put-back after a failed restore, extracting a base backup) opens it through noust.core.backup_files.open_backup, which authenticates the whole file before openssl reads a byte and gives the reader a decrypted copy in a private directory (<backups>/.open, 0700) that is removed when it is done, however it ends. A file that is not sealed is read where it is, so backups made before, dumps imported from elsewhere and copies downloaded in clear take the same path. An engine added later is sealed and opened without saying so (the guard is in the base class).
• What is recorded is the stored file. The size and the digest in a sidecar, its MAC, the dump record that is sent and checked against the destination are those of the sealed file, so everything that compares them keeps working. A push sends the sealed file as it is; a restore from a destination opens it the same way.
• Keys. Each key is a random 48-character passphrase in the secret store (backups/local-encryption/<id>), and its id is written in clear in the label of the files it sealed, so a rotation never strands a backup. A sealed backup is gone with its key: keep the key printed by enable where this server is not (a password manager, a safe). A backup whose key this server does not hold is refused with the command that puts it back, before anything is touched; so is one that was altered.
• What it protects, and what it does not. The key lives in the same server's secret store (root-only; sealed too when the store is sealed under a passphrase), so this defends the backups against being read without the secret store: an exposed directory or share, a snapshot or an rsync of /var/backups, a drive taken out when the store is not on it. It does not defend them against root on a running server, and it does not replace encrypting the disk. Backups made before it was turned on stay in clear until their retention removes them, or until you run seal-existing (status counts them).
• Sealing what is already there. noust backup encryption seal-existing seals the application archives, the data backups, the database dumps and the base backups of point-in-time recovery that were made in clear, in place and under the same names, and writes each record again with the stored file's size and digest (the sidecar and its MAC, the dump's record, the base's sidecar), so every check, restore and send goes on working. A backup that does not match its record is left in clear and said, never sealed into something that looks sound. A file that was sealed but whose record was not written, because the run stopped, is repaired the next time. An application that is being operated on is skipped and sealed by the next run. It needs room for one more copy of the largest backup for a moment, and it counts what is still in clear afterwards.
• What it costs. Sealing and opening copy the file (sealing needs about two extra copies of it for a moment; opening, one). The database dumps staged inside an application's archive are not sealed on their own: the archive that carries them is, whole.
• Not covered. The write-ahead log archive of point-in-time recovery (the engine's own directory, written by PostgreSQL itself one segment at a time), Noust's own state (already an envelope under its own passphrase), a dump handed to noust db import, files no record describes (a restore's safety copy, a dump put in the directory by hand) and what a destination already holds.

Does a backup come back?

A backup nobody restored is a hope. noust backup drill rehearses it:

noust backup drill shop.example.com            # the newest backup; --backup ID for another
noust backup drills shop.example.com           # the reports kept, newest first

The backup is verified in depth, unpacked with the extractor a restore uses into a scratch directory, and each database dump it carries is loaded into a temporary database of its engine, looked at and dropped. An application that keeps its data in a SQLite file gets a step per file too: SQLite's own quick_check runs on the unpacked copy, so a file that came back damaged fails the drill with SQLite's words. Nothing of the application, its databases or its units is touched, and it is not started: the report is dated and says what it proved (the files unpack, the data loads) and what it did not (that a new server would serve them). Starting the application from its backup and passing its health gate is not part of it, on purpose: the copy would run with the backup's .env, which points at the real databases and services, and a copy that cannot reach them fails for the wrong reason, so it would need a restored database of its own and an environment rewritten per application type. The dump of a database that lives in a Compose stack is loaded into a throwaway container of the image the service ran (the dump's manifest records it): no network, no published port, a memory and process limit, a name noust-drill-<random> and the label noust.drill=1, a password made for the occasion that reaches Docker through its environment and never an argument, and removed by its exact name, with its volume, whatever happened; what it holds is counted. A Redis or Valkey snapshot, which replaces every key and so cannot be tried beside the real data, is loaded into a second server that listens on no port but a private socket and is stopped afterwards. It exits 1 when something fails. A scheduled backup runs one by itself when the last good one is older than backup.drill_days (30 by default, 0 turns it off), after the backup and its destinations, in the same run; a drill that fails is announced as a failed backup and does not fail the backup's own run. POST /api/apps/{domain}/restore-drills (a job) and GET the same path are the API, and the backup history drawer shows the reports. For a whole server, noust restore-server --plan rehearses it, and the next section moves one.

Configuration

Permissions: the database import routes need databases.manage; volume and self backups use the backups.* permissions, and changing the schedule of Noust's own backup is root-equivalent (sudo mode). Everything is in the audit trail: db.import, db.import.upload, volume_backup.*, self_backup.* and restore_server.run.