Where backups go: destinations, encrypted

Part of Backups and disaster recovery.

Provider guides

A destination is an rclone remote; the --type decides which. Object stores take the bucket (or container) as the first part of the folder: --field path=my-bucket/noust-backups. A destination is tried when it is saved: noust backup destination add and update (and POST and PUT /api/backup-destinations) create the folder, which is the first real write so a key that can only read fails here, and list it, exactly as noust backup destination test NAME does, and keep nothing that does not answer. --skip-test ("verify": false in the API) saves one that is down on purpose. For an object store the bucket must already exist: Noust looks at it first, read only, and says so, instead of creating a bucket in your account out of a mistyped name. So the matrix that matters is your own: a provider is supported on your account when destination add succeeds. Noust's offline tests pin what is passed to rclone for each type below, not how a provider answers; the next subsection says what has been run against a real server.

A provider name outside the list of the S3 form is passed to rclone as written (provider=Alibaba): rclone knows more of them than the form offers. An address inside this machine's own networks (a NAS on the LAN, a local MinIO) is refused until it is named in security.allow_private_hosts.

What has been tried against the real service

The table above says what Noust passes to rclone. This one says what has been run against a real server, when, and with which rclone. A row is added when its run is, with the date it was run:

The rclone is the 1.60.1 package of the harness image (v1.60.1-DEV); a newer one is expected to behave the same and has not been tried. The SFTP and WebDAV rows prove Noust's side of the protocol (what it passes, what it keeps, what it refuses) against rclone's server for it, which is a different implementation from the client's side; they do not prove a particular vendor's quirks.

Not tried against the real service, and so supported as far as rclone documents it and no further: Amazon S3, Cloudflare R2, Backblaze B2, Hetzner Object Storage, Wasabi, DigitalOcean Spaces, Scaleway, IDrive e2, Ceph, Supabase Storage and Vultr, Azure Blob Storage, Google Cloud Storage, an FTP server, SMB, Nextcloud and ownCloud, Google Drive, OneDrive, Dropbox, Box, pCloud. Prove yours with noust backup destination add; if it connects, the same steps ran.

The runs are python tests/integration/backups_run.py --scenario s3_destination and --scenario protocol_destinations (Docker; they create noust-test-w6b-* containers with the label noust.test=1 and remove them).

Amazon S3 with the least access. Give the key only what a backup needs; this policy is the whole of it (replace the bucket):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::my-noust-backups"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::my-noust-backups/*"
    }
  ]
}

noust backup destination add offsite --type s3 \
  --field provider=AWS --field region=eu-west-1 --field access_key_id=AKIA... \
  --field path=my-noust-backups/server-1 --stdin < secret-access-key.txt
noust backup destination test offsite

Cloudflare R2. Create an API token with "Object Read & Write" on the bucket; R2 has no regions, so leave region empty and use the account's endpoint.

Google Cloud Storage. Create a service account, give it Storage Object Admin on the bucket only, download its JSON key and pass it whole: --stdin < key.json. bucket_policy_only=true is right for a bucket with uniform bucket-level access, which is the default for new ones.

Azure Blob Storage. The container must exist; --field path=container/noust-backups. The account key is under Access keys of the storage account.

Backblaze. The native b2 type uses an application key limited to one bucket; the S3 type works with the same key against the S3 endpoint of the bucket's region. Either is fine; native is one request fewer per file.

What a provider does with a bucket's own lifecycle rules (versioning, object lock, expiry) is its business and Noust's retention does not know about it: a lifecycle rule that deletes objects younger than the schedule keeps them makes a backup disappear from under the schedule, and noust backup verify is how you find out.

Destinations are encrypted, and watched

A destination made without a word about encryption wraps the remote in rclone's crypt backend with a key generated here and printed once by noust backup destination show-key: noust backup destination add --encrypt/--no-encrypt (on by default), encrypted in the API (true, false or omitted), and backup.encrypt_new_destinations: false turns the default off for a server that wants plain copies. What the operator said wins; a key given to read backups already there means encryption; otherwise the default. The ens-medium profile refuses an upload to an unencrypted destination whatever the default is.

The monitor announces a backup that was never taken, which fails nothing, and a destination that went away, which fails only the next upload: backup_missing when no successful run of a schedule, a database's policy, an application's data policy or Noust's own backup is newer than monitor.backup_missing_days (2; a weekly or monthly schedule is given its period and a day; 0 turns it off), once, then every day while it stays true; and destination_unreachable when a destination some policy sends to does not answer (asked every monitor.backup_check_hours, as noust backup destination test asks), once per outage with rclone's own words, and again when it answers. Both are switches under notifications.events.