Notifications and alerts

Noust tells you when something happens: a deploy failed, a certificate is about to expire, a backup did not run, a measurement stayed past a limit you set. This page is about where those messages go, who hears which, and what happens when a destination does not answer. The monitor that produces some of them is described in MONITOR.md; every setting is listed in CONFIG.md.

Everything here is in the console under Settings > Channels and Settings > Alert rules, and in the command line as noust notify ... and noust alert .... They are two views of the same notifications.* settings, so a change in one is a change in the other.

Channels

A channel is a place a message can go. There are thirteen types:

Every credential is write-only: it is never shown again, in the console, the command line, the API or the logs. Saving a form with a credential left alone keeps the stored one.

At rest the credentials of the channels (a webhook URL, a token, a bot's key, the signing secret, the headers) are sealed in /etc/noust/config.yaml with AES-256-CBC and HMAC-SHA256 under a key that stays beside it (/etc/noust/secrets.key, 0600, root's), so a copy of the file, a backup of /etc or a screenshot of it leaks nothing. They open when Noust reads the file and are sealed again when it writes it; a configuration written before 4.0 is sealed the first time Noust starts (or with noust config protect, which does nothing to a file with nothing in clear). Keep the key with the file in a backup, or the credentials cannot be opened: the error says which file to restore. See security.md and downgrading.md.

Headers of the generic webhook

A receiver that wants its own credential in the request (an Authorization: Bearer token for n8n or Home Assistant, a vendor's API key header) gets it from webhook_headers: one Name: value per line, at most 20, kept sealed like the rest.

printf '%s' 'Authorization: Bearer 3f9a...' | \
  noust config set notifications.channels.webhook.webhook_headers --stdin

A name is a token (letters, digits and a few symbols, - included), a value is one line of printable ASCII, a name cannot be given twice. The headers that frame the request or say what Noust says about a delivery (Content-Type, Content-Length, Host, User-Agent, Transfer-Encoding, Connection, X-Noust-Event, X-Noust-Delivery, X-Noust-Signature) cannot be replaced, so a configured header cannot forge a signature. A line that is not accepted is refused when it is saved, naming the line.

More than one channel of a type

The thirteen channels named after their type always exist. Any other name is another channel, which has a type of its own: a second Slack for staging, a Pushover for whoever is on call.

noust notify channel set slack-staging --type slack --secret webhook_url
noust notify channel set oncall --type pushover --secret user_key --secret app_token
noust notify list
noust notify test oncall
noust notify channel remove oncall

A name is lower-case letters, digits, - and _, starting with a letter. At most 32 channels may exist. Removing one of the thirteen defaults (one of each type) empties it instead.

In the console, Settings > Channels has an "Add a channel" button for these; the new channel opens its own drawer to ask for the destination.

Credentials are asked for with --secret KEY (a hidden prompt) or --secret-stdin KEY, never as an argument, because everything on a command line is visible in ps.

Email through a provider's API

Mail goes through the SMTP account of the monitor unless you choose a provider whose API takes it over HTTPS, which is the way out when your host blocks outgoing SMTP. The provider replaces SMTP, it is not a second way to send the same alert (one switch: monitor.email_api.provider).

noust config set monitor.email_api.provider resend
noust config set monitor.email_api.from_address alerts@example.com
printf '%s' "$RESEND_KEY" | noust config set monitor.email_api.api_key --stdin
noust config set monitor.email_recipients ops@example.com
noust monitor test-email

The sender has to be one the provider has verified. A key the provider refuses, or a sender it does not know, is final (no retries, the provider's own words are kept in the record of deliveries); a provider that is busy or unreachable is tried again like any destination. Credentials are write-only everywhere (*** in noust config show, never in the API), and the address of every provider is fixed: the only typed value that reaches a URL is the SES region, checked to be one. Leaving the provider empty goes back to SMTP. In the console it is Settings > Channels > Email > "Send through".

What each channel hears

By default every channel hears every event that is switched on under notifications.events. Each channel can narrow that in three ways, all in its Routing drawer in the console or as options of noust notify channel set.

Its own events. --event deploy_started=on or --event backup_failed=off gives this channel its own switch for an event, whatever the server-wide one says. An event a channel does not name follows the server's. --clear-events goes back to following it.

A route. A channel can be limited to some applications:

A route with no list reaches every application. Lists work together: an application has to match each list that is given. Tags need applications to carry them; an application without tags never matches a tag route.

Quiet hours. --quiet 22:00-07:00 --quiet-timezone Europe/Madrid holds messages during those hours. --quiet-allow says what still gets through: failed (the default), warning or none. What was held is sent when the hours end: as one message listing them when there are four or more (--no-digest sends them one by one instead), otherwise as they were. The held messages are in the record of deliveries as held until then.

A cap on how much a channel says. --flood 20/10 lets a channel send at most 20 messages in any 10 minutes. A server that reboots, or an outage that touches thirty applications at once, is thirty messages in a second; past the cap the rest are held and, once the window has passed, arrive as one message that lists them (fewer than four are sent as they were). A cap that is still full when they are due keeps them for another window. Held messages are in the record of deliveries as held (flood). --clear-flood removes it.

A summary instead of a message at a time. --summary-at 08:00 --summary-timezone Europe/Madrid holds everything below --summary-below (warning by default, so a success, a recovery or a heartbeat; failed also holds warnings) and sends it once a day at that time as one message. A warning or a failure still goes at once. --summary-days mon,wed,fri limits it to those days. A single item waiting for the hour is sent as it is, not dressed up as a summary. They show as held (summary) until then. --clear-summary removes it.

In the console they are in the channel's Routing drawer, under Cap on messages and Summary.

The quiet hours are judged first, then the summary, then the cap; whatever the summary or a cap releases during quiet hours waits for them to end. A paging channel (PagerDuty) ignores both: it already collapses repeats by the incident's key, and an incident that waited for a summary would be one nobody was paged for.

A channel's route, quiet hours, cap and summary live in config.yaml beside its settings:

notifications:
  channels:
    oncall:
      type: pushover
      user_key: ...
      app_token: ...
      route: {min_severity: failed, projects: [shop]}
      quiet: {start: "22:00", end: "07:00", timezone: Europe/Madrid, allow: failed, digest: true}
      flood: {max: 20, minutes: 10}
      summary: {at: "08:00", timezone: Europe/Madrid, below: warning, days: [mon, tue, wed, thu, fri]}
      events: {deploy_started: false}

A team's own channels

On a shared server a team keeps channels of its own and decides where the notices about its applications go. A team's channel is an ordinary channel (same types, same retries, same record of deliveries) that belongs to one team: it hears about the applications of that team and nothing else, whatever its route says. It does not hear the server itself (a full disk, a node going down) and it never hears another team's applications. The team's route narrows further inside the team (projects, environments, tags, min_severity).

noust notify channel set alerts --team acme --type slack --secret-stdin webhook_url
noust notify channel set alerts --team acme --route-project web --min-severity failed
noust notify channel remove alerts --team acme
noust notify list                     # the Team column

The same through the API, under the team's path (apps.read to look, apps.manage to change, in that team, and sudo mode to change one):

• Two teams may both have an alerts; the key in config.yaml is t<team id>-<name> and carries team: <slug>.
• Email is not offered to a team: it goes out through the server's own mail account, which its administrator gives. A team uses a chat or push channel.
• A team has at most 8 channels, and a destination on a private network is refused unless the administrator lists it under notifications.allow_private_hosts, as for the server's own channels.
• Another team's channels do not exist for a member: the team's path answers 404, the configuration a member reads leaves them out, and testing one by its key says there is no such channel. The administrator sees them all, with their team, in Settings and in GET /api/notifications/channels.
• The team's own page has them too: Projects > Teams > the team > Notifications.

Paging someone

A chat message is read; a page is answered for, and what a page has is state: the incident opens when the thing breaks, a person acknowledges it, it escalates to the next on call when nobody does, and it closes when the thing is fine again. PagerDuty keeps that state and its escalation policy; Noust tells it, with one key per thing, when the thing breaks, when somebody takes it and when it is fine again.

noust notify channel set oncall --type pagerduty --secret routing_key --set region=us \
    --min-severity failed
noust notify test oncall

Which notification is which is decided by its event and its state:

• A problem opens an incident (trigger): a failure pages as critical, a warning as warning.
• The way back closes it (resolve, same key): an application that answers again, a disk back under its limit, a node or a unit that recovered, an alert rule that cleared, the deploy that worked after the one that failed, the backup that ran after the one that did not. Two rules on one application are two incidents; the same problem twice is one.
• Anything else that is a problem opens one of its own and stays open until a person closes it (a certificate about to expire, a security update waiting).
• What is not a problem and closes nothing is not sent: a deploy that started, a heartbeat, a report, the digest of what quiet hours held, an approval.

A recovery always reaches the channel: it is not held by its quiet hours or stopped by its --min-severity (an incident that opened would stay open), and it follows the switch of the problem it closes, so a channel that hears backup_failed also hears the backup that fixed it though backup_success ships off. region is us (the default) or eu, the service region of the account.

When a destination does not answer

A notification is written to a record before it is sent. A delivery that fails is tried again after 1 minute, 5 minutes, 30 minutes, 2 hours and 6 hours, and then given up on; a destination that refuses it for good (a revoked webhook, a bad token) is not tried again, because nothing about a second try would be different. The monitor drives the retries on its scans.

noust notify deliveries                 # newest first; --state failed, --channel oncall
noust notify retry                      # what is due now
noust notify retry 42                   # one delivery, with a fresh set of attempts

The same record is in the console under Settings > Channels, with what the destination said, verbatim, and a "Send now" button on whatever can be sent again. A delivery that was given up on raises notify.delivery_failed in the audit log, never the message itself.

A test (noust notify test NAME, or "Send test" in the console) is not part of the record: it goes straight to the channel and shows what it answered.

Events

notifications.events.<kind> switches each one on or off for every channel; those marked "off" ship switched off.

noust notify events lists them with their state. The events produced by the monitor (the backup, patches, cron and token ones, certificates, disks) are produced only while the monitor runs (noust monitor status).

A failed backup says where all of it is. The excerpt of a backup_failed notice is the end of what the tool said, as many lines as a chat message holds. The whole output is kept (512 KiB at most, scrubbed, the newest 200 failures) and the notice carries noust backup failures show ID as the command to run next and, when web.public_url is set, a second link, output, to GET /api/backups/failures/ID/output: plain text, for a browser signed in to the console. The webhook publishes the command as the fact full_output and the page in links.

The checks have their own settings, all under monitor.:

They are in Settings > Alert rules, under "Built-in checks".

Alert rules

A rule says "tell me when this measurement has been above (or below) that limit for this long". It judges the history the monitor records every minute, so a rule needs the monitor on and the metrics history recording.

noust alert set cpu-high --metric cpu --above 90% --for 5m
noust alert set low-memory --metric app_memory --app shop.example.com --below 256MB
noust alert set errors --metric app_5xx --above 20 --for 10m --severity failed
noust alert list
noust alert status

What a rule can watch (noust alert metrics):

A limit is a number in the metric's unit, or text with it: 90%, 512MB, 2MB/s.

• --for is how long the limit must be breached, without a gap, before the rule fires: at least a minute (the monitor looks once a minute) and at most a day. A spike is not an alert.
• --hysteresis is how far back inside the limit the metric must come before the rule clears. A CPU rule for "above 90" with a hysteresis of 5 clears at 85, so a value hovering at 90 does not fire and clear all night.
• --cooldown is the least time between two messages for the same rule and application (1h by default). A rule that fires again inside it is silent. With --remind it is also how often a rule that keeps firing says so again.
• --severity is warning or failed: the state the message carries, and so which channels' min_severity hear it.
• --app limits an application metric to one application; without it the rule watches each of them, and each is judged and announced on its own.

A rule fires only when most of the minutes in its window were actually measured, so a monitor that was stopped for an hour does not fire a rule because of the silence, and it clears only when every reading of the last minute is inside the limit.

Rules are in Settings > Alert rules, with what is firing now. Changing a rule forgets what the old limit had decided: a rule moved from 90 to 95 is not firing because of 92.

Taking an alert

Whoever is on call takes a firing alert so that the rest know somebody is on it:

noust alert ack cpu-high                              # the server's alert
noust alert ack errors --app shop.example.com --by alice
noust alert unack cpu-high                            # hand it back

In the console it is the button on the alert in Settings > Alert rules, and in the inbox. Taking it stops the rule's reminders (--remind), tells every channel who took it (alert_acknowledged, "alice is looking into it") and, on a PagerDuty channel, acknowledges the incident the firing opened, which stops PagerDuty escalating it to the next person. It does not clear the alert: that still happens, and is still told, when the metric is back inside its limit, and a taken alert that clears and fires again is a new one that nobody has taken. Taking one that is taken changes nothing, and the first name stays. Taking needs the permission to operate applications, not the one to change rules. An acknowledgement made in PagerDuty's own app is not read back by Noust.

Disk and network of an application

The Metrics tab of an application charts its disk reads and writes and its network traffic beside CPU and memory, and the same series feed the alert rules. They come from the service's own accounting (IOAccounting and IPAccounting), which the unit Noust writes now asks for. A unit written before this one learns it on its next update or restart; until then the charts say so. A Compose stack's network is read from its containers.

Reading a site's web server logs

The Logs tab of an application also shows the web server's access log and error log for the site, and noust site logs DOMAIN prints them: the last lines, --kind error for nginx's or Apache's own account of what went wrong, and --contains TEXT to narrow them. A log that has not been written yet says where it will appear. Tokens, passwords and credentials in a line are masked like in every other log Noust shows.

The API

Details and permissions are in api.md. Probes of an address and the status page are in probes.md.