Resource monitor

noust monitor watches the machine and writes down what stands out: resource use, processes worth a look, units that stopped, certificates about to expire and full disks. It keeps what it noticed in a local database and can send you a report by email, or through any notification channel you configure.

It reports and does nothing else:

• it never signals, terminates or restarts a process;
• it never deletes or modifies a file, except the systemd unit it installs;
• it never decides anything from a process's command line (the process samples keep a redacted, truncated copy of the busiest processes' command lines so an operator can read what ran; see Process samples);
• what it sends leaves the machine only through channels you configured: raw observations by email when monitor.notify is on, and the disk, certificate and unit events through whichever notification channels you set up - webhook, Slack, Discord, Telegram, email, Pushover, ntfy, Gotify, Microsoft Teams, Mattermost, Matrix, Lark or PagerDuty (see Notifications). Nothing is sent anywhere else.

Before 1.0 the monitor sent process data to OpenAI and could kill processes and delete their files. All of that is gone. The flags --force-ai and --all on noust monitor scan are still accepted, ignored with a warning; the settings monitor.auto_terminate, monitor.terminate_malicious_only and monitor.dry_run are pinned to their safe values whatever the file says; monitor.use_ai, monitor.ai_interval and monitor.openai.* have no effect and are ignored on load. noust config show lists them apart as obsolete, and noust config clean removes them from config.yaml (comments kept, dated copy first), deleting the OpenAI API key an old file may still hold. See the configuration reference.

Commands

noust monitor scan          # look at the machine once and print what stands out
noust monitor run           # scan on a loop in this terminal, until Ctrl+C
noust monitor install       # write the systemd unit, without starting it
noust monitor enable        # start it now and at every boot, installing it if needed
noust monitor autoenable    # what the package runs: enable it unless it was turned off
noust monitor disable       # stop it and keep it from starting at boot
noust monitor uninstall     # remove the unit; recorded observations stay
noust monitor status        # whether it runs, and what it watches
noust monitor config        # the settings in effect and where the database lives
noust monitor test-email    # send one email to the recipients, to prove the settings

The unit is noust-monitor.service. It runs noust monitor run as root, restarts on failure, and is sandboxed (NoNewPrivileges, ProtectSystem=strict, ProtectHome=read-only, PrivateDevices). The console has the same controls under Server > System, and the Overview says when the history is not being recorded, why, and what to run.

The package installs and enables the monitor on install, and on an upgrade that finds none, because the charts' history is recorded here (see Metrics history). It never overrides a decision: a monitor that is installed stays as it is, and one you disabled or removed stays off until you run noust monitor enable. Where there is no systemd (a container) nothing is installed, and the console records the history while it runs.

"Turned off" is remembered in /var/lib/noust/monitor-declined, which noust monitor disable and noust monitor uninstall write from 3.1 on. A monitor removed with 3.0's noust monitor uninstall left no such file, so on the upgrade to 3.1 the package also leaves the monitor off when the journal still has lines from noust-monitor, or the observation database holds anything a monitor recorded, and writes the file then. Where neither is left (a rotated journal), keep the monitor off with:

sudo touch /var/lib/noust/monitor-declined    # before upgrading; noust monitor enable removes it

What it watches

Every scan interval (60 seconds by default, never less than 10):

Observations are kept in /var/lib/noust/observations.db (~/.local/share/noust/ when run without root). The same process and signal within an hour is recorded once; observations older than monitor.retention_days (30) are purged hourly, and at most monitor.max_observations (5000) are kept. Acknowledging one, from the console or POST /api/monitor/observations/{id}/acknowledge, marks it seen and does nothing to the process.

Every one of these checks, the alert rules and the retries of failed deliveries only happen while the monitor runs.

The web server

Every site of every application goes through one web server, and its unit is the one thing they have in common. When nginx falls, each application would otherwise be "unreachable" on its own and nothing would start the server again until somebody logged in. On every look the monitor reads the unit of the web server in use (nginx, and Apache for the sites that use it):

• Running: nothing. If an outage was open, it is closed.
• Crashed (failed, or stopped after a run that did not end well): the server's own configuration test runs first, because starting a configuration that does not pass takes nothing up and hides the cause. If it passes the unit is restarted and the notification says so (webserver.restarted); if it does not, the notification carries the test's words verbatim and nothing is restarted (webserver.config_invalid). A restart that does not hold is webserver.restart_failed.
• At most monitor.webserver_restart_limit restarts an hour (3). Past it the limit is said once (webserver.limit) and the rest is yours, so a server that falls at every start is not restarted forever.
• Stopped cleanly and still enabled: somebody stopped it. It is said once (webserver.stopped) and left alone. A web server that is disabled and stopped is off on purpose.

monitor.restart_webserver: false keeps the announcement and drops the restart (webserver.down). It is the one check of the monitor that acts, and the only thing it does is what systemctl restart would do; each restart is in the audit trail as monitor.webserver_restart. All of these go through the unit_failed switch of the notifications.

Containers that keep restarting

A Compose stack's unit runs docker compose up -d and finishes, so systemd cannot say that a container of the stack restarts every few seconds, and a stack whose web still answers reads as healthy while one of its workers dies in a loop. The monitor reads the restart counter Docker keeps for each container (RestartCount):

• A container Docker restarted monitor.container_restart_limit times (5) since it was first seen is announced once (container_restart_loop), with the exit code of its last run and its last lines. The count is a baseline, not an absolute: a container that restarted forty times last month is not a loop today. The announcement closes when the container has run five minutes without restarting.
• Docker's events (die, oom, restart, health_status) of those containers are kept beside the metrics, so that the timeline of an application can say that the container was killed for lack of memory at 03:12, long after the unit that started it finished.

Metrics history

The monitor also records the history behind the console's charts: CPU, memory, swap, disk, network and load of the machine, and CPU, memory, disk reads and writes and network traffic of every application, every 5 seconds. Until 3.1 the console recorded it, so it existed only while a console was running.

Every tier keeps the mean and the maximum of each bucket, so a five-minute spike is still there in the month view. The history lives in /var/lib/noust/metrics.db. The whole of it takes about 60 MB for twenty applications with the default retention (measured with 49 series), most of it the hourly tier: 24 rows per series per day, so metrics.retention_days scales it directly.

Only one collector writes at a time: the monitor takes a lease in the database, and the console samples only while no monitor does (a container, where there is no systemd). The console reads the history with GET /api/metrics/query, several series per request, at the tier the window needs. noust monitor status says whether the history is being recorded and, when it is not, why and what to run.

What is measured per application depends on how it runs:

An application behind the proxy has its requests and 5xx per minute counted from the same access log as well, whatever it runs as. See Traffic of an application.

A Docker Compose stack also has series for each of its containers (app.<domain>.ctr.<container>.<what>: CPU, memory, disk reads and writes, network received and sent), beside the stack's total, so a chart can say which service is the one reading the disk. The Metrics tab of the stack charts them by container; a container that lives for one command (docker compose run) is in the total and gets no series of its own.

Memory is the working set (memory.current less reclaimable page cache), CPU is a percentage of one CPU. Disk is the bytes per second the unit's cgroup read and wrote (IOAccounting) and network the bytes per second its processes received and sent (IPAccounting), both of which the unit Noust writes switches on; a unit written before them has no counters until its next update or restart, and a Compose stack's network is read from its containers' namespaces. To tell the workers of one PHP-FPM pool from another's, the monitor reads the title FPM gives each worker (php-fpm: pool <name>): the only process title it reads, and nothing is decided from it.

Traffic of an application

The proxy is where an application's requests, its errors and its visitors are seen, and its access log (<domain>.access.log, which every site Noust writes has) says all of it. The monitor reads what each log gained since the last look (a few hundred kilobytes at most per tick, never twice, rotation recognised) and keeps it by the hour in metrics.db:

• requests, how they ended (2xx, 3xx, 4xx, 5xx) and the bytes of the bodies served, kept as long as the hourly metrics (metrics.retention_days);
• the pages asked for most (GET /orders/:n), the networks they came from (203.0.113.0/24), the status codes and the referrers (a host), kept 90 days. Each hour of each application keeps at most 50 keys of each, so a scan of a thousand paths cannot make the table grow;
• how long requests took, as the median and the 95th percentile (p50, p95), from a histogram kept by the hour: 13 buckets from 5 milliseconds to 30 seconds (and one for what takes longer), so a day or a month is the sum of its hours and the percentile is read from the sum, never the mean of percentiles;
• how many different visitors there were, by the hour and by the stretch asked for. A person who came at nine and again at ten is one visitor of the day, so the hours cannot just be added. What is kept is a sketch of hashes (a short list up to 100 visitors, so a small site's count is exact; a HyperLogLog past that, within about 5 percent) from which no address can be recovered.

noust app traffic shop.example.com                    # the last 24 hours
noust app traffic shop.example.com --window 30d --top 20 --json

GET /api/apps/{domain}/traffic?window=24h|7d|30d|90d&top=10 answers the same: by the hour up to two days, by the day beyond, every step present (a quiet hour is a point of zeros), and the Metrics tab of the application charts it. What is kept is chosen to be safe to keep:

• No query string, ever. It is where tokens, e-mail addresses and session ids live. A page is its method and path, and a segment that is a number, a UUID or a long hexadecimal string becomes :n, :uuid or :hash, so GET /orders/48211 and GET /orders/48212 are one row.
• No address of a person. A client is its network: the first 24 bits of an IPv4 address, the first 48 of an IPv6 one. Whoever needs the exact address has the access log, which the web server rotates.
• Behind a CDN the log shows the CDN's addresses, not the visitors': tell the site where the real address is (noust app policy set DOMAIN --real-ip cloudflare, see network.md) and the networks become the visitors'.

There is no GeoIP: it needs a database whose licence is not Noust's to ship. The networks are the key to put one on yourself.

The log format. The time of a request is not in nginx's default format. The sites Noust writes define their own (log_format noust_<domain>: the combined format and then rt=$request_time; an Apache vhost ends its format with rtus=%D, microseconds) and log with it. A site the operator wrote is never rewritten, so it keeps its own format and shows no times; a log line in the old format, in a file that also holds new ones (the site was written before this release and rewritten since), is counted like any other and simply has no time. A request without a time is not part of the percentiles. An alert rule can watch app_latency_p95 (milliseconds, see notifications.md). The Metrics tab shows the 95th percentile (with the median beside it) and the visitors as tiles above the charts.

Comparing servers

A central keeps no series of its own. noust fleet metrics --metric cpu --window 24h (and the fleet Overview's metrics comparison, GET /api/fleet/metrics) asks every server for the same metric over the same window, from each one's own recorded history, and puts the answers side by side: the points, the latest value, the average and the peak. --metric is one of cpu, memory, swap, disk, load, net_rx and net_tx (memory, swap and disk are the share of what each server has, so a small server and a large one compare); the windows are 1h, 6h, 24h, 7d and 30d. A server that is down is a gap in the comparison, said so in its own words, never an error; one too old to offer the history is unsupported. Nothing is stored on the central.

Exporting the metrics

The history answers the console. A Prometheus, a Grafana Alloy, a Netdata or an OpenTelemetry Collector asks "what is it now?" every few seconds, in a text format they all read, and GET /api/metrics/prometheus answers that: the newest value of every metric of the server, of each application and each container of a stack, and of each database, as gauges.

curl -H "Authorization: Bearer $TOKEN" https://noust.example.com/api/metrics/prometheus
noust monitor export                       # the same text, from this server's own history
noust monitor export --format openmetrics

# prometheus.yml
scrape_configs:
  - job_name: noust
    scheme: https
    metrics_path: /api/metrics/prometheus
    authorization:
      credentials_file: /etc/prometheus/noust.token
    static_configs:
      - targets: ["noust.example.com"]

The credential is an API token that cannot change anything (noust token create prometheus --scope read, or narrowed to the one permission the route asks for, --permission server.read): there is no separate export secret. Send Accept: application/openmetrics-text (a Prometheus configured for it does) and the answer is OpenMetrics. An OpenTelemetry Collector reads the same endpoint with its prometheus receiver, which is how OTLP consumers are served.

Only the series described here are exported, never whatever the store happens to hold, so a new series cannot change the shape or the size of a scrape by accident. A value older than two minutes (an application that stopped being recorded) is left out, and noust_metrics_recording says whether the collector is running at all.

Process samples

A chart says that the CPU peaked five hours ago, not what did it. Wherever the metrics history is recorded (the monitor, or a console that records it itself), the collector also keeps, once a minute, the five processes that used the most CPU and the five that used the most memory:

• Retention. The samples live as long as the per-minute metrics they explain (26 hours; see the table above), in the same database, /var/lib/noust/metrics.db. There is no history from before the collector first ran with 3.2, and a process that started and ended between two minutes is never seen. There is no setting to turn the samples off.
• Command lines are secrets carriers, so each argument is redacted before it is stored: the value of an option or a NAME=value whose name looks secret (--password, --api-key, DB_PASS=), -p<password> of the MySQL clients, any argument that looks like a token or a key on its own, and the password inside every URL. What is stored is shown only to a caller who holds secrets.reveal; anyone else gets the process name. The monitor decides nothing from the command line.
• Where they are read. GET /api/timeline (below), and the console's Investigate this stretch on a chart (console.md).

What happened in a stretch

GET /api/timeline?start=<epoch>&end=<epoch> merges, ordered by time, everything Noust knows about a stretch of up to 31 days: the process samples above, and

The route itself needs server.read, and each source keeps the permission its own page needs. A source the caller may not read comes back as withheld with the permission it lacks, never silently dropped; one that cannot be read says why, with the system's own words, and the rest are still answered. app=<domain> narrows the stretch to one application: its units' journal, the audit events that name it, its deployments and jobs, its units' failures and its processes. sources=journal,audit (repeated or comma-separated) asks for only some. Each source has a ceiling and says when it reached it. On a central the call goes through the node's proxy, and each server answers for what it saw.

Email

noust config set monitor.smtp.host smtp.example.com
noust config set monitor.smtp.port 465
noust config set monitor.smtp.username alerts@example.com
printf '%s' "$SMTP_PASSWORD" | noust config set monitor.smtp.password --stdin
noust config set monitor.email_recipients admin@example.com,oncall@example.com
noust config set monitor.notify true
noust monitor test-email

The password never has to be typed into the command line: --stdin reads it from standard input (a single trailing newline is stripped), and noust config set monitor.smtp.password --prompt asks for it interactively without echoing it back. Typing a secret straight into VALUE on a real terminal prints a warning suggesting one of the two - see Secrets via --stdin. monitor.email_recipients takes a comma-separated list directly, the same as --list, because it already defaults to a list.

Mail can go through a provider's HTTP API instead (Resend, Postmark, Brevo or Amazon SES), for a host that blocks outgoing SMTP: monitor.email_api.provider and its credentials replace the SMTP account, see Email through a provider's API.

With monitor.notify on, a scan that records new observations mails them to the recipients. A process that keeps matching the same signal does not send a message every scan. Credentials are only sent over TLS: use_ssl (implicit TLS, the default) or use_tls (STARTTLS) - not both at once, and never neither when a username or password is set.

monitor.smtp.host, .port, .from_address and every address in monitor.email_recipients are validated when set - a bad hostname, a port outside 1-65535, both use_ssl and use_tls enabled, or an address that is not name@domain are refused with the reason, whether set with noust config set monitor.smtp.* or through the console's Settings form (GET/PUT /api/config/smtp, see API).

Configuration

noust config get and noust config set take these keys; secret values print as ***. Give monitor.watch_units as a list: noust config set monitor.watch_units nginx.service,postgresql.service --list.

Notifications

The events above (unit_failed, app_unreachable, app_recovered, cert_expiring, disk_threshold, backup_missing, cron_failed, cron_success, os_patches, token_expiring, alert_firing, alert_resolved, alert_acknowledged, cron_missed, cleanup_failed, cleanup_success, probe_failed, probe_recovered, security_release, container_restart_loop, recipe_outdated and central_silent) go through Noust's notification channels (webhook, Slack, Discord, Telegram, email, Pushover, ntfy, Gotify, Teams, Mattermost, Matrix, Lark, PagerDuty, and any others you add by name), which are configured under notifications.* in the console's Settings > Channels or with noust notify and noust config set, and only when notifications.enabled is on and the event is enabled. Each channel can hear its own events, only some applications, and not at night; a delivery that fails is tried again and kept on record. All of that is in notifications.md. Test a channel with noust notify test <channel>.

app_unreachable and app_recovered are on by default, like unit_failed (notifications.events.*, see CONFIG.md). The first names the application, says since when it has not answered, quotes the last probe verbatim (the request and what came back, or one line per container that is not fine for a stack with no published port) and points at noust diagnose DOMAIN; the second says how long it was down. An application stopped on purpose, being deployed or running a job, or whose unit failed (that is unit_failed) is not announced.

API

Troubleshooting

• Nothing is recorded. psutil must be installed (apt install python3-psutil, dnf install python3-psutil, zypper install python3-psutil, or pip install psutil; the RPM package only recommends it), and the monitor needs root to see every process.
• An application has no chart. The chart says why: the unit is stopped or missing, it is a static site (see its requests instead), or accounting is off for its unit. Units written from 3.1 on carry MemoryAccounting=yes and CPUAccounting=yes; for an older one, redeploy it or add them (systemctl edit <unit>) and restart it.
• No email. Run noust monitor test-email: it prints the SMTP server's own error. Check that the firewall allows outbound 465 or 587.
• Too many resource-use observations. Raise monitor.cpu_threshold or monitor.memory_threshold. The same process with the same signal is recorded at most once an hour.
• A chart has no process samples for the stretch. They are kept 26 hours and only exist from the minute the collector first ran with 3.2; an older peak has the chart and nothing that explains it. The console says so instead of showing an empty list.
• An application is announced as unreachable and answers when you try. The monitor asks what the deploy's health gate asks: its health path and expected statuses (noust app health DOMAIN), on the port that serves now. Fix the check, not the monitor. A Compose stack with no published port is judged by its containers (docker compose ps -a).