Running an application

What an application is while it runs, and what an operator can do to it without opening a shell: the work it does beside the web process, how many copies serve it, what its unit is allowed to do, the files it is given, the files it has, where its logs go, and what has been changed but not applied yet. Deploying and going back are in releases.md.

Every command here takes --reason (recorded in the audit log, required under the ENS profile) and, where it changes something, --dry-run. Every one has a route in the API (api.md) and a place in the console, which is a client of that API. Over the API and in the console every change needs sudo mode, and so does reading what may be a secret: the text of a file, of a mount, or a download.

• Processes
• Copies of the web process
• Service security
• Tuning the unit
• Disk: what an application takes, and a limit
• Could it run on several servers?
• Mounted files and paths
• Files
• Log shipping
• Changes not applied yet
• Scheduled tasks: what ran and what did not
• Reading a deployment's log
• Deleting an application

Processes

A queue worker, a scheduler, a stream consumer: a process that belongs to an application and must run the code the application is serving, not a copy that was current once.

noust app process set shop.example.com worker "node dist/worker.js" --instances 2 --memory 512
noust app process list shop.example.com        # every process, with the state of each copy
noust app process logs shop.example.com worker # the journal of all its copies, interleaved
noust app process scale shop.example.com worker 4
noust app process remove shop.example.com worker

The command is split like a POSIX shell would split it and run without one. It runs as the application's account, in the release that serves, with the application's environment and the sandbox of its unit, and each copy has its own limits (--memory, --cpu, --tasks).

• A process follows the release. It runs from <app>/workers, a link that moves with current, so it is never left on code that was current once.
• Atomic with the web process. Every activation restarts the processes together with the web unit and judges them with it. A copy that does not stay up on the new release fails the health gate, and the release is put back for the web process and the processes alike.
• Its unit is Noust's (<app>-<process>@.service, one instance per copy). It is not a row of noust service: a generic unit that nothing ties to a release is the other tool.
• --instances 0 defines a process and keeps it stopped.
• A repository can declare them. The worker: of a Procfile or of processes: in noust.yaml (build.md) becomes a process whose source is the repository (noust app process list says so). Every deployment makes the repository's processes what it declares, with the release that serves, and the number of copies stays what you set. A process you define under the same name is yours and the repository's is not used.
• Going back brings the processes back. A rollback of the history (console and API), noust releases rollback, noust update --commit onto a release that is still on disk and noust rollback to a backup of an application on releases declare the processes of the release they restore, the way a deployment does, before the health gate judges it: a worker the older release did not have is stopped once it holds, one it had comes back with its command, and a copy that does not stay up on it puts the release that served back with its own processes.

Over the API: GET /api/apps/{domain}/processes, PUT and DELETE /api/apps/{domain}/processes/{name}, POST .../scale and GET .../logs. In the console it is the Processes subsection of the application's settings.

Copies of the web process

The application's own unit and some more, each on a port of its own, behind the same nginx site:

noust app replicas shop.example.com            # how it is scaled now
noust app replicas shop.example.com 3          # the unit and two more
noust app replicas shop.example.com 1          # alone again

nginx shares the traffic among the copies (least_conn, and a copy that fails twice is left out for a few seconds). A new copy has to answer the application's health check on its own port before nginx is told about it; one that does not leaves everything as it was. A release reaches the copies the way it reaches the unit: they are restarted one after another behind the same gate.

Scaling out is one release in several copies; zero-downtime mode is two releases side by side. They exclude each other. What cannot run twice (a static site, a Compose stack, a monorepo, a site with routes of its own, Apache) is refused with the reason. The ports are Noust's to pick and keep, so scaling down and up brings the same ones back.

Service security

An application's unit already had NoNewPrivileges, PrivateTmp and, since 3.2, an account of its own. From 4.0 it can also have the rest of what systemd takes away for free:

noust app hardening status shop.example.com   # the profile, what it allows, and how exposed the unit is
noust app hardening set shop.example.com default --writable /var/lib/shop
noust app hardening set shop.example.com off

• New applications are hardened from their first deploy with the profile deploy.hardening names (default unless changed; see CONFIG.md). Applications that exist keep their unit until you move them: a redeploy never changes the sandbox of a running one.
• Changing it restarts the application behind the health gate. When it does not answer, the previous unit is put back and the application is restarted as it was. --no-restart writes the unit and leaves the running process in the sandbox it has.
• Who is hardened: an application whose unit runs its own code (Node, Python and the like) and a monorepo: every workspace's unit gets the profile (a Node one, so strict leaves writable executable memory alone), they are rewritten together and restarted together behind the gate that probes each workspace, and a workspace that does not answer puts all the previous units back. A monorepo created from 4.0 is hardened from its first deploy; one from before keeps its units until noust app hardening set. Not a Compose stack or a container (their processes are Docker's, and Docker confines them) or a PHP site (its pool lives in the PHP-FPM every PHP site shares).
• status shows how exposed the unit is as systemd-analyze security scores it (a read-only probe), and set shows the score before and after.

Moving an application to an account of its own is the other half of this: noust app identity migrate --all moves every application that can have one, one after the other, and reports the ones that fail without stopping. While an application still runs as the shared account, the server health card says so under Application accounts. See releases.md.

Over the API: GET /api/apps/{domain}/hardening (apps.read) and PUT (apps.manage). In the console it is Service security in the application's settings.

Tuning the unit

noust app limits sets the hard limits (memory, CPU quota, tasks). Docker offers more knobs than that, and systemd has a native one for each, acting on the application's own cgroup instead of a container around it. noust app tuning sets them:

noust app tuning status shop.example.com
noust app tuning set shop.example.com --memory-high 384 --swap-max 0 --cpu-weight 50
noust app tuning set shop.example.com --cap NET_BIND_SERVICE --nofile 100000
noust app tuning set shop.example.com --gpu nvidia          # the device groups an NVIDIA GPU needs
noust app tuning set shop.example.com --unset cpu_weight    # take one option away, keep the rest
noust app tuning clear shop.example.com

• What is not named is kept. set lays what you typed over what the unit has; --cap and --device replace their list. An empty tuning is the unit the template writes.
• After the sandbox. The lines go at the end of [Service], after the sandbox's, so a capability or a device you ask for wins over its empty capability set and its private /dev. Everything you do not name stays taken away.
• Capabilities are a short list. NET_BIND_SERVICE, NET_RAW, NET_BROADCAST, IPC_LOCK, SYS_NICE, SYS_RESOURCE and SYS_CHROOT. A root-equivalent capability (SYS_ADMIN, SYS_PTRACE, DAC_OVERRIDE, SETUID...) is refused: an application that needs one is a custom unit, not a process Noust runs as an unprivileged account.
• Devices are a node under /dev (/dev/dri/renderD128:r) or a device group of /proc/devices (char-nvidia, char-drm), opened r, w or rw. The GPU presets are nvidia (char-nvidia, char-nvidia-uvm, char-nvidia-caps) and dri (char-drm, Intel and AMD); the directives are checked with systemd-analyze verify, and whether a given GPU works inside the sandbox is up to the driver.
• Not here. shm_size, DNS and the hostname of Docker's options have no counterpart in a native process: it shares the host's. A Compose stack or a container application keeps them in its compose file.
• Like the sandbox, a change rewrites every unit the application runs as (the two instances of a blue/green application are written once, into their template; the copies of the web process take it when the health gate restarts them) and restarts it behind the health gate. When it does not answer, the previous unit and the previous options are put back and it is restarted as it was. --no-restart writes the unit and leaves the running process as it is; it then shows under changes not applied yet.
• Who is tuned: the same applications that can be hardened: those whose unit runs their own code.

Over the API: GET /api/apps/{domain}/tuning (apps.read) and PUT (apps.manage, sudo mode; a job). PUT carries the whole tuning, so a client changes one option by sending the rest as it read them. In the console the limits and ulimits are under More limits in the application's Resources, and the capabilities and devices in Service security.

Disk: what an application takes, and a limit

The server's storage page (Server > Storage) says what takes space on the machine; it has no row per application, so an uploads folder or a release history that keeps growing is invisible until the whole disk is full. This says it per application:

noust app disk shop.example.com --measure       # walk its trees now, and keep the answer
noust app disk shop.example.com                 # the last measurement, and how it stands
noust app disk shop.example.com --limit 2048    # hold new releases to 2048 MB
noust app disk shop.example.com --no-limit

Its backups are listed apart: they are kept elsewhere and do not count against the limit. Docker named volumes are not included, and a Compose stack's bind-mounted data is, because it is inside the folder.

• Measuring is a job. du over a release tree with a node_modules in it takes a minute, so nothing measures in a request: the console and POST /api/apps/{domain}/disk/measure queue it, the answer is kept with its time, and GET /api/apps/{domain}/disk (apps.read) only reads it. Each tree has its own deadline, so one slow tree is named in errors and does not cost the rest. It uses the same du, run at the lowest I/O and CPU priority, as the storage analysis.

• The limit is Noust's, not the kernel's. A deploy that would build a new release (create, update, a webhook, update --commit) is refused in its pre-flight while the application is over it, with the figures and the command that raises it. Going back to an earlier release, which builds nothing, is never held up by it: an outage is not a reason to wait for room. The limit does not stop the application from writing, and it is not checked while the application runs. An application with a limit is measured before each deploy, which costs a walk of its trees; a tree that could not be measured in time is not taken as evidence of being over. The least a limit can be is 64 MB. PUT /api/apps/{domain}/disk (apps.manage) sets it or, with {"soft_mb": null}, removes it.

• The kernel's quota: the filesystem is the operator's, the project is Noust's. A filesystem project quota stops the writes themselves, and it needs the filesystem mounted with prjquota, which is a change to a running machine that no process of Noust's makes. The page says whether the filesystem the application lives on (findmnt -T) would accept one: enabled, available (XFS or ext4, not mounted with it), unsupported or unknown. Once it is enabled, giving the application's limit to the kernel is one command:

  ShellCopy

  noust app disk shop.example.com --limit 2048
  noust app disk shop.example.com --kernel      # set the project on its tree and the limit on the filesystem
  noust app disk shop.example.com --no-kernel   # take both off again; Noust's own limit stays

  Over the API it is PUT /api/apps/{domain}/disk/kernel with {"enabled": true} (apps.manage, sudo mode; a job, because the project id is written on every file already in the tree), and the console offers it under Settings > Disk once the filesystem has project quotas. What it does, and does not:

  • It sets a project id of Noust's own range (3000000000 plus the application's id, kept in the store) on the application's directory, with xfs_quota on XFS and chattr and setquota on ext4, and the limit on the filesystem. It writes nothing to /etc/projects or /etc/projid, so it cannot meet a name someone wrote there nor the ids Docker uses. A limit changed afterwards with --limit follows into the kernel, and deleting the application takes the quota off.
  • It refuses, saying what to do, when the application has no limit, when the filesystem is not mounted with project quotas, for a site Noust adopted and for a directory outside the apps directory: those files are yours. It never mounts, remounts or tunes a filesystem.
  • It holds an account without CAP_SYS_RESOURCE at the limit on both filesystems, which is every account an application runs as and the build sandbox. XFS holds root too; ext4 does not, so Noust's own work as root (a restore, a backup extracted in place) can pass it there. Files moved into the tree from another directory keep the project they had, which is the kernel's rule, so move them in by copying.
  • The kernel's number is the one that stops the writes; what the page shows next to it is what the kernel counts (xfs_quota report, repquota), not Noust's measurement. They differ by what each counts (block sizes, hard links).

  To mount the filesystem, with the tools the kernel's documentation names, on a machine you can restart:

  ShellCopy

  # XFS: add prjquota to the mount options in /etc/fstab (a remount does not turn it on; the root
  # filesystem needs rootflags=prjquota on the kernel line), then reboot.
  # ext4: enable the feature on the unmounted filesystem, then mount with prjquota.
  tune2fs -O project -Q prjquota /dev/sdX
  mount -o prjquota /dev/sdX /var

  Both were checked on real XFS and ext4 loop filesystems mounted with prjquota (tests/integration/run.py, scenario kernel_project_quota_stops_the_writes): the application's account is refused at the limit ("No space left on device" on XFS, "Disk quota exceeded" on ext4), a changed limit follows, and --no-kernel lets it write again. Try it on a machine you can lose first.

• Not done yet. There is no alert for an application approaching its limit, and no per-application breakdown of Docker volumes.

Could it run on several servers?

Putting an application on several servers is the central's, not the server's: it keeps a record of where the application runs, rolls one commit out to each and copies it to a server that lacks it (the central's guide). None of that looks inside the application, and it cannot, because the central holds no copy of what a server runs. What decides whether a second server would serve the same thing is on the server the application is on, and this says it:

noust app replication shop.example.com

It reads this server's records of the application, its .env (the names of the variables, never their values), a Compose stack's compose file and the list of the server's scheduled tasks, and changes nothing and reaches no other server. GET /api/apps/{domain}/replication (apps.read) answers the same, and the console shows it under Settings > Export.

A finding that blocks is one a second server would get wrong; one that warns would work, with a cost, and the operator decides. An application that runs a registry image (--type container) is told it can be rolled out by tag, so no server builds it.

What is not here, on purpose: Noust is not a cluster. There is no autoscaling and no scale to zero, no scheduler that decides where an application runs, and nothing moves a running application by itself. The copies of one application on one server are the web process's; across servers they are the central's placement, behind a load balancer of yours.

Mounted files and paths

A configuration file that is not in git (a .htpasswd, a certificate, a config/production.json with a token) has to be where every release finds it, and be readable and editable by an operator.

noust app mount file shop.example.com config/production.json --from ./production.json --mode 640
noust app mount list shop.example.com
noust app mount show shop.example.com config/production.json
noust app mount remove shop.example.com config/production.json

• A file lives in shared/<target> and is linked into every release, exactly like a persistent path, so a new release has it before it starts. In place it is written at <app>/<target>; for a container application it is a file under <app>/mounts/, bound at the target in the container. It is written with the mode you name (0600 by default), handed to the application's account, and never through a link: a path whose parent is one is refused. Its text is the file on disk, not a column, so there is one copy to edit and back up. At most 5 MB. A running application reads it at its next restart.
• A path of the server bound into a container application (noust app mount bind /srv/data /data), read-only unless --writable. It is root-equivalent, so the server's own state and the Docker socket are refused, and under the ENS profile only read-only is accepted.
• An .env file is not a mount: the environment has its own validation, masking and audit. A static site and a Compose stack take no mounts (the stack's own compose file mounts what it needs).

Over the API: GET, PUT and DELETE /api/apps/{domain}/mounts, and GET .../mounts/content, which needs the permission that reveals secrets (secrets.reveal) because a mounted file may be one. In the console it is Mounts in the application's settings.

Mounts in an export, a clone and a copy to another server

The mounts travel with the application. noust app export writes them in a mounts section, noust app import and noust app clone make them again, and noust fleet placement copy (and the console's copy to another server) carries the same document, so a copy has the files its original has. What each part of the document says:

• A file is its target, its mode and its text. The text may hold a token, so it is treated like a secret variable: it is in the document only with --with-secrets (in sudo mode over the API), and null with "secret": true otherwise. Reading it for the document is audited like any read of a mounted file. A file that is not on the disk, is not text, or would take the document past its size (about 600 KB of text in all) is named with a note that says why and the command to copy it by hand, never cut.
• A path of the server is its source, its target and whether it is read-only. The other server must have the path: one it does not have, or one that is the server's own state, is skipped and said so. A document that binds paths asks for a yes before it is imported, because a container that writes there changes the server.
• When it exists. A file is written before the application's first start, in shared/ (or in the tree in place), so an application that cannot start without its configuration still passes its first health check; the import then records it as a mount. A container is bound to its mounts when its compose file is written and a monorepo has no shared/ to seed, so for those two the mounts are made after the first start and the import says to run noust update APP (container) or noust service restart APP (monorepo) to use them.
• What is not in it. An export without --with-secrets lists each file as skipped with noust app mount file APP TARGET --from FILE --mode MODE to give it afterwards. A type that takes no mounts (a static site, a Compose stack) skips them with the reason.

A container application's settings (image or Dockerfile, command and entrypoint, port, volumes, restart policy, build arguments) are in the same document, in a container section, so an imported container is created the way its original was. Its registry password and the values of its build secrets are Noust's own credentials and never go in: the registry account and the ids of the secrets are named, and the import lists each as skipped with the noust app container set command that gives it back. A clone takes the files of the original. A container is not cloned (it is on the in-place layout, which a clone refuses), so a clone never binds the paths of the original.

Files

The files of an application, from the console or the command line, as the application's own account, never as root: the kernel's permissions decide what can be read and written, exactly as for the application itself and for noust app exec. It is as powerful as a shell as the application, so every route asks for the terminal's permission (apps.terminal).

noust app files list shop.example.com uploads
noust app files get shop.example.com uploads/logo.png ./logo.png
noust app files put shop.example.com ./robots.txt public/robots.txt
noust app files mkdir shop.example.com uploads/2026
noust app files move shop.example.com uploads/a.png uploads/b.png
noust app files remove shop.example.com uploads/old

• A path is relative to the application's directory. It never holds .., a NUL byte or a control character, and a link in the tree (current is one, and a repository can plant others) is shown, can be renamed or removed, and is never walked through.
• .env files are not here. They are the environment's, with their own masking.
• Bounded: a file edited in the console is at most 5 MB, an upload at most 100 MB, a listing is cut at 2000 entries and says so.
• An edit or an upload is written beside the file and renamed over it, keeping the mode of the file it replaces. The audit trail records who, what and how many bytes, never the content.

Over the API: GET /api/apps/{domain}/files (list), DELETE (remove), GET/PUT .../content, GET .../download, PUT .../upload, POST .../directory and POST .../move. In the console, Files is in an application's More actions menu; the folder is in the address (/apps/shop.example.com/files?path=uploads).

Log shipping

The logs of applications, sent besides the journal to Grafana Loki, Axiom, New Relic, an HTTP endpoint (Fluent Bit, Vector, Logstash), an OpenTelemetry collector or a remote syslog.

printf '%s' "$TOKEN" | noust logdrain add grafana --kind loki --url https://logs.example.com \
    --tenant acme --label env=prod --app shop.example.com --token-stdin
noust logdrain list                 # each destination, how it is doing, what it has sent
noust logdrain test grafana         # one line, and what the destination answered
noust logdrain disable grafana
noust logdrain remove grafana
noust logdrain service              # the unit that runs the shipper

• New Relic (--kind newrelic) takes the license key as its token (sent as Api-Key) and --region us or eu (the Log API of that region; --url overrides it for an endpoint of your own). Each line carries its application, process, unit and level as attributes, and the --label pairs; a message longer than 4000 characters is cut, because the API refuses it.
• The token is only written. It is read from standard input (or a hidden prompt), kept in the secret store and never shown again; editing a destination without typing one keeps the one it has.
• Each line is scrubbed of the secrets of its application before it leaves.
• Without --app, all applications. Docker Compose stacks are shipped too, container by container, from where each left off.
• A destination that fails shows what it answered, verbatim. Its position only moves after the destination accepted a batch, so the next cycle sends the same lines again instead of losing them.
• Destinations are not trusted by accident. An address has no user or password in it. A plaintext http destination is allowed only on this machine (elsewhere it is refused under the ENS profile, and otherwise warned about: the console says Not encrypted). A syslog over tls checks the certificate and the name, with a CA file (--ca) or a pin (--pin-sha256).

Over the API: GET /api/log-drains (settings.read), and PUT and DELETE /api/log-drains/{name} and POST .../{name}/test (settings.manage). In the console it is Log shipping in the server's settings.

Changes not applied yet

A variable edited, a memory limit raised, a build command changed: each is stored the moment it is saved and reaches the process only at the next restart or the next build.

noust app pending shop.example.com

lists what the application is set to that it does not run with yet, and what each change needs:

The baseline is taken when a deployment succeeds and when the unit restarts, so it is always the configuration the application was last put to work with. The values of secret settings are never shown and never kept: a variable, a hook or a build block is compared by a salted hash, so the answer is "this changed", never "to what". Settings that are not secret (limits, the health check, the sandbox) say what they were and what they are. The console shows the same list as a banner at the top of the application, with a button for what a restart applies. GET /api/apps/{domain}/pending.

Scheduled tasks: what ran and what did not

A scheduled task (noust cron) is a systemd timer with Persistent=true: when the machine was off at the moment it was due, systemd runs it once at boot. It says nothing about the other moments that were due and are gone. The ledger sets every moment a schedule fell due against the runs the journal saw:

noust cron ledger nightly-backup --days 7

• It judges only from when it can. Not before the journal begins, not before the timer was written (the schedule may have been different), and not before the oldest run it read when the journal held more entries than it reads. The answer says from when.
• A disabled timer has nothing due. A moment still inside its two minutes is not judged yet.
• The same facts are in the timeline. A moment that did not run on time is an event of the jobs source (cron_missed, a warning for a missed one, a notice for a late one), with the task's unit and application, so the stretch around an outage shows what it cost.
• GET /api/cron/{name}/ledger?days=7 (apps.read) is the same answer; --json prints it.

Reading a deployment's log

noust app deploy-log shop.example.com                    # the latest deployment
noust app deploy-log shop.example.com 41 --level error   # only the errors of deployment 41
noust app deploy-log shop.example.com 41 --search prisma --no-timestamps
noust app deploy-log shop.example.com 41 --sanitize      # to paste it somewhere

The secrets the build knew are taken out when the log is written. --sanitize also masks what only looks like one (a connection string's password, a token, a key block) and strips control sequences, which is what the console's Copy to share does. --level warn keeps warnings and errors; the same filters are query parameters of GET /api/deployments/{id}/log. In the console a deployment's log has the same filters and says how many lines it hides.

Deleting an application

noust delete shop.example.com                       # asks for the name to be typed
noust delete shop.example.com --remove-volumes --remove-databases
noust delete shop.example.com --keep-files --keep-certificate -y
noust --dry-run delete shop.example.com --remove-databases     # what it would do, and nothing else

Deleting stops the units (or takes the Docker stack down), removes the site, the application directory, the certificate and the records. What holds data is never removed by default, and each is its own choice:

• The Docker volumes of a Compose stack or a container application go only with --remove-volumes (remove_volumes in the API). Without it the stack comes down and the volumes stay on the server.
• The databases go only with --remove-databases (remove_databases), and only the ones that belong to the application (made for it, backed up with it) and that no other application uses. Each is dumped first and the dump is reported (noust db drop does the same for a database on its own). A database the application merely uses, or that another application also uses, stays, and the deletion says so with the command that would drop it.
• The certificate goes with the application unless --keep-certificate says otherwise.
• The directory goes unless --keep-files; one outside Noust's apps directory (an adopted stack) stays unless it is named (compose.md).

The command asks for the application's name to be typed, not a y; -y skips it for scripts. Over the API, DELETE /api/apps/{domain} takes the same choices as query parameters (remove_files, remove_ssl, remove_volumes, remove_databases) and GET /api/apps/{domain}/deletion lists what the dialog that confirms it should show: the directory, whether it has volumes, and each database with whether it would be dropped and who else uses it.