Health probes & rollback

Defining health probes

For container releases, Meshanics uses two separate health gates. They run at different times and in different execution contexts:

Gate	Where it runs	What it protects
Signed candidate check	Inside an isolated candidate container, before cutover.	Prevents an unhealthy candidate from replacing the incumbent workload.
Post-activation probe	From the device host, immediately after cutover.	Confirms the activated workload is serving correctly and triggers rollback if it is not.

The signed candidate check is part of the immutable artifact definition. A rollout cannot override it. To change that command, publish a new artifact revision with Correct on the Artifacts page. The correction can reuse the same pinned image digest while changing the signed candidate definition. The post-activation probe belongs to the rollout and can be selected when the rollout is created.

Container candidate checks

The candidate starts without its live host ports. Its signed command is executed directly inside the container, using binaries and paths provided by that image. It must address the container port, not the eventual host port.

For an nginx image mapped from host port 8080 to container port 80, the two checks might be:

Candidate check:       curl -fsS http://127.0.0.1/
Post-activation probe: http://127.0.0.1:8080

The candidate command must use a client that exists in the image. Meschanics does not inject curl, wget, or a shell into customer containers.

Probe types

The rollout form defines the post-activation probe. These probes run from the device host, not inside the workload container.

Type	What it checks	Healthy when
HTTP	A `GET` from the device host to a URL your workload exposes, typically through its published host port.	The response status is `2xx`.
File	A file path on the device - for example, a readiness file your app writes once it is up.	The file exists and is non-empty.
Exec	An allowlisted health-check program on the device host.	The program exits `0`.
Model eval	An allowlisted evaluator that scores the activated model against a local golden set.	The reported metric meets the configured minimum.

The file probe pairs well with applications that write a readiness or heartbeat file on the host. The HTTP probe suits services that publish a health endpoint. The exec probe is the most flexible and the most tightly controlled - see below.

Timing and retries

Each probe carries a few timing controls so you can match it to how long your workload takes to come up:

Initial delay - wait this long after the swap before probing the first time, giving the workload room to start.
Timeout - how long a single attempt may take before it counts as a failure.
Attempts - how many times to try before declaring the update unhealthy (always at least one). Attempts are spaced out, so a slow start can still recover.

Rollback only triggers once every attempt has been used.

Exec probes are allowlisted, never a shell

The exec probe is deliberately constrained. The platform is the supply chain for these devices, so an exec probe must never become a way to run arbitrary code as root.

The target must be an absolute path to a binary that the device operator has put on that device's allowlist. If the path is not absolute or not on the allowlist, the probe is refused.
The binary is executed directly, with no shell - there is no sh -c, no argument string, and no interpolation. A check that needs arguments is wrapped in an allowlisted script on the device.
By default the allowlist is empty, so exec probes are refused until an operator explicitly permits specific binaries on that device.
Configure it during install with --probe-exec-allow /absolute/path/to/check. The installer persists the list as MESHANICS_PROBE_EXEC_ALLOW in /etc/meshanics/agent.env; restart meshanics-agent after changing it by hand.
Multiple trusted paths may be supplied as a comma-separated list. The allowlist remains local to each device and is never widened by a rollout.
The active artifact's path is passed to the binary through an environment variable, so a single health-check program can inspect whatever version is currently live.

This means the probe definition that travels with a rollout cannot, by itself, cause a device to run anything new: it can only invoke a binary the device's own operator already trusts.

Choosing a probe

A good probe fails fast and unambiguously on a bad version, and only reports healthy once the workload is genuinely serving. Prefer checks that exercise the real path - a readiness endpoint or file the app controls - over checks that merely confirm a process started. The stricter the probe, the safer the automatic rollback that backs it.