Runtime, model destination & eval
A model is an opaque, signed payload - the weights. The runtime is the container that loads those weights, serves inference, and scores a candidate model against a golden set. Drawing that line is what lets the platform do three things well: attach the bill of materials and vulnerability watch to the real dependency surface (the runtime), deliver and roll back the model independently, and gate a rollout on how the new model actually performs, not just whether it loads.
This page takes you end to end: build the runtime, decide where the model lands, get the golden set onto the device, and compose a model + runtime bundle that advances and rolls back as one unit on a quality check.
1. The runtime container
Your runtime is an ordinary container with three responsibilities. A reference implementation (an inference/health server plus an eval entrypoint) is provided to start from; the pattern is small:
FROM python:3.11-slim # multi-arch: works on arm64 (Jetson) and amd64
RUN pip install --no-cache-dir onnxruntime numpy # your inference surface = the SBOM
WORKDIR /app
COPY serve.py eval.py entrypoint.sh /app/
# The model is mounted here read-only; the runtime never bakes a model in.
ENV MODEL_PATH=/models/model.onnx
EXPOSE 8080
ENTRYPOINT ["/app/entrypoint.sh"] # `serve` (default) | `eval`
CMD ["serve"]
For GPU or TensorRT, swap the base image for your accelerator's base and install
the matching runtime - the contract below (a health signal, a fixed model mount,
and an eval mode) does not change. Attach this image's SBOM to the runtime
artifact so vulnerability scanning tracks exactly its dependencies.
Mark the container as a runtime when you publish it (the publish form has the checkbox). That is what lets it be paired with a model as the runtime member of a bundle.
2. Where the model lands
The runtime needs to read the model from a known path. Publish the model with a
destination - the absolute path the signed weights are placed at on the device,
for example /var/lib/meshanics/models/model.onnx. The device must allowlist that
path prefix (a one-line agent setting), so a payload can only ever land somewhere
the operator already permitted.
Your runtime reads that path by mounting it read-only through the container's run arguments:
runtime run args: --volume /var/lib/meshanics/models:/models:ro
runtime reads: /models/model.onnx (MODEL_PATH)
The model is delivered, versioned, and rolled back on its own - drop a new model at the same destination and the previous one is restored automatically if the update is unhealthy.
Three ways your runtime adopts a model
How a model takes effect depends on how your runtime reads it. Pick the mode that matches yours - the platform delivers, verifies, versions, and rolls back the weights in every case; the difference is how the live runtime picks them up.
- Standalone (no destination). The agent stages the model in its own data directory and keeps the previous version. Use this when your runtime watches a path and hot-reloads (e.g. Triton in poll mode, TorchServe), or when a batch or on-demand job reads the current model on its next run. Placement itself is the signal; no destination needed.
- Destination + reload. Set a destination so the weights land at a fixed path, and add a reload (a systemd unit to restart or a command to run). Use this for a runtime that loads the model once at startup and does not watch the file - the reload tells it to pick up the new weights after the swap. Without it, a load-once runtime keeps serving the old model even though the new one is in place.
- Model + runtime bundle. Pair the model with its runtime container in a bundle; the agent restarts the runtime as part of applying the bundle, so the restart adopts the new model, and the model and runtime advance or roll back as one unit.
A note on adoption mode: the model-eval probe scores the freshly delivered weights, so choose the adoption mode (watch, reload, or bundle restart) that ensures your live runtime actually serves the version that was scored.
Leave the destination blank only when your runtime watches the path or re-reads on its own (the first mode above).
3. The golden set
The eval scores the new model against a small labeled dataset - the golden set. Get it onto the device one of two ways:
- As a signed config artifact placed at a known path (e.g.
/var/lib/meshanics/golden/golden.npz), delivered and verified like any other artifact. - Baked into the runtime image, so it versions with the runtime.
Keep it small and representative - it runs on the device on every rollout, so a few hundred labeled examples that capture the cases you care about beats a large set that makes the gate slow.
4. The eval
A model-eval health probe runs an allowlisted binary on the device and reads the score it prints. The eval's only job is to measure: load the candidate model, run it over the golden set, and print one line of JSON.
{"metric": "accuracy", "score": 0.94}
The pass/fail threshold lives in the signed rollout, never in the eval - so a tampered eval cannot quietly lower the bar, and the operator sees exactly why a device rolled back. The agent provides the path to the freshly delivered model and the golden set through the environment; your eval reads them and scores.
Because scoring needs the same framework and libraries as inference, the recommended
shape is to run your runtime image's eval mode against the staged model and
golden set, so the eval versions together with the runtime.
5. Compose and roll out
Now assemble the pieces:
- Publish the runtime (kind container, marked as a runtime, with the model volume in its run args), the model (kind model, with a destination), and the golden set (kind config, with a destination).
- Bundle the model and the runtime - pick each from the registry by role on the Bundles page. A bundle is the unit that gets tested and rolled out.
- Roll out the bundle to a group with a model eval health probe pointing at
your eval binary, your golden set, and a
min_score.
bundle vision-stack@1.0.0
model yolo@2.1.0 -> /var/lib/meshanics/models/model.onnx
runtime vision-runtime@1.0.0 (mounts /var/lib/meshanics/models:ro)
rollout -> plant-7
probe: model eval, min_score 0.90, golden /var/lib/meshanics/golden/golden.npz
strategy: 10% canary -> 100%
The agent applies the model and the runtime as one transaction: it places the model, starts the runtime, and runs the eval against the new weights. If the score clears the bar, the bundle commits. If it falls short - a model that loads fine but performs worse - the whole bundle reverts together: the model goes back to the previous weights and the runtime to its previous version, in one step, on the device, without reaching the platform.
That is the quality gate the platform adds on top of crash-only rollback: a model that runs but is worse never silently stays in production.