MeshanicsDocs
AI/ML model OTA

Model quality telemetry

Signed model delivery does not require changes to your model code. Model Quality is a separate, optional integration: one fail-open SDK call reports confidence, latency and error statistics from your serving loop. The device agent aggregates them into time windows and attaches the identity of the model it verified and installed. Raw inputs and outputs are never sent.

Model Quality is observe-only. It helps you compare model versions and hardware profiles, but it does not automatically halt or roll back a rollout. Use a signed model eval rollout probe when a candidate must pass an on-device quality threshold before it is accepted.

Before you start

Install the current Meshanics agent using the command generated in the console. Current onboarding enables the local telemetry receiver automatically on both direct and gateway-connected Linux devices.

Verify that the receiver is ready:

sudo test -S /run/meshanics/ml.sock && echo "ML receiver ready"
sudo journalctl -u meshanics-agent -n 50 --no-pager | grep "ml quality receiver"

The socket appears only while the agent is running. Existing devices installed before this capability was enabled should be reinstalled with a fresh console command so their service environment contains the device REST endpoint.

Install the Python client

Pin the public SDK release in the same environment as your inference process:

python3 -m pip install \
  "https://github.com/meshanics-com/ml-sdk/releases/download/v0.1.1/meshanics_ml-0.1.1-py3-none-any.whl"

For an offline image, vendor src/meshanics_ml/__init__.py from the tagged SDK repository. It uses only the Python standard library and performs no network or disk I/O.

Report one inference

import time

from meshanics_ml import report_inference

started = time.perf_counter()
detections = model(frame)

report_inference(
    scores=[float(d.confidence) for d in detections],
    latency_ms=(time.perf_counter() - started) * 1000,
    ok=True,
    seq=frame_number,
)

Scores must be finite values from 0 through 1. An empty score list is valid and records an inference with no detections. Set ok=False when inference fails. The optional monotonic seq lets the agent account for missing datagrams.

The call has no retries, network connection, disk spool or background thread. It sends one non-blocking Unix datagram. A missing agent, full socket buffer or bad value is counted and immediately dropped without raising into your application.

from meshanics_ml import dropped, sent

print({"sent": sent(), "dropped": dropped()})

sent() means the local socket accepted the datagram. It does not mean the control plane has stored the resulting aggregate yet. Charts update after the agent closes and forwards its current window.

What leaves the device

The agent forwards windowed counts and histograms:

  • inference and error counts
  • confidence distribution
  • latency distribution
  • local drop and sequence-gap counts
  • model name, version and digest stamped from installed state
  • device and hardware profile derived from the authenticated device identity

It does not forward individual inference records, raw inputs, raw outputs, images, prompts or model payloads. input_summary is reserved by the versioned local protocol and is not included in the current fleet aggregates.

Direct and gateway-connected devices

Direct devices send aggregate windows to the cloud device REST endpoint using their existing mutual-TLS identity. Gateway-connected devices send to the gateway's opaque device REST relay on port 9445; device TLS still terminates at the control plane. The SDK itself never holds a credential and never talks to a gateway or cloud service.

In courier-only operation with no live control-plane connection, aggregate windows are not spooled on the device or gateway. Reports remain fail-open and are dropped until the device REST relay is reachable again. Signed delivery, local model-eval gates and rollback continue without this observational channel.

Troubleshooting

If Model Quality stays empty:

  1. Confirm a model artifact is installed. The agent refuses to attribute statistics until it has a committed model identity.
  2. Confirm /run/meshanics/ml.sock exists.
  3. Check dropped() in the serving process.
  4. Check the agent journal for receiver or forwarding errors.
  5. Confirm /etc/meshanics/agent.env contains MESHANICS_ML_METRICS_SERVER=https://<device-host>:9445.

Do not manually invent or copy another tenant's endpoint or certificate. If the setting is absent, reinstall from the console so onboarding writes the correct host, gateway route and trust material together.

Related