Remote execution

RunMat jobs execute locally on the user's machine by default, including jobs opened from cloud projects. Remote execution is optional: use it when a workload needs more memory, more or different accelerators, greater parallel capacity, or an administered environment beyond the local machine. When selected, Server coordinates the run and enrolled compute nodes execute it with the open-source Runtime. A remote run binds an authorized project revision and entrypoint to a target, environment, resource policy, lifecycle, logs, and result artifacts.

The usual flow is: submit, authorize, queue, lease to an eligible node, stage inputs, execute, stream status and logs, commit artifacts, and finalize usage. Cancellation and lease expiry are explicit states. Workers must make terminal updates idempotent because retries can cross network failures.

Administrators define cluster and node enrollment, supported Runtime versions, hardware labels, concurrency, network access, artifact retention, and secrets injection. Worker credentials carry the permissions required for worker duties and exclude organization administration.

When a run is queued, check target eligibility and capacity. When staging fails, check project/blob access. When execution fails, inspect the Runtime diagnostic and captured environment. When finalization fails, preserve the worker output until Server confirms artifact commit.

For project-side commands and worker details, see Runtime remote execution.

Select a remote target

The client shows remote targets only when the deployment has enrolled capacity and policy allows the project to use it. Choose one for a workload that exceeds local memory or accelerator capacity, needs more concurrent jobs, or must run in a centrally managed environment. For ordinary development, keep the local target.

A target can advertise architecture, CPU/GPU labels, Runtime versions, concurrency, and environment capabilities. Submission records the project revision, entrypoint, arguments, requested target, and declared inputs. Pin dependencies and data identities so a worker can stage the same job later.

Run lifecycle

1. Server authenticates the caller and checks project and compute policy.
2. The run enters the queue with an idempotent submission identity.
3. An eligible node leases it for a bounded period.
4. The node stages source, dependencies, datasets, and scoped secrets.
5. Runtime executes while the node publishes status, logs, and heartbeats.
6. The node uploads result artifacts and sends a terminal update.
7. Server finalizes usage and exposes the completed run to clients.

Lease expiry returns abandoned work to recovery policy. Terminal updates and artifact commits are idempotent because a worker can lose connectivity after completing one side of the protocol. Cancellation moves through requested and acknowledged states. The node or lease confirmation moves the UI into the cancelled state.

Node security and operations

Use a worker-specific credential with cluster duties only. Isolate jobs, restrict host mounts and network egress, inject secrets per run, and remove them after execution. Define image/Runtime rollout, drain behavior, concurrency, disk cleanup, artifact retention, and node quarantine.

Monitor queue time, lease acquisition, heartbeat age, stage duration, execution duration, cancellation latency, artifact finalization, and retry count. A queued run points to target policy or capacity. A staging error points to project/package/blob access. A Runtime error belongs to the workload. A finalization error requires preserving local worker output until commit succeeds.