Cloud storage and large data

RunMat cloud projects give Desktop, the browser, CLI workflows, Agent, and remote jobs a shared project filesystem. Source, notebooks, configuration, datasets, and generated artifacts stay attached to the project, with access controlled through its organization and membership.

Cloud storage changes where project data lives. Jobs still run on the user's machine by default. Remote execution is an optional target for workloads that need more memory, accelerators, concurrency, or different hardware.

One project across RunMat

A cloud project provides the same file tree whenever an authorized user opens it. Changes made in Desktop become available to the browser and connected tools after synchronization. A remote job receives the project inputs associated with its run and returns output to the same project history.

Use a cloud project when you need any of the following:

• Work from more than one computer.
• Share source and data with a team.
• Give Agent an authorized project instead of copying files into a conversation.
• Keep file versions and project snapshots.
• Submit optional remote jobs from the same project used for local development.
• Retain run artifacts alongside the source that produced them.

Local projects remain a good fit for offline work and data that must stay on one computer. You can move selected work into a cloud project when collaboration or managed storage becomes useful.

Source, datasets, and artifacts

Different project content has different storage and access patterns:

• Source and configuration are ordinary project files. They synchronize quickly and retain useful version history.
• Notebooks keep Markdown source in the project and can reference larger saved outputs.
• Datasets can be much larger than source and may be divided into independently transferred parts.
• Run artifacts include captured workspace values, figures, logs, traces, and other results selected by the project's persistence policy.
• Package content is resolved through the package workflow and lock state described in Package registry.

Keep source and small configuration files in the normal project tree. Place generated outputs under the configured artifact root. Use RunMat's dataset workflow for large, structured data so clients and jobs can access it without loading the entire dataset into an editor buffer.

Large data

Large uploads transfer in parallel and commit to the project after every required part arrives. A failed part can be retried independently, reducing the amount of data repeated after a network interruption. Large reads can use parallel ranges, and sharded datasets can fetch the parts needed by a workload concurrently.

Transfer speed is tied primarily to the available network and storage capacity. Projects can grow beyond the memory of a Desktop session because files and dataset parts transfer as needed. Storage capacity, interactive collaboration, and remote-compute capacity can scale independently.

Workloads dominated by many small files behave differently from workloads dominated by a few large datasets. Small-file projects spend more time listing, versioning, and synchronizing entries. Large datasets spend more time transferring bytes. For the best results:

• Combine very small generated records into a dataset format suited to batch access.
• Split very large datasets into parts that can transfer and retry independently.
• Keep dataset manifests and schemas small enough to inspect and version with the project.
• Read only the ranges or shards required by the current computation.
• Put re-creatable intermediate output under an artifact retention policy.
• Run compute near the data when repeated local transfer becomes the limiting cost.

Moving compute near the data is optional. Local execution remains the default for ordinary Desktop and CLI work. A remote target becomes useful when data transfer, memory, hardware, or concurrency exceeds the practical limits of the user's machine.

Versions and snapshots

Each committed file change creates a new project version. File history lets a user compare or restore an earlier state. A project snapshot records the versions across the whole tree, which is useful before a broad refactor, data migration, or coordinated release.

Restoring a snapshot creates a new current project state while preserving the history that preceded it. Team members should coordinate restores when other clients have unsaved or pending changes.

Version history and snapshots can retain data after it disappears from the current file tree. This is intentional: recovery depends on those retained versions remaining available.

Storage usage and retention

RunMat distinguishes the size of the current project tree from total retained storage. Retained storage can also include:

• Historical file versions
• Content preserved by project snapshots
• Dataset parts referenced by retained dataset versions
• Saved run and notebook artifacts

Deleting a current file may therefore leave the retained total unchanged until its history and snapshot references expire. The project storage summary shows current and retained usage so administrators can find the source of quota pressure.

Set retention according to the value and replacement cost of the data. Source history is usually inexpensive and valuable. Large generated artifacts may need a shorter window. Snapshot-pinned releases and irreplaceable datasets may justify longer retention.

Sharing and access

Project membership controls who can list, read, or change cloud files. Organization and project roles apply consistently across Desktop, browser, CLI, Agent, and remote execution. Removing a member ends future authorized access to the cloud project; exports already created by that member remain outside the project boundary.

Use separate projects when teams need distinct membership, retention, or data-residency policies. Keep credentials outside source, notebooks, and stored output. Remote jobs should receive only the secrets required for that run.

Deployment and recovery

RunMat Cloud operates the storage service as part of the hosted product. For supported private deployments, administrators choose storage capacity, quotas, retention, backup policy, and regional placement according to their organization's requirements.

A complete backup must preserve both project structure and file content from the same recovery point. Test restores should cover current files, historical versions, snapshots, datasets, and retained-storage totals. Capacity monitoring should track active usage, retained history, transfer failures, synchronization delay, and cleanup backlog.

Troubleshooting

• A file is missing: confirm the active organization, project, path, and project membership, then check file history.
• A collaborator sees an older file: preserve unsaved edits, reconnect synchronization, and refresh the affected project scope.
• A large transfer stalls: check local connectivity, proxy timeouts, available project capacity, and retries for the failed parts.
• A remote job cannot read data: verify that the input was committed to the cloud project and that the job uses the expected project version.
• Deleting files does not reduce usage: inspect retained history, snapshots, datasets, and saved run artifacts.
• A restore affects unexpected files: compare the snapshot with the current tree and reconcile any open editor buffers before continuing.

See History, snapshots, and synchronization for recovery and client synchronization. Current hosted-service allowances are listed on Pricing; private deployments use their configured capacity and retention policy.