Work with large projects and data
A cloud project can hold far more data than fits on a laptop. Desktop keeps only the part of a project you are using on your device, and brings the rest from the server when your code or editor asks for it. Your own edits stay on the device until the server has them.
Where project data lives
| Tier | What it holds | Size | Lifetime |
|---|---|---|---|
| Memory | Recent file reads and folder listings for the open project | Up to 256 file reads and 256 folder listings per filesystem client | The session; cleared when the file changes |
| Disk cache | Files you have opened or your code has read, as downloaded copies | 10 GiB of file content per project | Kept across sessions, cleaned up in least-recently-used order |
| Server | Every file, every saved version and every dataset chunk in the project | No per-project size limit in Desktop | Kept with the project and its history |
The installed app keeps its disk cache on your computer. In the browser, the same app uses browser-managed storage for editor recovery and short-lived caches, and reads other content from the server each session. Local folder projects don't use these caches: their files are the files on your disk.
Large datasets load in pieces
Store large arrays as datasets rather than single files. A dataset is split into chunks, and reading a slice downloads only the chunks that slice touches. A script that reads one day of sensor data from a multi-terabyte dataset transfers that day's chunks, not the dataset. Writes go to the chunks they change. This is how a project can reach terabytes or petabytes of data while a run on a laptop stays fast.
Set the chunk shape to match how the code reads the data; see Choosing chunks.
How the disk cache cleans up
When the downloaded content for a project passes 10 GiB, Desktop removes the least recently used clean files until the cache is back to 8 GiB. A clean file is one whose content matches the server.
Cleanup never removes:
- files with edits that haven't reached the server;
- files with pending moves, renames or deletes;
- the project's file tree and file details, so a removed file still shows in the tree and downloads again when opened.
Removing a cached copy doesn't touch the file on the server or its history. Package storage and saved run artifacts are managed separately and don't count toward the 10 GiB.
Shared-editor recovery copies have their own cleanup: copies of closed documents whose edits have reached the server are removed after seven days, or sooner to stay within 128 MiB per account. Pending edits, review proposals and documents open in any window are kept.
Change the cache size or location
Not yet releasedStatus Settings › Storage sets the disk cache size per project and for the whole app, moves the cache to another drive, and clears the cache for one project or for all of them. Clearing keeps files with edits that haven't reached the server. Until it ships, the limits above are fixed.
Work offline with a large project
Open the files and run the code you need while connected, so their content is in the disk cache. Datasets need the chunks your code will read; running the script once while connected brings them down. See Work offline and recover edits for what works during a disconnect.