btoa(String.fromCharCode(...data))

this is a slippery slope for various reasons:

• JS runtimes have an arguments length limit that is not standardized, a bigger file (as in, bigger VFS) would fail here or there
• JSON is fully compatible with an array of 256 bytes (aka: Unint8Array) so that using a more fragile atob and btoa based approach, when proper binary is meant, feels both slower and underrated
• if it's binary we are saving, let's keep it binary because both JS and Pyodide/MicroPython understand binaries pretty well, with or without JS atob and bota or base64 dependency / transformation involved

That being said, everything else is more than welcome but I need to double check technically this is the best we can offer although I do admire the effort and ideas you put into this to date and I'd be more than happy to finalize a concrete solution around this, of course with your review (or mine) needed to validate we're all good with the idea/proposal and results ... so, thanks a lot for digging this path, it all makes sense to me, the devil is in the details and we should try to provide the best/fastest details we can around it, I hope you would agree on that.

Thanks for the response. Yeah, I know about the btoa problems and, as I mentioned in the call, I used this approach because it was simple and easy and not really important to the core concepts in this discussion (i.e. we can use something else as we refine a potential solution ~ base64 encoding of file content is sub-optimal and was only used for convenience; it also demonstrates my relative lack of JS knowledge in this area, so I'll defer to your suggestions in this regard).

The important point I wanted to make was it appears we could freeze the filesystem relatively easily via the Emscripten FS API.

But as you say, the devil is in the details. But that's where the fun lies. 😉

"Devils to be detailed" that I want to explore 😈:

• Do we need to freeze the whole filesystem? Perhaps we just freeze directories on the user's "path"; or WASM-y equivalent thereof. Places like lib/python3.13/site-packages (where packages reside - presumably in the case of Pyodide, we need to check) or the user's home (where files end up) immediately spring to mind. I wonder where mip (in MicroPython) puts its assets? I vaguely remember this might be the user's home directory.
• DX/tooling: in a similar way to your recent work around Positron dependency capture via a headless browser, perhaps we could take a similar approach so folks can pip install pysfreeze (for example) and just pysfreeze inside the directory of their PyScript project to produce a project_name.frz "image"..? (.frz for "freeze")
• PyScript configuration: we already have entries for packages and files but perhaps we need a restore entry that points to the URL of the project_name.frz image? Also, this should be a logical "OR". Either you have packages and files OR just restore in your PyScript config, and we'll raise an error if you have both.
• Definition of idiomatic workflow: clearly you need to specify the packages and files BEFORE you freeze the filesystem. But our tooling could, with the aid of a flag (so the user has a choice), read the referenced configuration file for PyScript and remove the previously needed packages and files entries before replacing them with a restore section pointing at the resulting ./project_name.frz..?

@ntoll I took a chance to revisit your logic so these are my variants:

• no string concatenation
• no string conversion (aka: no base64 involved)
• no splits needed
• what's passed is the FS not a wrapper that needs to have an FS entry ... this feels better because interpreter in there is never really useful, it's just the FS that is expected to be an Emscripten one, the fact it's reachable via interpreter.FS is irrelevant for this logic

function asNode({ mode, mtime }, data) {
  return { mode, mtime, data };
}

function freezeDirectory(FS, path) {
  const entries = FS.readdir(path);
  const result = {};
  for (const entry of entries) {
    if (entry === '.' || entry === '..') continue;
    const fullPath = path === '/' ? '/' + entry : path + '/' + entry;
    try {
      const stat = FS.stat(fullPath);
      if (FS.isDir(stat.mode)) {
        // Directory: recursively freeze its contents.
        result[entry] = asNode(stat, freezeDirectory(FS, fullPath));
      }
      else if (FS.isFile(stat.mode)) {
        const data = FS.readFile(fullPath, { encoding: 'binary' });
        // TODO: the [...array] conversion *might* not be necessary,
        //       although this way it works as JSON too.
        result[entry] = asNode(stat, [...data]);
      }
      else if (FS.isLink(stat.mode)) {
        result[entry] = asNode(stat, FS.readlink(fullPath));
      }
    }
    catch (e) {
      // Log stuff that can't be accessed.
      console.warn(`Skipping ${fullPath}: ${e.message}`);
    }
  }
  return result;
}

function freezeFileSystem(FS) {
  return freezeDirectory(FS, '/');
}

function restoreDirectory(FS, path, frozen) {
  for (const [name, node] of Object.entries(frozen)) {
    const fullPath = path === '/' ? '/' + name : path + '/' + name;
    if (FS.isDir(node.mode)) {
      try {
        FS.mkdir(fullPath);
      } catch (e) {
        // Directory already exists, just ignore.
      }
      restoreDirectory(FS, fullPath, node.data);
    }
    else if (FS.isFile(node.mode)) {
      FS.writeFile(fullPath, Uint8Array.from(node.data));
      FS.chmod(fullPath, node.mode);
    }
    else if (FS.isLink(node.mode)) {
      FS.symlink(node.data, fullPath);
    }
  }
}

function restoreFileSystem(FS, frozen) {
  return restoreDirectory(FS, '/', frozen);
}

now ... this version of mine:

in comparison, for some reason, your version fails at restoring so that all timings are kinda irrelevant but I still believe my version is easier to reason about, there's less magic involved, those buffers will compress well after gzip ... what do you think?

your screenshot, if interested:

OK, keeping the discussion relevant, now that we have an easy/fast enough way to produce a fs.json file #2417 (comment) that works with any file size, what I'd like to test now is how long would it take to have matplotlib as package so that:

• we use our current cache to compare bootstrap time
• we freeze that into an fs.json
• we change the config to not include matplotlib, yet ...
• we use the very same code to import from it and test that everything is fine

My concerns I'd like to validate is that micropip might be deeply integrated in Pyodide, as example, so that maybe if modules don't get to be requested via it or registered somehow, things might break ... but I think we need to be sure that once we manage to freeze the FS, bootstrapping from it will actually work.

In an ideal world, we should be able to bootstrap the FS ourselves automatically, but let's do one step after the other ... I will update this with results, once I have some!

Update

I have everything finally working as desired:

• the operation takes a few milliseconds to create a Blob (which carries an ArrayBuffer) that contains all details for files, directories, and links
• the buffer is linear and using matplotlib as imported package, it produces 39MB of raw file VS 125MB that JSON for the very same FS graph
• once compressed and saved as fs.gz smart CDNs or even GitHub pages will not re-compress it, and we'll be down to 14MB instead
• with both compression and fetching it takes a few milliseconds to retrieve the frozen FS
• to restore 39MB of data it takes consistently less than 100ms ... meaning Pyodide with matplotlib and all related dependencies can bootstrap in less than 2 seconds, fetching included 🥳
• if the blob is stored into IndexedDB it takes nothing to read it (not compressed) and restore it

I've published a module so I could test online and on GitHub pages with ease, the module is called emscripten-fs-blob and you can test it live on GitHub pages (remember to open the devtools console).

That live demo bootstraps pyodide with matplotlib and everything else without needing a config to do so, and it "smokes" any other variant/cached alternative we offered to date but keep in mind we are using IndexedDB to cache the frozen micropip env, here we're using a fetch to retrieve the data.

If needed, I can improve the test page to compare gzip VS plain Blob so we can see how the browser cache could help too in there because decompressing is actually not super fast despite me using the latest/greatest Web APIs to do so.

The streaming bit has been left behind because pipeThrough is super slow with native APIs, I won't expect me implementing some convoluted logic around it being faster than just "fetch it all and parse", which is the current approach (edit: approach that is still compatible with streaming as bytes are ordered and it's linear 0 to buffer size operation).

Here a screenshot of various tiny benchmarks and related details (around operations):

I am confident you'll have similar results on your machine but we all know CPU matters specially for decompression related operations, yet I am extremely happy and glad this variant of @ntoll original idea really works well.

To be discussed

Part of the final size amount is due mtime which is nowhere used ... I think it was a good idea to have it in there, but I am not fully sure (if we don't use it) why that's part of the buffer ... I can see moreorchestration around the fact if such mtime (last modified time) is less than current file/folder/link, because that also exists, we could skip override and just use latest, resulting in hopefully less conflicting situations where Pyodide might be a different version and overwriting stuff that's newer might cause troubles because here we freeze the whole thing and the whole thing doesn't get versioned in the FS (i.e. site-packages for python3.13 ... that doesn't reflect the Pyodide version anywhere).

As summary

• yes, we can freeze the whole FS in a fast and portable way
• yes, frozen FS can be bundled within an app, for both Briefcase/Positron and PWA or any other offline/online scenario
• the binary (Blob/Buffer) format wins for most operations except it won't be "human readable" like JSON is, although I believe that's a good thing
• the project works on the front-end like in the backend, this opens doors to Pyodide frozen FS out of Cloudflare Workers and whatnot, but it also can work wonderfully on PyScript .com or GitHub pages

Happy to clarify or expand on any point, right now I hope this is a welcome feature pre x-mas time, and that people would play around and see how powerful is this possibility to entirely freeze a whole environment so that it will reproduce exact same results every single time it gets to be bootstrapped 👋

Wow, this feature has really developed quickly!

I think what you've got here suits PyScript well. But let me just say a few words related to MicroPython and its existing ROMFS feature:

1. A ROMFS is created "offline", currently using mpremote romfs build <directory>. It just scoops up the entire directory and turns it into a frozen filesystem.
2. The resulting file/data then needs to be made available on the target you want to use it with. For microcontrollers, that means writing the data into flash somewhere. For webassembly, it means having the data as a Uint8Array accessible from Emscripten.
3. A key feature of ROMFS is that it's read-only-memory and actually must be available in addressible memory, that MicroPython can read using standard memory read operations. That leads to a lot of nice optimisations: eg (1) precompiled bytecode is executed inplace from the ROMFS without any need to copy it into RAM; and (2) large assets like fonts and images can stay in the ROMFS and be used directly without loading them into RAM either.
4. The benefits of (3) are also available to MicroPython webassembly: precompiled bytecode is instantly executable, and assets are essentially already loaded into memory, ready to use.

So I think ROMFS would still have its place in MicroPython webassembly, independent to PyScript's frozen filesystem feature. Or maybe we could somehow make a PyScript frozen filesystem be also memory mappable... then bytecode could be executed directly from it.

I have a branch with ROMFS enabled for the webassembly port here: https://github.com/dpgeorge/micropython/tree/webassembly-combined-patches . To use it it's very simple, you just need to load a ROMFS as a Uint8Array and then pass it into the interpreter constructor, eg:

const { loadMicroPython } = await import(`${base}/micropython.min.mjs`);
const romfs = new Uint8Array(await (await fetch("app.romfs", {responseType:"arraybuffer"})).arrayBuffer());
const mp = await loadMicroPython({ url: `${base}/micropython.wasm`, romfs: romfs });