basically Emscripten offers VFS as MEMFS or IDBFS or other variants ... the current one is always synchronous and instant, the IDBFS variant requires syncfs() calls to reflect the state or pause before reads cna be atomic/consistent.

If we provide an API we are probably better off by making this fully async: await read(...), await write(...), await stat(...), await mkdir(...) ... and so on.

If we just override the MEMFS mount point with an async IDBFS one, most code that work now might fail if users don't change their code too, due to the different nature of the two FS, one is sync, one is async.

Not sure I've explained it any better ... but to add anything, if a worker writes (or main) and then the other part reads without any pause on syncfs() the results might surprise while if both sides do syncfs() I don't know Emscripten internals enough to be sure that would work if the second call happens while the first one is still going but a new write was queued ... I assume yes, but we need some testing around this.

Thanks, yeah, that makes sense. It's a great topic for the community call tomorrow :)

Update ... I've just realized MicroPython doesn't include IDBFS so that FS.filesystems.IDBFS is undefined and we cannot use it.

However I've also failed at using the same /idbfs mount point in Pyodide in a persistent manner and multiple pyodide instances seem to be incapable of sharing the same FS ... likely I am doing something wrong but so far I had no luck at all with neither interpreters ... @hoodmane any idea why would that be or if it's even possible?

I'm not sure about mounting custom file systems at the root directory -- it seems unlikely to me to work. Try mounting in a new directory first.

I'm pretty sure that idbfs wasn't designed to be robust to sharing between multiple instances there will be a bunch of race conditions. You could try to write your own version that's thread safe but it's probably quite tricky.

You might want to check out browserfs:
https://github.com/jvilk/browserfs

I do have a module for that https://github.com/WebReflection/fs and it's been proven to work already as generic root mount point and to work reliably cross realms (main + worker) so it could work, but after trying with Pyodide to always bootstrap the same folder and read/write to it, always after explicitly awaited sync, the new pyodide can't find anything in there and I start wondering if you have any test page to understand where I am failing at using IDBFS in my code, thanks!

About that ... one thing comes to mind ... we do create object URLs for workers code and I wonder, at this point, if that might be the culprit: every runtime created object URL won't share the same IndexedDB as the main thread that created such object URL ... I will investigate this and provide answers and, if that's a browsers' issue, I am afraid we can't move forward with this idea as that would not work by standards meaning of objects URLs 😢

edit that being said, my tests with Pyodide were both on the main thread so something more to understand here to me.

Hi @hoodmane, I'm the maintainer of BrowserFS. I'd like to let you know that BrowserFS has been deprecated. You can read more about that on the BrowserFS readme (https://github.com/jvilk/BrowserFS). ZenFS is an updated and active fork that I created.

I can answer most questions:

• users have been asking about persistent data across sessions ... so you refresh, data is already there ... right now every refresh or session starts from nothing and localStorage or even sessionStorage are a (very limited and blocking) JS affair, not really Python
• the /home/{username} makes little sense here as the persistent storage is per browser (and per domain), not stored in the server ... if you want to create /home/{username} you can already do that with the current VFS in memory but it needs to be created every single time you go back to that page ... we can't prevent users to mess with the VFS but most importantly, IndexedDB has no race conditions issues, it's ATOMIC within its transactions and the only race is if you read data while another thread is writing data ... in such case you won't get the latest/greatest but we have SharedWorker to the resque, eventually, to guarantee that there is a single orchestration of the read/write cycle and we can synchronize in there all the things
• if we keep it minimal, stat, read, write, dir are all we need but I agree we should not reinvent the wheel, specially with a PL such as Python that already has its own FileSystem primitives ... that's why I'd like to be integrated eventually with the VFS in Pyodide or MicroPython so that they just use everything they use already but that would persist as IDBFS storage type, which is allowed by Pyodide but it doesn't work out of the box (it's absent in MicroPython but this is just Emscripten providing it, nothing to maintain from our side and nothing different from the current in memory VFS we use already).
• AFAIK IndexedDB is ACID and so is SQLite ... I've combined both already and it works great but it shares nothing across multiple instantiations ... as both APIs could be used in a SharedWorker and SQLite is very well documented and also already available in Python, all we could do is create a SharedWorker and provide a shared_sqlite module (not in stdlib) that could allow both Pyodide and MicroPython to share the same data per domain and browser, as it is already for IndexedDB and SharedWorker and dictated by Web standards security concerns, and offer an async SQLite API with ease ... without abstracting, just bootstrapping the right thing in the right way. This could be doable, but I am not sure if it fulfills current users requirements ... although having it always async removes the SharedArrayBuffer headers burden from the equation, working with or without those headers around.

edit I did write SharedArrayBuffer instead of SharedWorker, now corrected.