Our current config.toml (or .json) offers definition of both packages and files but it became obvious, over time, that a lot of projects using [files] are meant to bring in "custom packages".

Not only the question around GitHub integration came up in various occasions, the reason for such question was almost always around the fact people would like to rather host their "custom packages" elsewhere, and these are the most common/obvious reasons:

• cloning a PyScript project on pyscript.com is super simple, yet if the initial project changes, such changes don't automatically get propagated. This approach has pros and cons:
  • Pros: there is no code/logic hijacking at distance, if the original source of truth suddenly changes or breaks cloned projects won't be affected
  • Cons: there is literally no easy way to keep cloned projects updated, when the original source of truth is trusted or well maintained to grant, somehow, no breaking
• adding a list of files on PyScript .com is not as straight forward as working in your IDE of choice
• packages are easy to reason about, the process to officially publish these so that these can be available through both Pyodide and MicroPython is not clear, or even defined, at all
• we do support remote/foreign wheels through .whl suffixed remote packages, yet we haven't documented, explained, or provided, a way to create such packages
• the previous point is not standardized across Pyodide and MicroPython

Proposal

As opposed to asking everyone to update each file for each cloned/forked version of the original project, we could extend with relative ease the packages list of dependencies itself:

• if the entry is a canonical / fully qualified URL and:
  • it ends with .whl, we import that remote package in both Pyodide/MicroPython assuming it's a Pure Python module that "just works"
  • it ends with .zip, .tar.gz or .tgz, we fetch that remote file at runtime and extract its content within the Virtual FileSystem automatically
  • it ends with .toml or .json, we merge that remote config resolving its own [files] definition accordingly with the original fully qualified URL

About Disambiguation

The current [files] directive places sources within the Virtual FileSystem (VFS from now on) and it's already extracting .zip and .tar.gz files automatically but that's some DX stretch we backed into the config logic.

Arguably, it's not clear if anyone actually needs to truly deliver a compressed asset, why we would do that, while when it's intent comes from the packages list, it's clear that MUST be done in order to allow such package to work.

On the other side, if packages contains something like .toml or .json files from a URL, it's obvious that's about a foreign config that should provide all information needed to bring in such module, so that forks will fetch at runtime that config and understand what to do without guessing.

The config format

I believe it would be silly to reinvent the wheel here, so that I hope we agree that using exact same format we have for PyScript's configs would be lovely.

My thinking is that a foreign package would contain at least these fields:

name = "the_package_name"

# any other PyScript related flag ... including ...

[js_modules.main]
"https://lovely.cdn.service/main.js" = "js_dependency"
"https://lovely.cdn.service/main.css" = "js_dependency"

# the module structure

[files]
"./__init__.py" = "./"
"./logic.py" = "./"
"./extra/__init__.py" = "./"
"./extra/logic.py" = "./"

As the Web cannot crawl remote folders, and because we already have a way to relatively bind files, assuming that packages entry was https://remote-website.com/module/config.toml:

• we create in the VFS the the_package_name folder
• we add in it:
  • the_package_name/__init__.py
  • the_package_name/logic.py
  • the_package_name/extra/__init__.py
  • the_package_name/extra/logic.py
• we merge the amount of js_modules.main or js_modules.worker within the current config
• we keep processing the rest of the config as we've always done

Doing it this way, if a project contains the following config

# A PyScript basic config with a remote package
packages = ["matplotlib", "https://remote-website.com/module/config.toml"]

The final running code could:

import matplotlib
import the_package_name

and no error would be shown.

Benefits

This approach would solve and answer the following requirements / questions:

• "how about GitHub integration?" - you can host your project and provide a raw.github... url that points at that project config. This way nobody needs to use CLI to create .whl, .zip or other archives, this way every single forked project will automatically keep up with latest version
• "how about non-breaking forking?" - it's up to users to use a commit based config file instead, or abuse by any mean the remote static host they are pointing at to provide such package
• "do I need to specify files as I used to before?" - you can specify files for other assets, you can however now just explicitly indicate that group of files is about a module which goal is to be imported, or available, in PyScript code

And that's all folks, I am looking forward to discuss any gotcha, hint, improvement, idea around this topic, but I think this (at least intermediate) step would improve by far foreign packages integration, sharing of PyScript projects that are well maintained and whatnot 👋