Skip to content

[DEV]: add pixi.lock and add pixi to pyproject.toml - #31384

Open
jklymak wants to merge 16 commits into
matplotlib:mainfrom
jklymak:dev-add-pixi-toml
Open

[DEV]: add pixi.lock and add pixi to pyproject.toml#31384
jklymak wants to merge 16 commits into
matplotlib:mainfrom
jklymak:dev-add-pixi-toml

Conversation

@jklymak

@jklymak jklymak commented Mar 25, 2026

Copy link
Copy Markdown
Member

Edit: apologies for the CI churn --- I needed to iterate on the Pixi workflow to get it working end-to-end.

Overview

This PR adds Pixi configuration to pyproject.toml and introduces a committed pixi.lock.

The main goal is to provide a project-committed, pinned dependency snapshot for Matplotlib, addressing #23548. In particular:

  • pyproject.toml defines the Pixi environment under [tool.pixi]
  • pixi.lock records the fully resolved dependency set
  • CI exercises that lockfile to verify that it can be used reproducibly

Why pyproject.toml

Pixi configuration is placed in pyproject.toml rather than a project-level pixi.toml.

The reasoning is:

  • pyproject.toml is already the canonical project configuration
  • it keeps the shared Pixi definition attached to the project itself
  • developers who want local customization can still create an untracked pixi.toml, which Pixi will prefer over pyproject.toml

This gives the project a canonical shared configuration while still allowing personal overrides.

Lockfile policy

A main goal of this PR is to commit a pinned dependency list, and pixi.lock is the mechanism used for that.

pixi.lock should therefore be treated as part of the repository state:

  • changes to Pixi dependency inputs should be accompanied by a regenerated pixi.lock
  • CI uses the lockfile to verify reproducibility
  • old commits retain the dependency snapshot that was current at that point in history

In that sense, pixi.lock is the committed pinned dependency snapshot for the project.

Developer workflow

Basic usage:

pixi install

pixi run python -m pip install \
  --no-build-isolation \
  --config-settings=setup-args="-Db_lto=false" \
  --editable .

After that, for example:

pixi run tests-core
pixi run build-doc

Dependency changes:

If dependencies change, then pixi.lock should be regenerated. It is ignored by default in .gitignore so developers who want to change pixi.lock will need to regenerate it, and force add it to a commit:

mv pixi.toml pixi.backup
pixi lock
git add -f pixi.lock

tools/pixi.example.toml

A template is provided for developers who want to customize their local Pixi setup.

For example:

cp tools/pixi.example.toml pixi.toml

This is intended for local experimentation or system-specific tuning. It is not the project's canonical Pixi configuration.


Key design decisions

Editable install is not managed directly by Pixi

This PR intentionally does not use:

    [pypi-dependencies]
    matplotlib = { path = ".", editable = true }

In testing, that led to lockfile instability (lock-file not up-to-date with the workspace), especially in CI.

Instead, responsibilities are split:

  • Pixi manages the environment and pinned dependencies
  • pip install -e . installs Matplotlib itself

This keeps pixi.lock stable while still supporting editable development installs.

Single environment

Although Pixi supports separate environments such as test and doc, this PR currently uses a single environment containing the relevant dependencies.

The main reason is developer convenience:

  • avoids repeated editable installs in separate environments
  • keeps the workflow simple
  • still allows the lockfile to capture the relevant dependency sets

This can be revisited later if stricter separation is desirable.

CI

This PR adds a minimal CI job that checks that the Pixi environment can be recreated from the committed lockfile.

At the moment the job is intentionally lightweight: it verifies that the lockfile works and that Matplotlib can be built/imported in that environment. It can be extended later to run tests and/or documentation builds.


Notes / future work

  • extend Pixi CI coverage to run more tests and build docs
  • decide whether additional environment separation is worthwhile
  • refine developer-facing documentation around local pixi.toml
    overrides

AI disclosure

I went back and forth w/ chatGPT quite a bot on how to organize this and the various tradeoffs. It also helped with writing this summary, which is of course long-winded, but hopefully complete.

@github-actions github-actions Bot added the Documentation: devdocs files in doc/devel label Mar 25, 2026
@jklymak jklymak changed the title Dev add pixi toml [DEV]: add tools/pixi.toml and doc how to have a development environment for pixi Mar 25, 2026
@jklymak
jklymak marked this pull request as ready for review March 25, 2026 23:41
@jklymak

jklymak commented Mar 26, 2026

Copy link
Copy Markdown
Member Author

.. I'm surprised the tests are running given I labelled this [ci doc]

Comment thread tools/pixi.toml Outdated
depends-on = ["clean-build", "dev-install"]
description = "Clean and rebuild editable install"

[tasks.install-test]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should make features for tests + docs.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we could do it that way, but I'm not sure of the pros and cons. It did not seem there was a way to make conditional install dependencies so I wasn't sure the advantage of having separate feature envs for a developer versus just having one env.

Comment thread tools/pixi.toml Outdated
Comment thread tools/pixi.toml Outdated
description = "No-op on non-mac platforms"

[target.osx-arm64.tasks.prepare-freetype]
cmd = "cp subprojects/packagefiles/freetype-2.6.1-meson/src/gzip/zconf.h subprojects/freetype-2.6.1/src/gzip"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also don't understand this, is something going wrong with unpacking the subproject? This seems like something we should be fixing in the meson files rather than here?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's a general problem, but it's definitely a problem on macOS with the version of zconf

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK< I think this was not necessary, as it now works fine without it.

@tacaswell

Copy link
Copy Markdown
Member

It is my understanding that if you have metadata about the build you are (or will soon be able) to point to a repo as a conda dependency and pixi-build will build the conda-package on the fly (the same way it does with pypi-deps). Given that we are going to want pixi.toml in the lop level eventually.

That said, I agree we do not want to check in the pixi.lock file.

@tacaswell

Copy link
Copy Markdown
Member

I mean add a section like

[pypi-dependencies]
matplotlib = { path = "..", editable = true }

@jklymak

jklymak commented Mar 26, 2026

Copy link
Copy Markdown
Member Author

Ah I see. No that didn't work as I don't think it's compatible with build isolation.

@jklymak

jklymak commented Mar 26, 2026

Copy link
Copy Markdown
Member Author

OK< my comment above was incorrect. The new pixi.toml has removed the font copying things, and uses

[pypi-dependencies]
matplotlib = { path = ".", editable = true }

which indeed seems to be fine.

Still not sure if we should run with features or just a separate install steps?

@jklymak
jklymak force-pushed the dev-add-pixi-toml branch from 7d73c47 to 27d86a1 Compare March 28, 2026 04:49
@jklymak
jklymak marked this pull request as draft March 28, 2026 04:58
@jklymak
jklymak force-pushed the dev-add-pixi-toml branch 2 times, most recently from aedf224 to 055029d Compare March 28, 2026 16:22
@jklymak jklymak changed the title [DEV]: add tools/pixi.toml and doc how to have a development environment for pixi [DEV]: add pixi.lock and add pixi to pyproject.toml Mar 28, 2026
@jklymak
jklymak force-pushed the dev-add-pixi-toml branch from 055029d to 85257fe Compare March 28, 2026 16:32
@jklymak
jklymak marked this pull request as ready for review March 28, 2026 16:50
@jklymak
jklymak requested a review from tacaswell March 28, 2026 19:51
@jklymak
jklymak force-pushed the dev-add-pixi-toml branch 3 times, most recently from a0a2ff5 to c0467de Compare March 30, 2026 17:51
@jklymak jklymak added the CI: testing CI configuration and testing label Mar 30, 2026
@jklymak
jklymak force-pushed the dev-add-pixi-toml branch from c0467de to f3e0de0 Compare March 30, 2026 18:09
Comment thread .github/workflows/pixi_tests.yml Outdated
Comment thread .github/workflows/pixi_tests.yml
Comment thread pyproject.toml
Comment thread .github/workflows/pixi_tests.yml Outdated
Comment thread .github/workflows/pixi_tests.yml
Comment thread doc/devel/development_setup.rst Outdated
Comment thread doc/devel/development_setup.rst Outdated
@jklymak
jklymak force-pushed the dev-add-pixi-toml branch from a40ae3f to 3ba6a65 Compare April 1, 2026 14:27
@jklymak

jklymak commented Apr 21, 2026

Copy link
Copy Markdown
Member Author

Anyone else have comments on this? @tacaswell or @QuLogic ? I needed to install something and realized I can't actually use this until its merged ;-)

@tacaswell

Copy link
Copy Markdown
Member
  • I remain deeply skeptical of mixing pixi config in to pyproject.toml stuff. I think it crosses two packaging ecosystems in bad ways (for example, the current lock file has 986 things installed from pypi which is very much less than ideal). I'll push a commit to fix that. One selling point of pixi is it is fast, but the other major selling point is getting away from the wheel ecosystem all together.
  • I still prefer to let pixi do the editable install, but I have not thought through possible issues in CI. The instruction of pixi run test and pixi run docs being so simple is something it feels hard to give up. I think we can give up the CI check that the lock is consistent if saves us install steps (particularly because mpl-sphinx-theme depends on us so we always install a version and then install over it).
  • I think nice things are coming with https://pixi.prefix.dev/dev/build/backends/pixi-build-python/#ignore-pypi-mapping which will hopefully let us drop the duplication and make it so that you can add the main branch of mpl as a conda dep to another pixi config.

@tacaswell

Copy link
Copy Markdown
Member

The lock check failing may be a v6 vs v7 thing on the lock file (which is coupled to pixi 0.68).

@jklymak

jklymak commented May 22, 2026

Copy link
Copy Markdown
Member Author

I'm more than happy for someone else to take this over or redo it. I maintain a willful ignorance of the packaging ecosystem. Conversely, it'd be nice to have something that works.

@timhoffm

Copy link
Copy Markdown
Member

the current lock file has 986 things installed from pypi

That’s strange. I thought conda-forge is given precedence, and the environment.yml in our project root shows that most of the dev environment can be built on conda.

Anyway, we might want to use a pixi.toml for configuration and better separation. Inspiration could be taken from scipy, who have a fairly comprehensive setup.

Comment thread .github/workflows/pixi_tests.yml Fixed
Comment thread .github/workflows/pixi_tests.yml Fixed
@tacaswell
tacaswell force-pushed the dev-add-pixi-toml branch from e4bbbd9 to 2538a83 Compare July 31, 2026 03:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI: Run cibuildwheel Run wheel building tests on a PR CI: testing CI configuration and testing Documentation: devdocs files in doc/devel

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants