Intro

This is an implementation following the pattern of https://github.com/scipy/scipy-release of building release builds outside of the main repo.

Currently it lives at https://github.com/ksunden/matplotlib-release

I am in particular looking for technical review from @QuLogic, and policy review from the steering council (@tacaswell @efiring @dopplershift @timhoffm).

Catalyzing incident (failure to upload v3.10.8)

The build failed, not because of anything actually wrong with matplotlib, but because
one of the dependencies (Pillow) lacked a wheel and the headers to build from source were missing.
The failure was in the test stage, not the wheel building stage.

Rather than fix in repo and tag as 3.10.9, skipping yet another version in this series,
I decided to accelerate the timeline for building in an external repo.

Technical details

Usage

Branches mirror the main repo with main and v3.XX.x branches

Target is set by editing the SOURCE_REF_TO_BUILD: main at the top of the workflow file.
For releases, this should point to the tag

Release builds are done via dispatch_workflow with the pypi option chosen. (Most often web UI, also available vi CLI etc)

Changes from existing workflows

• License file munging

  • In doing a review, noticed that our auxilliary licenses are not being included and have not been since 3.9.0
  • Meson-python version that we use does not handle multiple license files
    • May look into newer versions, we were pinned for unrelated reasons, worth investigating
  • Auxilliary licenses combined into one file, which lives in the matplotlib-release repo
  • That file is cat'd onto the main LICENSE file, which is then included in wheel .dist-info
  • Happens as a separate job that uploads the artifact for auditability
  • Interstitial sdist uploaded with a name that is not caught by publish artifact downloader

• Trigger conditions:

  • since this repo is separate, I do just trigger on push/pull request, but final uploads must be done with workflow_dispatch (which is new)

• CIBW_AFTER_BUILD does not actually exist, so those steps were not run previously

  • Moved some into CIBW_TEST_COMMAND, but in the process of getting things to work mostly elided
    • Twine check removed because python 3.13t proved hard due to cryptography trying to install there (CFFI under the hood the problem)
    • License check removed due to combining into one file

• CIBW_TEST_ENVIRONMENT added PIP_PREFER_BINARY=1

  • This is the actual fix for why 3.10.8 failed
  • Used the general solution to avoid similar problems in the future

• Upload tasks

  • Largely taken from scipy's version, with only minor edits (mostly s/scipy/matplotlib/g)
  • Include testpypi upload option
  • Include check that upstream reference is a tag
    • I did not change this other than the repo it points to, seems potentially underspecified but didn't look too hard at it
  • Attestations now baked into the pypa/gh-action-pypi-publish, so removed that extra step
    • id-token only permission necessary it seems
  • I have not fully tested these, as fully doing so requires uploading to PyPI, sooo

Followup steps

Near term/get 3.10.8 out

• move matplotlib-release into matplotlib github org
  • ensure permissions are restricted to necessary parties only (release managers, steering council, perhaps it)
• set up PyPI trusted publishers to look to matplotlib-release
• Run workflow with appropriate settings to publish on the v3.10.x branch
• update devdocs for new release workflow (Would like to do it once first to make sure, but don't want to put it off)

Future work

• Work towards removing CIBW jobs from main repo
  • Make the matplotlib-release version triggerable from the main repo
    • Currently we run the job on merges and on labeled PRs, can probably look towards doing the same, though opportunity to think about it
  • Upload to nightlies from matplotlib-release
    • scipy does it on scheduled runs of the cibw job, we could copy that
    • we currently do as a separate job that runs on a schedule but picks up the latest successful run
  • Better notification of failures
    • e.g. open issues like "upcoming nightly failures" does
    • possibly needs to check backport branches too?
    • impact is lessened by the ability to build from the same tag with different CIBW settings, so less important than it would otherwise be
• Consider features that may be useful
  • Some of this can be shared with scipy/etc who are also doing similar things
  • e.g. we could make the target a variable that can be passed into the job
    • Useful for triggering from the main repo
    • Could make release workflow easier (i.e. don't need to edit the workflow file)
      • On the other hand, a little friction is often a good thing, so, perhaps not a priority
  • Consider if other settings should be moved to the release repo
    • e.g. build flags, pyproject.toml, etc.
    • Current leaning: only when proven necessary, so hopefully never, but escape hatches may exist
• Investigate meson-python update and enabling multiple license files
  • Would allow removal of step/artifact
  • Original reason for pinning was to do with symlinks, 0.18 supposedly fixes our problem, and has at least some support for multiple license files
  • things to look for: dynamic license-files (we specify in meson.build, but don't think they get used)
  • in particular the QHull license is a little funky since it is not in the repo by default