Would you be open to using a tracking issue or project board for the font work (sample categories: draft, waiting on other PRs, ready for review, please review first)?

I think breaking everything up into small PRs is fantastic but I'm finding it a bit overwhelming to keep track of the order for reviewing things (and figure I'm not alone here) and how these PRs relate to each other.

Sure, I'll do that. I'm sorry for the mess, but this is basically me trying to remember all the patches or even just ideas I have accumulated over the years that went nowhere because they would thrash all the baseline images, and that I now try to present given that I see an opportunity with the FreeType upgrade. Also, some of them are not fully complete patches but things that I believe can be made complete with relatively little effort (for some value of "little effort"...) simply because I do not have the time now to finalize them but want to put them up for discussion as the underlying idea should be clear.

Edit: see the wiki; attn @QuLogic as well too.

I'm sorry for the mess, but

No apologies needed, I commend you for taking advantage of the freetype upgrade. And thanks for the wiki, though some reason can't link cleanly ☹️

link fixed

Edit: fraction bar rendering has been added too.

A few warnings from compiling locally:

[2/4] Compiling C++ object src/ft2font.cpython-313-x86_64-linux-gnu.so.p/ft2font_wrapper.cpp.o
../../src/ft2font_wrapper.cpp: In lambda function:
../../src/ft2font_wrapper.cpp:1785:22: warning: unused variable ‘error’ [-Wunused-variable]
 1785 |             if (auto error = FT_Load_Glyph(face, idx, static_cast<FT_Int32>(flags))) {
      |                      ^~~~~
../../src/ft2font_wrapper.cpp:1788:22: warning: unused variable ‘error’ [-Wunused-variable]
 1788 |             if (auto error = FT_Render_Glyph(face->glyph, FT_RENDER_MODE_NORMAL)) {
      |                      ^~~~~
../../src/ft2font_wrapper.cpp: In lambda function:
../../src/ft2font_wrapper.cpp:1804:26: warning: unused variable ‘error’ [-Wunused-variable]
 1804 |                 if (auto error = FT_Glyph_To_Bitmap(&g, FT_RENDER_MODE_NORMAL, &origin, 1)) {
      |                          ^~~~~

This is semi-intentional: if these checks fail (error > 0) then the error should really be raised using throw_ft_error, not with a plain raise as I currently do, but throw_ft_error is not visible from ft2font_wrapper.cpp so some code needs to be moved around, e.g. make throw_ft_error an inline function defined in ft2font.h, or move the new implementations into real C++ FT2Font methods in ft2font.cpp and add wrapper lambdas in ft2font_wrapper.cpp, or (would be my preferred choice) switch error checking to the FT_CHECK macro used in mplcairo (see mplcairo's macros.h).
I left the warning as is just to keep this as a side discussion point, but it can trivially be suppressed by not capturing the error value too.

On a side note, I'm in favour of this in general as removing the intermediate buffer will make emoji/colour glyphs easier since they won't need to go through it.

On the call, we were discussing whether it made sense to target all these font-related (and requiring full image rebuild) PRs to a separate feature branch, which would contain the final test image updates. Do you have any thoughts on retargetting this PR (instead of folding it in to #29816)?

On the call, we were discussing whether it made sense to target all these font-related (and requiring full image rebuild) PRs to a separate feature branch, which would contain the final test image updates. Do you have any thoughts on retargetting this PR (instead of folding it in to #29816)?

Sure, I don't really mind either way? Should the branch live on this repo and be ultimately pruned out, or on yours?

Fixed #30059 (comment) by using FT_CHECK (thus also needs #30123).

What this PR provides:

• Fix for various subtle issues: support subpixel positioning for texts & fix jittering when moving text ([Bug]: 3D tick label position jitter when rotating the plot view #29551), improve horizontal and vertical spacing in mathtext (Update FreeType to 2.13.3 #29816 (comment)), fix alpha compositing of overlapping glyphs (Fix alpha compositing in ft2font's draw_bitmap. #30043) (*).
• Simplify implementation of color emoji support (Add support for (some) colour fonts #30725).

(* this is about alpha compositing of opaque glyphs with partial coverage of some pixels due to antialiasing, not about alpha compositing of semitransparent glyphs as mentioned in the comment above, which is a separate issue)

Certainly all of these issues can also be fixed while maintaining the intermediate buffer, but it would be more complicated (though perhaps not overly so).

I think the main reason for merging this into 3.11 is to have a single baseline image update, because this change will certainly break all baseline images (that include text) again.

Looking at this again I would guess we can still go in with this PR (if it's good enough for Chrome and Firefox it's good enough for us, as you say) and then later, if there's demand, move back to a (new, different) intermediate-buffer based implementation; it should be possible (if slightly tedious) to write it in a way that's fully backcompatible with this PR except for the case of overlapping semitransparent glyphs, which likely don't occur in the test suite, so that later change shouldn't trigger a mass baseline-image regen again.

My thoughts are that I think an intermediate buffer is the only way to fix the transparent text, but it is also currently implemented in the wrong place, causing the issues that @anntzer outlined. I think it makes sense to take this PR and then switch to an intermediate buffer only in the case of transparent text, but at a location much closer to the rendering step. It should be possible to then handle emojis reasonably there as well. However, as there are no tests that could fail this, there's no rush to get it done, and it could certainly wait until after the rc.

I just merged main into the text-overhaul branch to pick up the resampling bug fixes, so this will need a rebase; I'll look into that tomorrow.

There is something odd going on here with font/glyph caches; if I run lib/matplotlib/tests/test_text.py::test_complex_shaping then everything is fine. In fact, if I run all of test_text.py then everything is okay. But when running the whole test suite, the PDF and EPS both lose the Arabic: text in the second line.

(Note, I have updated the test-overhaul-figures branch to be based on this PR in anticipation of merging, so if you want to test it out, you can just fetch that, but you can also just run the test by itself first to generate the right results.)

I've narrowed it down to just running anything with mathtext before will cause the failure, e.g.,

$ pytest lib/matplotlib/tests/test_mathtext.py::test_mathtext_cmr10_minus_sign lib/matplotlib/tests/test_text.py::test_complex_shaping -v
=============================== test session starts ===============================
platform linux -- Python 3.13.11, pytest-8.3.3, pluggy-1.5.0 -- venv/bin/python
cachedir: .pytest_cache
rootdir: matplotlib
configfile: pyproject.toml
plugins: timeout-2.3.1, rerunfailures-14.0, cov-6.0.0, memray-1.7.0, anyio-4.10.0, xdist-3.8.0, xvfb-3.1.1
collected 5 items                                                                                                                                                                                                                                                                         

lib/matplotlib/tests/test_mathtext.py::test_mathtext_cmr10_minus_sign PASSED                                                                                                                                                                                                        [ 20%]
lib/matplotlib/tests/test_text.py::test_complex_shaping[png] PASSED                                                                                                                                                                                                                 [ 40%]
lib/matplotlib/tests/test_text.py::test_complex_shaping[pdf] FAILED                                                                                                                                                                                                                 [ 60%]
lib/matplotlib/tests/test_text.py::test_complex_shaping[svg] PASSED                                                                                                                                                                                                                 [ 80%]
lib/matplotlib/tests/test_text.py::test_complex_shaping[eps] FAILED                                                                                                                                                                                                                 [100%]

==================================== FAILURES =====================================
______________________________________________ test_complex_shaping[pdf] _____________________________

args = (), kwds = {'extension': 'pdf', 'request': <FixtureRequest for <Function test_complex_shaping[pdf]>>}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 28.259):
E           	result_images/test_text/complex_pdf.png
E           	result_images/test_text/complex-expected_pdf.png
E           	result_images/test_text/complex_pdf-failed-diff.png

/usr/lib64/python3.13/contextlib.py:85: ImageComparisonFailure
______________________________________________ test_complex_shaping[eps] _____________________________

args = (), kwds = {'extension': 'eps', 'request': <FixtureRequest for <Function test_complex_shaping[eps]>>}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 28.222):
E           	result_images/test_text/complex_eps.png
E           	result_images/test_text/complex-expected_eps.png
E           	result_images/test_text/complex_eps-failed-diff.png

/usr/lib64/python3.13/contextlib.py:85: ImageComparisonFailure
============================= short test summary info =============================
FAILED lib/matplotlib/tests/test_text.py::test_complex_shaping[pdf] - matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 28.259):
FAILED lib/matplotlib/tests/test_text.py::test_complex_shaping[eps] - matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 28.222):
=========================== 2 failed, 3 passed in 2.62s ===========================

I suspect that somewhere a _set_transform is missing.

When Agg's draw_mathtext calls _set_tranform with an offset, this sticks around and isn't reset anywhere.

So something like this does fix it for PDF:

diff --git a/lib/matplotlib/backends/backend_pdf.py b/lib/matplotlib/backends/backend_pdf.py
index c6b7cf0ac9..813d4a73ff 100644
--- a/lib/matplotlib/backends/backend_pdf.py
+++ b/lib/matplotlib/backends/backend_pdf.py
@@ -598,6 +598,8 @@ class Stream:
 
 def _get_pdf_charprocs(font_path, glyph_indices):
     font = get_font(font_path, hinting_factor=1)
+    hf = font._hinting_factor
+    font._set_transform([[round(0x10000 / hf), 0], [0, 0x10000]], [0, 0])
     conv = 1000 / font.units_per_EM  # Conversion to PS units (1/1000's).
     procs = {}
     for glyph_index in glyph_indices:
@@ -1121,6 +1123,8 @@ end"""
     def embedTTF(self, filename, subset_index, charmap):
         """Embed the TTF font from the named file into the document."""
         font = get_font(filename)
+        hf = font._hinting_factor
+        font._set_transform([[round(0x10000 / hf), 0], [0, 0x10000]], [0, 0])
         fonttype = mpl.rcParams['pdf.fonttype']
 
         def cvt(length, upe=font.units_per_EM, nearest=True):

and similar would probably work for EPS.

However, I wonder if we should be resetting this somewhere more central? Maybe in font_manager.get_font?

Ah, good catch, thanks for the debugging. I suspect that the most efficient way would be to reset the transform just once at the end of draw_mathtext(), and the most "principled" way would be indeed to do it in get_font() and document that the latter always returns a font with the default transform.

While looking at #22852, I noticed that some fraction bars were missing, both in that PR and here. For example, mathtext_cm_45 lost its fraction bar from this PR. I assume this has something to do with snapping to smaller than a pixel; should we impose some kind of minimum height when snapping is enabled here?

Note if we were to simply give up on snapping for the rectangles, then that would affect lib/matplotlib/tests/test_text.py::test_antialiasing, lib/matplotlib/tests/test_quiver.py::test_quiver_animate and 115 mathtext images. About 20 of the mathtext ones are cases where the fraction bar is currently missing and in the remainder the bar would be a bit blurrier (and in very few cases a bit longer.)

If we do something like this, then it affects <20 images and seems like the lightest fix.

Oops. Yes, that definitely seems like the reasonable fix; good catch.

OK, I pushed that change along with updated test images to the other branch. I'm a bit uncertain why test_pie_center_radius[png] is only failing on macOS though.

Ah, I just noticed that those tests have a tolerance; probably they changed little enough that I didn't replace them, but just enough that they were different on another platform. I force regenerated them and running CI again.

No luck, so I've just pushed a minor tolerance bump there and I can clean that up later when refreshing the images for good.

Thank you for shepherding this through!