WRT the original #22852 (comment):

• r'$\sqrt{1+\sqrt{1+\sqrt{1+\sqrt{1+\sqrt{1+\sqrt{1+\sqrt{1+x}}}}}}}$' (There are neither fractions nor superscripts or superscripts. And yet, this image failed the image comparison test.)

Also unsure about this one; it appears that one square root sign is sized a bit different?

• r"$f'\quad f'''(x)\quad ''/\mathrm{yr}$" (Looks like apostrophes affected by the superscript positioning.)

Seems to be the case, but apostrophes are handled by the subsuper method, so this seems okay.

• test_operator_space (In this check_figures_equal test, the 's' of 'cos' is displaced slightly compared to the reference image, even though 'co' is placed identically. This is strange, because operator kerning should have remained unchanged.)

This is no longer failing; I think this may be one of the rounding issues for the initial character found (and fixed) in #30059.

TODO: Since the height of fractions is a little bigger, I think I may need to tweak the AutoHeightChar tests to ensure that all sizes are correctly included.

I've gone through the sub/super script changes and they seem fine as well, other than a couple known shortcuts.

I'm only uncertain about the constants; they may have only been chosen to minimize image changes instead of matching TeX. For CM, it was also calculated with our current hinting settings and not the defaults that have been switched to in the text-overhaul branch.

While looking at the TeX algorithms, I found out about tftopl which can show some metrics from TFM files. For Computer Modern, I looked at the output for cmsy10:

(FAMILY CMSY)
(FACE O 352)
(CODINGSCHEME TEX MATH SYMBOLS)
(DESIGNSIZE R 10.0)
(COMMENT DESIGNSIZE IS IN POINTS)
(COMMENT OTHER SIZES ARE MULTIPLES OF DESIGNSIZE)
(CHECKSUM O 4110426232)
(FONTDIMEN
   (SLANT R 0.25)
   (SPACE R 0.0)
   (STRETCH R 0.0)
   (SHRINK R 0.0)
   (XHEIGHT R 0.430555)
   (QUAD R 1.000003)
   (EXTRASPACE R 0.0)
   (NUM1 R 0.676508)
   (NUM2 R 0.393732)
   (NUM3 R 0.443731)
   (DENOM1 R 0.685951)
   (DENOM2 R 0.344841)
   (SUP1 R 0.412892)
   (SUP2 R 0.362892)
   (SUP3 R 0.288889)
   (SUB1 R 0.15)
   (SUB2 R 0.247217)
   (SUPDROP R 0.386108)
   (SUBDROP R 0.05)
   (DELIM1 R 2.389999)
   (DELIM2 R 1.01)
   (AXISHEIGHT R 0.25)
   )

Comparing the values, we have:

Remember that we multiply everything by x-height, so here I've divided that metric out in the last column. Also, we use subdrop in for both sub/superscripts, so it appears as the current values for both.

So, the numbers we have for sub1, sub2, are fairly close. subdrop / subdrop are quite different from what we have, and sup1 is also a bit bigger, but if we change the code to apply each separately, then it's mostly that superscripts drop by another pixel or so. If we make this change, about 100 test images are affected.

For the fraction metrics (num1, num2, num3, denom1, denom2) there is a quite a bit of a difference in some of them. If we apply those, it affects about 70 test images, and mostly fractions "close up" a bit, but this is closer to how TeX does it.

Before we had this (with usetex=True on the right):

and with these updated constants we have:

It's still a tiny bit off, but I think that's because we aren't the ones rendering the usetex text; with #30039 it's even closer.

I haven't found any corresponding TFM files with the same metrics in them for the other fonts we have. I believe modern LaTeX can synthesize these directly from the font, and I do see some embedded MATH tables in there, but I haven't worked out the conversions for those yet.

I've pushed the changes to the Computer Modern constants as two commits for ease of review.

This extra +/- rule is odd compared to TeX, but it is explained in the original PR

Vlists containing Hrules don't render vertical spaces correctly. The code for reproduction given in #23763 still works!

• test_operator_space (In this check_figures_equal test, the 's' of 'cos' is displaced slightly compared to the reference image, even though 'co' is placed identically. This is strange, because operator kerning should have remained unchanged.)

This is no longer failing; I think this may be one of the rounding issues for the initial character found (and fixed) in

I reported this as a bug back then (#23474). Changing the script rendering logic is what exposed it. After merging a fix (#23482), the test passed even with the updated logic.

I'm only uncertain about the constants; they may have only been chosen to minimize image changes instead of matching TeX.

🎯💯

subdrop / subdrop are quite different from what we have, and sup1 is also a bit bigger, but if we change the code to apply each separately, then it's mostly that superscripts drop by another pixel or so. If we make this change, about 100 test images are affected.

For the fraction metrics (num1, num2, num3, denom1, denom2) there is a quite a bit of a difference in some of them.

I remember being puzzled that using the same constants (from either Computer Modern or Latin Modern—I can't remember which one I had checked) didn't yield neatly aligned denominators. The multiplication by the x-height appears to be the key, which didn't strike me then.

Vlists containing Hrules don't render vertical spaces correctly. The code for reproduction given in #23763 still works!

It appears to me that this specific issue has been fixed by #30059, can you confirm? (I haven't looked carefully yet at how this interacts with the fraction rendering.)

re: the +/-rule issue. Empirically it looks like the +/-rule can be removed by changing how Rules are being rendered; i.e. the following patch appears to work:

diff --git i/lib/matplotlib/_mathtext.py w/lib/matplotlib/_mathtext.py
index f45e1044bc..bfe3f3ec29 100644
--- i/lib/matplotlib/_mathtext.py
+++ w/lib/matplotlib/_mathtext.py
@@ -1424,7 +1424,7 @@ class Rule(Box):

     def render(self, output: Output,  # type: ignore[override]
                x: float, y: float, w: float, h: float) -> None:
-        self.fontset.render_rect_filled(output, x, y, x + w, y + h)
+        self.fontset.render_rect_filled(output, x, y - h, x + w, y)


 class Hrule(Rule):
@@ -2687,9 +2687,9 @@ class Parser:
             den_clr = max(axis_height - delta + den_shift_down - cden.height, min_clr)
             # Possible bug in fraction rendering. See GitHub PR 22852 comments.
             vlist = Vlist([cnum,                     # numerator
-                           Vbox(0, num_clr - rule),  # space
+                           Vbox(0, num_clr),  # space
                            Hrule(state, rule),       # rule
-                           Vbox(0, den_clr + rule),  # space
+                           Vbox(0, den_clr),  # space
                            cden                      # denominator
                            ])
             vlist.shift_amount = cden.height + den_clr + delta - axis_height

It would seem credible to me that we got the definition of boxes mixed up at some point (especially due to the mix between upwards y's and downwards y's, which always confuses me), though I haven't actually tracked that down.

Edit: I convinced myself that the patch is correct. At the call site

Happy to see you're still around @tfpf and thanks for answering any questions we have.

re: the +/-rule issue. Empirically it looks like the +/-rule can be removed by changing how Rules are being rendered; i.e. the following patch appears to work:

If I make this change, it affects 65 test images. Most of them do not contain fractions, but \overline and square roots. In the case of heavily nested roots (image 53), it actually looks better aligned. There are several tips of the ⎷ that are about a pixel above the corresponding bar. I'd need to check the overlines. Since this almost always shifts up rects by one pixel (after snapping, presumably), I guess this is the original origin of the incorrect fraction bar?

Happy to see you're still around @tfpf and thanks for answering any questions we have.

Yes, I didn't mention this, but: @tfpf: I'm sorry your (nice!) work didn't go in last time, turns out we needed an even bigger overhaul of many parts to fix all the issues; thanks for coming back to help. I'll also take that opportunity to thank @QuLogic for keeping all the moving parts together and taking care of the more-or-less complete patches I keep throwing around 😅

re: the +/-rule issue. Empirically it looks like the +/-rule can be removed by changing how Rules are being rendered; i.e. the following patch appears to work:

If I make this change, it affects 65 test images. Most of them do not contain fractions, but \overline and square roots. In the case of heavily nested roots (image 53), it actually looks better aligned. There are several tips of the ⎷ that are about a pixel above the corresponding bar. I'd need to check the overlines. Since this almost always shifts up rects by one pixel (after snapping, presumably), I guess this is the original origin of the incorrect fraction bar?

I suspect so? Not that I looked.

By the way, https://www.tug.org/TUGboat/tb27-1/tb86jackowski.pdf has some nice figures explaining fraction layout, and https://www.tug.org/~vieth/papers/bachotex2008/math-font-paper.pdf some more discussion on inferring font parameters (though neither the x-height nor the axis-height...).

According to The LaTeX Font Catalogue, I don't think DejaVu Sans provides math support in LaTeX, so I'm a bit uncertain how to compare our math constants with it. Opening up the MATH table with fontTools and picking out the values from https://ntg.nl/maps/38/03.pdf it seems that pretty much all of them are 0 and thus useless.

Instead, I've managed to find values in FontForge's TeX tables in a slightly-hidden settings page, then dividing by the xheight in that same setting page (1120), we get:

supdrop/subdrop are quite different, but that's because they used to be one constant. num2/num3/denom2 all shrunk quite a bit, but the same thing happened with Computer Modern above.

Before, we had:

and now we get:

Note that the right side of the figure is usetex=True with \usepackage{arev} for the preamble. These are both from the same lineage of Bitstream Vera Sans, but I'm not sure if they're comparable if DejaVu Sans never had math support, so take those with a grain of salt, especially since the x doesn't really look the same style.

DejaVu Serif has the same metrics, except the xheight is 1063 instead, so I won't post the table over again. Instead, just before and after images:

There is a dejavu math font: https://www.gust.org.pl/projects/e-foundry/tex-gyre-dejavu-math / https://ctan.org/pkg/tex-gyre-math-dejavu?lang=en but it's only a serif font :/ not sure how well the constants would compare?

Well, we do have DejaVu Serif as well with almost the same metrics; only the xheight is different at 1063 instead of 1120 (updated above with those images as well.) But it looks like the TeX Gyre fonts are for unicode-math and would require the other LaTeX engines to plug in to our usetex setup, too.

Well, we do have DejaVu Serif as well with almost the same metrics; only the xheight is different at 1063 instead of 1120 (updated above with those images as well.) But it looks like the TeX Gyre fonts are for unicode-math and would require the other LaTeX engines to plug in to our usetex setup, too.

#31064 enables to use the TeX Gyre fonts from within usetex. Currently, it does not dynamically set the "constants" but uses the defaults, but one could in principle read the otf MATH table.

For STIX, I again took the TeX tables and divided by xheight (450):

It's largely different in the same places as before, I think.
Before:

After:

And for STIX Sans, it's the same font, so it's the same final metrics, but the metrics were slightly different before:

Before:

After: