Add custom math fallback by andrzejnovak · Pull Request #16509 · matplotlib/matplotlib

andrzejnovak · 2020-02-14T13:34:34Z

PR Summary

Follow-up on #11644

I've switched the logic so it's more transparent when mathtext.fallback is by default 'cm' . The original mathtext.fallback_to_cm defaults to None, but either True or False will raise a warning and override the mathtext.fallback setting.

PR Checklist

Has Pytest style unit tests
Code is Flake 8 compliant
New features are documented, with examples if plot related
Documentation is sphinx and numpydoc compliant
Added an entry to doc/users/next_whats_new/ if major new feature (follow instructions in README.rst there)
Documented in doc/api/api_changes.rst if API changed in a backward-incompatible way

andrzejnovak · 2020-02-15T16:56:41Z

@anntzer This should about do it. Seems mpl ships only with DejaVu Sans/STIX so I couldn't test a real figure against an incomplete font that would have to get substituted.

Unsure what's failing the docs though

anntzer · 2020-02-15T17:12:21Z

@geib7592 Can you please comment on this PR? Thanks!

anntzer · 2020-02-15T22:16:10Z

Looking a bit below, it appears that DejaVuFonts (the actual default) already implements its own fallback solution, which goes to stix, not not cm. If you set mathtext.fallback, does that also override the fallback for dejavu?

I guess this suggests that the proper solution may instead to make mathtext.fontset accept a list of fonts, falling back from one to the next (so e.g. mathtext.fontset: dejavusans, stix or custom, cm or whatnot -- where custom is again defined by mathtext.{cal,rm,tt,it,bf,sf}).

andrzejnovak · 2020-02-15T22:51:44Z

Looking a bit below, it appears that DejaVuFonts (the actual default) already implements its own fallback solution, which goes to stix, not not cm. If you set mathtext.fallback, does that also override the fallback for dejavu?

I guess this suggests that the proper solution may instead to make mathtext.fontset accept a list of fonts, falling back from one to the next (so e.g. mathtext.fontset: dejavusans, stix or custom, cm or whatnot -- where custom is again defined by mathtext.{cal,rm,tt,it,bf,sf}).

IIUC fallback_to_cm only came to effect when using a math.text = custom and the behaviour should not be changed in this PR.

I agree it would be nicer to have a fallback list, but that seems like it would require major code changes, meanwhile the functionality in this PR (or the original PR indeed) is good enough.

Though it could be easily extended to also so StixSansFonts.

geib7592 · 2020-02-15T23:04:33Z

Hello, it's me from #8619. I'm excited to see someone working on this issue. It seems that since matplotlib ships with stix and dejavu sans, adding a fallback to dajavu sans (and stix sans) could be another (very similar) PR, and in my opinion, it would be worthwhile for flexibility in using custom sans-serif fonts.

Allowing a custom fallback font is likely more work, but would be useful for cases I pointed out in #8619. (@anntzer, your suggestion for having mathtext.fontset be a list seems to preclude the future possibility of having a custom fallback font)

I think @anntzer also suggested I add my two cents about why this PR is important. Since posting the feature request in 2017, I've thought a fair amount about how to typeset publication-quality scientific figures. Currently, I don't think any free (or even non-free) software does it perfectly, and it's an area where, with a few well chosen tweaks, matplotlib could really excel.

For example, major scientific publishing companies like Nature and Science explicitly prefer Helvetica for the text of their figures. For Greek letters and mathematical symbols, they use a similarly weighted serif font used in their main text. This PR as it stands really helps move in the right direction; the example in #11644 shows how to use Helvetica with a fallback to stix, which gets closer to Nature's guideline.

I'd love to see more development toward industry standard design principles in the future with similar tweaks to the mathtext and latex engines. If the matplotlib devs are interested, I'd be happy to discuss and get more involved.

Thanks for all your work!

anntzer · 2020-02-15T23:11:37Z

I'm fine with not doing the most general approach; I just want the various fallback options and their interaction to be properly documented (e.g. whether mathtext.fallback affects dejavu fallback as well).

your suggestion for having mathtext.fontset be a list seems to preclude the future possibility of having a custom fallback font

why?

geib7592 · 2020-02-15T23:18:42Z

Maybe I misunderstood, but right now custom mathfont requires mathtext.fontset=custom, so if you also wanted a custom fallback, you'd need a second custom setting, like allowing mathtext.{cal,rm,tt,it,bf,sf} to be lists too, where missing characters are sourced down the list.

anntzer · 2020-02-15T23:24:35Z

Ah yes, you want to fallback from one custom font to another one...
I guess making all of mathtext.cal/rm/... lists too would work indeed? (not that it has to be done in this PR)

anntzer · 2020-02-16T10:20:44Z

If you look a bit below, class DejaVuFonts has

class DejaVuFonts(UnicodeFonts):
    use_cmex = False

    def __init__(self, *args, **kwargs):
        # This must come first so the backend's owner is set correctly
        if isinstance(self, DejaVuSerifFonts):
            self.cm_fallback = StixFonts(*args, **kwargs)
        else:
            self.cm_fallback = StixSansFonts(*args, **kwargs)
        self.bakoma = BakomaFonts(*args, **kwargs)
        TruetypeFonts.__init__(self, *args, **kwargs)

(the point I was mentioning above)

I'm actually now a bit confused by why DejaVuFonts inherits from UnicodeFonts but calls TruetypeFonts.__init__ in its __init__, and am not even sure anymore that DejaVuFonts.cm_fallback is ever used?

andrzejnovak · 2020-02-16T10:28:38Z

I think we all agree that what we want is a fallback list for math as well. However, since that's major surgery, especially since the default fonts have a bunch of no-warnings/no-docs fallback which should likely be overhauled, we should open this as a new issue for a new PR.

This PR bring major improvement in just being able to have a sans-serif math fallback for custom fonts. Fulfilling most of what was requested in the 3 separate issues #8619, #11644 and #16492

Re some details, I added the option to explicitly ask for stixsans but actually have trouble finding characters that would be missing in a default font (say Arial) and would be different between stix and stixsans, which seem to differ for regular alphabet chars, but not so much for math symbols. (Basically I can't test is)

Otherwise, if we can agree on this functionality I'll update the tut/whats_new etc...

anntzer · 2020-02-18T10:00:34Z

I think you can add a test by picking some math strings that include a character that's not present in e.g. dejavusans but present in cm/stix/stixsans, and then saving a svg file with that string and rcParams["svg.fonttype"] = None, with fallback configured to cm/stix/stixsans/nofallback, which will saves the individual glyphs; you can then check that fallback correctly occurred (or not, if not falling back).

andrzejnovak · 2020-02-19T22:20:50Z

Ok, I did the SVG check and the fallbacks work fine, the issue is just that some of the math characters are the same for both stix/stixsans e.g.("#STIXGeneral-Regular-10938") In reality it looks like it won't matter if the math fallback is stix or stixsans

anntzer · 2020-02-19T23:49:55Z

It's fine if the test doesn't cover all cases, it would still be an improvement over the current situation.

anntzer · 2020-02-20T09:57:29Z


    def __init__(self, *args, **kwargs):
        self._stix_fallback = StixFonts(*args, **kwargs)
+        self.name = "Computer Modern"


Well, the hope was that you would not have to hardcode this but could instead infer it from the FT2Font object, but I guess that somehow didn't work?

Maybe I'm doing it wrong, but the only reasonable way I found was self.cm_fallback.default_font_prop.get_family() but that just returns ['sans-serif'] which doesn't help here.

Or fontmap which returns sth like {'cal': '/home/anovak/matplotlib/lib/matplotlib/mpl-data/fonts/ttf/cmsy10.ttf', ... 'cmsy10': '/home/anovak/matplotlib/lib/matplotlib/mpl- data/fonts/ttf/cmex10.ttf', 'cmex10': '/home/anovak/matplotlib/lib/matplotlib/mpl-data/fonts/ttf/cmex10.ttf'} which would need some extra matching overhead to spit out a useful message.

I was basically considering something like

if (fontname in ('it', 'regular') and isinstance(self.cm_fallback, StixFonts)): fontname = 'rm' g = self.cm_fallback._get_glyph( fontname, font_class, sym, fontsize) _log.warning("Substituting with a symbol from {}".format(g[0].family_name)) # or .postscript_name return g

As it turns out, while this would work for stix fallback and most other fonts (I guess), this would fail specifically for computer modern fallback because cmmi10.ttf has poor metadata (it lists its family as "cmmi10"...), but I guess you can always special-case cm; at least this should work for "all" other fallbacks...

Ah nice, that's quite fun, because you can report per symbol. Checkout:

import matplotlib as mpl test_str = r'a$b\AA\Finv\breve\gimel\succnapprox$' mpl.rcParams['mathtext.fontset'] = 'custom' mpl.rcParams['mathtext.rm'] = 'Courier New' mpl.rcParams['mathtext.it'] = 'Courier New:italic' mpl.rcParams['mathtext.bf'] = 'Courier New:bold' fig, ax = plt.subplots(figsize=(8, 8)) mpl.rcParams['mathtext.fallback'] = 'cm' txt = fig.text(.5, .5, test_str, fontsize=40, ha='center')

Output:

Substituting with a \Finv from STIXGeneral Substituting with a \gimel from STIXGeneral Substituting with a \combiningbreve from Computer Modern Substituting with a \succnapprox from STIXGeneral Substituting with a \Finv from STIXGeneral Substituting with a \gimel from STIXGeneral Substituting with a \combiningbreve from Computer Modern Substituting with a \succnapprox from STIXGeneral

Can't figure out why is it doing two loops though. It will do yet another loop if a giure is saved, which is probably too many warnings to show.

@anntzer yeah, warnings fail tests

You can use recwarn or pytest.warns (https://docs.pytest.org/en/5.3.5/warnings.html#assertwarnings) to catch them.

My point was that you actually want to emit a test here, but in the tests you can check that the warning is correctly thrown using pytest.warns.

better just go back to logging, than changing all those tests

that's fine with me too, I don't think the extra logging is too bad(?)

andrzejnovak · 2020-02-24T11:11:26Z

@anntzer that should be it

anntzer · 2020-02-24T11:18:43Z

A test (per #16509 (comment)) would be nice.

anntzer · 2020-02-25T17:04:58Z

lgtm, can you please squash?

andrzejnovak · 2020-02-25T20:06:21Z

phew I'm just gonna drop this here when I inevitably need it again https://blog.appsignal.com/2016/09/27/git-rebasing-strategies.html

anntzer · 2020-02-25T20:50:55Z

looks like you got it to work :)

andrzejnovak · 2020-02-25T22:02:11Z

what's failing the codecov?

anntzer · 2020-02-25T22:19:32Z

It's not failing?

anntzer · 2020-02-25T22:25:46Z

flake8 is unhappy though

./lib/matplotlib/rcsetup.py:443:25: E128 continuation line under-indented for visual indent
2390./lib/matplotlib/rcsetup.py:444:25: E128 continuation line under-indented for visual indent
23912     E128 continuation line under-indented for visual indent

andrzejnovak · 2020-02-25T23:08:59Z

It's not failing?

somehow it's transient?

anntzer · 2020-02-25T23:12:47Z

Now it's back.
It's pretty flaky anyways (they likely have some race conditions against when the CI run finishes), so we don't overly worry about it.

anntzer

conditional on CI

andrzejnovak · 2020-03-02T13:23:29Z

@anntzer what about the codecov

anntzer · 2020-03-02T13:44:19Z

it's fine.

…tix', 'stixsans'

anntzer mentioned this pull request Feb 14, 2020

Feature request: allow mathtext fallback font other than Computer Modern for custom mathtext setup #8619

Closed

anntzer added the topic: text/mathtext label Feb 14, 2020

andrzejnovak requested a review from anntzer February 15, 2020 16:55

anntzer reviewed Feb 15, 2020

View reviewed changes

Comment thread tutorials/text/mathtext.py Resolved

anntzer reviewed Feb 15, 2020

View reviewed changes

Comment thread lib/matplotlib/mathtext.py Resolved

andrzejnovak commented Feb 16, 2020

View reviewed changes

Comment thread matplotlibrc.template Resolved

anntzer mentioned this pull request Feb 16, 2020

Improve test failure messages on warnings. #16528

Merged

6 tasks

anntzer reviewed Feb 18, 2020

View reviewed changes

Comment thread lib/matplotlib/rcsetup.py Resolved

anntzer reviewed Feb 19, 2020

View reviewed changes

Comment thread lib/matplotlib/mathtext.py Resolved

anntzer reviewed Feb 19, 2020

View reviewed changes

Comment thread lib/matplotlib/mathtext.py Resolved

anntzer reviewed Feb 20, 2020

View reviewed changes

Comment thread lib/matplotlib/mathtext.py Resolved

anntzer reviewed Feb 20, 2020

View reviewed changes

Comment thread lib/matplotlib/mathtext.py Resolved

anntzer reviewed Feb 20, 2020

View reviewed changes

Comment thread lib/matplotlib/mathtext.py Resolved

anntzer reviewed Feb 20, 2020

View reviewed changes

Comment thread lib/matplotlib/mathtext.py Resolved

anntzer reviewed Feb 25, 2020

View reviewed changes

Comment thread lib/matplotlib/tests/test_mathtext.py Resolved

anntzer reviewed Feb 25, 2020

View reviewed changes

Comment thread lib/matplotlib/tests/test_mathtext.py Resolved

anntzer reviewed Feb 25, 2020

View reviewed changes

Comment thread lib/matplotlib/tests/test_mathtext.py Resolved

andrzejnovak force-pushed the fallbackto branch from 4ccc0d4 to ee790a3 Compare February 25, 2020 19:27

andrzejnovak force-pushed the fallbackto branch from 644957d to a5b8b42 Compare February 25, 2020 20:28

andrzejnovak force-pushed the fallbackto branch from a5b8b42 to fe162b7 Compare February 25, 2020 22:40

andrzejnovak force-pushed the fallbackto branch 3 times, most recently from 8c6734f to 5424ff5 Compare February 26, 2020 09:40

anntzer approved these changes Feb 26, 2020

View reviewed changes

timhoffm reviewed Mar 2, 2020

View reviewed changes

Comment thread lib/matplotlib/mathtext.py Resolved

timhoffm reviewed Mar 3, 2020

View reviewed changes

Comment thread lib/matplotlib/mathtext.py Resolved

feat: mathtext.fallback replaces fallback_to_cm with options 'cm', 's…

9f363ed

…tix', 'stixsans'

andrzejnovak force-pushed the fallbackto branch from 3253a3e to 9f363ed Compare March 3, 2020 08:37

timhoffm approved these changes Mar 3, 2020

View reviewed changes

timhoffm merged commit 09436b4 into matplotlib:master Mar 3, 2020

QuLogic changed the title ~~Add custom math fallback [WIP]~~ Add custom math fallback Mar 3, 2020

QuLogic mentioned this pull request Mar 10, 2020

Add feature to fallback to stix font in mathtext #11644

Closed

6 tasks

This was referenced Aug 23, 2020

ATLAS style susceptible to matplotlib math font bugs scikit-hep/mplhep#178

Closed

Font/Mathfont and fallbacks buggy #14829

Closed

Conversation

andrzejnovak commented Feb 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

PR Checklist

Uh oh!

andrzejnovak commented Feb 15, 2020

Uh oh!

Uh oh!

anntzer commented Feb 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anntzer commented Feb 15, 2020

Uh oh!

Uh oh!

andrzejnovak commented Feb 15, 2020

Uh oh!

geib7592 commented Feb 15, 2020

Uh oh!

anntzer commented Feb 15, 2020

Uh oh!

geib7592 commented Feb 15, 2020

Uh oh!

anntzer commented Feb 15, 2020

Uh oh!

anntzer commented Feb 16, 2020

Uh oh!

andrzejnovak commented Feb 16, 2020

Uh oh!

Uh oh!

Uh oh!

anntzer commented Feb 18, 2020

Uh oh!

andrzejnovak commented Feb 19, 2020

Uh oh!

Uh oh!

Uh oh!

anntzer commented Feb 19, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrzejnovak Feb 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

anntzer Feb 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andrzejnovak commented Feb 24, 2020

Uh oh!

anntzer commented Feb 24, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

anntzer commented Feb 25, 2020

Uh oh!

andrzejnovak commented Feb 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anntzer commented Feb 25, 2020

andrzejnovak commented Feb 14, 2020 •

edited

Loading

anntzer commented Feb 15, 2020 •

edited

Loading

andrzejnovak Feb 20, 2020 •

edited

Loading

anntzer Feb 20, 2020 •

edited

Loading

andrzejnovak commented Feb 25, 2020 •

edited

Loading