BUG: .notdef glyph has to be present in fonts in fontlist

@story645

          > This also happens with the xkcd plotting style using the "Humor Sans" font.

So uh, this is why #26854 is failing (the plt.xkcd examples seem to work because they don't have a negative in the labels anywhere): throws "glyph missing" warning for title, omits hyphen for ticks:

Originally posted by @story645 in #17007 (comment)

per the discussion downthread, if a font does not have a designated .notdef then the missing glyph character (the .notdef) is not rendered.

Suggested Fix

For this one character, we should have a fallback such that we always render some glyph when a glyph is not found. This is particularly because of situations like the missing - at the front of a text, where it may be easy to miss that it's missing.

what would you suggest we do about this?

A glyph missing from a font is not really something we have control over.

Also, I just installed Humor Sans locally (from both http://antiyawn.com/uploads/humorsans.html and the debian archives), and it appears to work fine with the negative sign, including in the style reference that is failing in that PR.

This includes explicitly adding the text "-\N{MINUS SIGN}"

Further the default formatter uses "-" (chr(45) not chr(8722), as far as I can tell, so not sure where that is coming from? are we getting leaking rcParams from another example?

Perhaps the xkcd style file should explicitly set :rc:axes.unicode_minus to False, as that may be the source of the unicode character being used. (Though I'll note that locally I still did not get a warning when that rcparam was set to true)

This appears to be False by default, but seems related.

matplotlib/lib/matplotlib/ticker.py

Line 253 in b318dbf

return (s.replace('-', '\N{MINUS SIGN}')

Perhaps the xkcd style file should explicitly set :rc:axes.unicode_minus to False

Already do: https://github.com/matplotlib/matplotlib/pull/26854/files#diff-c8d8664596e18df275867e8057148fa2f964640f41e0e89269b32ebd41c75836R13

And this is the build error I get: https://output.circle-artifacts.com/output/job/a08b57c8-782a-4b83-8c0d-6aa57dab2b41/artifacts/0/doc/build/html/gallery/style_sheets/style_sheets_reference.html#sphx-glr-gallery-style-sheets-style-sheets-reference-py

Also, I just installed Humor Sans locally (from both http://antiyawn.com/uploads/humorsans.html and the debian archives), and it appears to work fine with the negative sign, including in the style reference that is failing in that PR.

Are you forcing it to only use Humor Sans? This is how I reproduced it - in jupyter and tkagg, w/ dev and 3.7.1

matplotlib.font_manager._load_fontmanager(try_read_cache=False)
with plt.xkcd():
    mpl.rcParams['font.family'] = 'Humor Sans'
    plt.plot([-2, 2])
    t = plt.title("\N{MINUS SIGN}" + f"{mpl.rcParams['axes.unicode_minus']}", fontfamily=['Humor Sans'])

I also don't get a build error when I build locally, only on CI.

A glyph missing from a font is not really something we have control over.

Yes but disappearing minus is concerning. I think what's happening here is we're not trying to replace the \unicode_minus w/ a regular minus and also not replacing it w/ a tofu character. Which, a maybe related aside is that tofu characters seem to not work in xkcd mode - plt.text(0, 0, "🙃") doesn't show tofu either. And again this is on 3.7 so it's independent from anything I've done in #26854

Basically, I think we only ever do the replacement in the `- -> '\N{Minus SIGN}'' direction.

It looks like we get the tofu from the first font in the fallback list:

matplotlib/src/ft2font.cpp

Lines 515 to 523 in cc46ee7

    
           if (!was_found) { 
        
               ft_glyph_warn((FT_ULong)codepoints[n], glyph_seen_fonts); 
        
               // render missing glyph tofu 
        
               // come back to top-most font 
        
               ft_object_with_glyph = this; 
        
               char_to_font[codepoints[n]] = ft_object_with_glyph; 
        
               glyph_to_font[glyph_index] = ft_object_with_glyph; 
        
               ft_object_with_glyph->load_glyph(glyph_index, flags, ft_object_with_glyph, false); 
        
           }

which suggests that the tofu in that font is empty?

Maybe the fix is to use a fallback font with Humor Sans?

Maybe the fix is to use a fallback font with Humor Sans?

I can do that in the gallery example, but is there something we can do that generalizes? which I think would mean shipping/storing our own tofu glyph?

I suspect that in xkcd() we can set the font to be something like ['humor sans', 'deja vu'] ?

I guess what I'm thinking is if someone who doesn't know that this is happening, if there's a way to make sure they see tofu.

I will try adding dejavu to my list of fonts in xkcd.mplstyle

matplotlib/lib/matplotlib/mpl-data/stylelib/xkcd.mplstyle

Line 7 in 8dc310e

font.family : xkcd, xkcd Script, Humor Sans, Comic Neue, Comic Sans MS

also yes no tofu:

Ok, so for my immediate need for #26854 I can find a different font, but I think the resolution for this issue is that we should have a fallback for the missing glyph glyph such that it's visible in the image that the glyph is missing.

xref has2k1/plotnine#451

Good first issue - notes for new contributors

This issue is suited to new contributors because it does not require understanding of the Matplotlib internals. To get started, please see our contributing guide.

We do not assign issues. Check the Development section in the sidebar for linked pull requests (PRs). If there are none, feel free to start working on it. If there is an open PR, please collaborate on the work by reviewing it rather than duplicating it in a competing PR.

If something is unclear, please reach out on any of our communication channels.

Good first issue cause the work is fairly independent of most of the matplotlib mechanics, but medium difficulty b/c it requires some experience to determine how to ensure that there's always a fallback glyph.

Off hand, I think either:

add a 1 character font to matplotlib that just defines a .notdef glyph
have a default fallback to a system font that packages a .notdef glyph

And then add a test that given a font where:

the .notdef character isn't defined, it fallsback to the default .notdef character.
.notdef is defined, it uses the font's notdef character

I am not convinced that this is something we want to do.

Do we have any evidence that this is a wider problem than that one font?

If I am reading our code and https://freetype.org/freetype2/docs/tutorial/step2.html, correctly it looks like the ".notdef" glyph is index 0 in freetype's API and in

matplotlib/src/ft2font.cpp

Lines 704 to 714 in cc46ee7

    
           void FT2Font::load_glyph(FT_UInt glyph_index, FT_Int32 flags) 
        
           { 
        
               if (FT_Error error = FT_Load_Glyph(face, glyph_index, flags)) { 
        
                   throw_ft_error("Could not load glyph", error); 
        
               } 
        
               FT_Glyph thisGlyph; 
        
               if (FT_Error error = FT_Get_Glyph(face->glyph, &thisGlyph)) { 
        
                   throw_ft_error("Could not get glyph", error); 
        
               } 
        
               glyphs.push_back(thisGlyph); 
        
           }

we will throw c++ exceptions (which I am 99% sure we turn into Python exceptions via the CALL_CPP macro). So it is not clear to me that the .notdef is actually missing here, but rather that it is empty.

Looking at https://freetype.org/freetype2/docs/reference/ft2-character_mapping.html#ft_get_char_index it seems that freetype will always ensure that 0 is the "missing glyph" so it is unclear to me if it is even possible to detect that it is missing via freetype. I think this also means that freetype as already done "the .notdef character isn't defined, it fallsback to the default .notdef character." for us. If there default is empty, then I think we need to understand why they made that choice before we try to second-guess it.

I am not convinced that this is something we want to do.

Currently we drop a minus and don't warn that we're dropping it. Besides being not great for folks doing a quick scan, it's not great for folks using a screen reader that they can't trust that the code they wrote is producing the image they expected.

ETA: Pushing on this more, of course users should know what their data looks like but frankly plenty of our users don't and while they likely shouldn't be using weird xkcd font they may and silently dropping the - is breaking the basic assumption that the visualization is accurately mapping the data b/c it implies the data is something else.

Which maybe in addition/alternatively ticks should raise a glyph not found error if it can't find a -.

Do we have any evidence that this is a wider problem than that one font

There's no way to determine if this is just a problem w/ this one font, but technically any custom font could have this problem & we explicitly aren't trying to monitor what fonts our users are using. And also it's a problem for this font that is prominently used for xkcd & even if I remove it from the docs (which I do in #26854), there's nothing to stop our users from using this font and users have flagged this error before.

As far as I can tell, the convention for .notdef characters is that it's always character 0 in any arbitrary font.

ETA2: to be clear/fair, I don't have a strong preference on the solution. Something like we ship a .notdef we always use and minus always falls back to hyphen, whatever is workable - I just think fundamentally we shouldn't knowingly be taking the risk of generating labels that don't match what the users specified or data calls for where visible not defs count as matching cause they're a warning).

However, it looks like freetype is already ensuring that .notdef is available:
So the questions are:

for this font did freetype make the default .notdef or does the font actually have an .notdef and it is just empty? (I think it is the second making the rest of this moot for this case)
if freetype made the default .notdef did they pick empty? and if so why?
from the freetype API can we tell that any of the above have happened?

A bigger question is a font even a valid font without a .notdef (it seems that type42 requires it and TTF at at least requires the 'post' table which in some forms requires .notdef)?

Poking at this font in fontforge it looks like it does indeed provide .notdef it is blank. If I am understand this correctly, then there is no way we can detect if the .notdef glyph is "good" so none of this proposal would actually fix the problem.

At some level, a font can randomize the mapping between the numbers and the glyphs (e.g. it render "1" to look like "7" or something) and there is nothing we can do about that.

The minimal solution is we should burn Humor Sans out of our code base (both for this reason and because it replaces € with an emoji of Hitler). It is probably with cleaving that off of the bigger xkcd PR so we can backport it.

I think InDesign has an interesting/possibly we should steal approach to missing glyphs: Missing Glyph Protection

where they default to coloring the space for the missing glyph light pink (which we probably can't do that, but we could possibly outline or hatch the textarea?), but also allow users to fallback to filling in with another font (which is what we already do with fallback)

Are we alright with closing this now that #29356 is in?

Yup, seems to work in xkcd mode (and some other PR fixed the missing - in xkcd), thanks!

Navigation Menu

Repository navigation

BUG: .notdef glyph has to be present in fonts in fontlist #27232

Suggested Fix

Good first issue - notes for new contributors

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if (!was_found) {
	ft_glyph_warn((FT_ULong)codepoints[n], glyph_seen_fonts);
	// render missing glyph tofu
	// come back to top-most font
	ft_object_with_glyph = this;
	char_to_font[codepoints[n]] = ft_object_with_glyph;
	glyph_to_font[glyph_index] = ft_object_with_glyph;
	ft_object_with_glyph->load_glyph(glyph_index, flags, ft_object_with_glyph, false);
	}

	void FT2Font::load_glyph(FT_UInt glyph_index, FT_Int32 flags)
	{
	if (FT_Error error = FT_Load_Glyph(face, glyph_index, flags)) {
	throw_ft_error("Could not load glyph", error);
	}
	FT_Glyph thisGlyph;
	if (FT_Error error = FT_Get_Glyph(face->glyph, &thisGlyph)) {
	throw_ft_error("Could not get glyph", error);
	}
	glyphs.push_back(thisGlyph);
	}

BUG: .notdef glyph has to be present in fonts in fontlist #27232

Description

Suggested Fix

Activity

ksunden commented on Oct 31, 2023

story645 commented on Oct 31, 2023

tacaswell commented on Nov 2, 2023

story645 commented on Nov 2, 2023

tacaswell commented on Nov 2, 2023

story645 commented on Nov 2, 2023

story645 commented on Nov 2, 2023

story645 commented on Nov 2, 2023

story645 commented on Nov 3, 2023

github-actions commented on Nov 8, 2023

Good first issue - notes for new contributors

story645 commented on Nov 8, 2023

tacaswell commented on Nov 9, 2023

story645 commented on Nov 9, 2023

tacaswell commented on Nov 9, 2023

story645 commented on Nov 10, 2023

QuLogic commented on Jul 1, 2025

story645 commented on Jul 1, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions