Add a last resort font for missing glyphs by QuLogic · Pull Request #29356 · matplotlib/matplotlib

QuLogic · 2024-12-20T11:23:56Z

PR summary

Some time ago, I came across https://github.com/unicode-org/last-resort-font. Through some charmap magic, the "High Efficiency" font from this project is able to provide a glyph for every Unicode codepoint while only being about 550K. (The expanded version in that repo in 8.6MB.)

This is a proof-of-concept for applying this font as final fallback at (almost) all times. I have added an rcParam to turn it off, mostly for one test to work, but we may not want to use that. I'm also not sure if it's been inserted in the best place. We may instead want to insert the glyph at the point where we have none, so that we can warn about it too.

These glyphs show a representative character from the Unicode block, a square frame, and if large enough, the block name and code point range:

import matplotlib.pyplot as plt

text = '\N{Bengali Digit Zero}\N{Hiragana Letter A}\ufdd0'
sizes = [
    (0.90, 6),
    (0.85, 8),
    (0.80, 10),
    (0.75, 12),
    (0.70, 16),
    (0.63, 20),
    (0.55, 24),
    (0.45, 32),
    (0.30, 48),
    (0.10, 64),
]

fig = plt.figure()
for y, size in sizes:
    fig.text(0.01, y, f'{size}pt: {text}', fontsize=size)
plt.show()

PR checklist

[n/a] "closes #0000" is in the body of the PR description to link the related issue
new and changed code is tested
Plotting related features are demonstrated in an example
New Features and API Changes are noted with a directive and release note
Documentation complies with general and docstring guidelines

story645 · 2024-12-20T17:43:25Z

would this also address #27232 ?

QuLogic · 2024-12-20T21:21:15Z

I haven't tested it, but I think so.

QuLogic · 2025-02-22T10:03:45Z

IIRC, our conversation in a previous meeting concluded with moving this fallback to the missing-glyph warning trigger. While this is possible with some refactoring, I've realized we reach a problem with font size. This would not have been set for the last-resort font, and it's a bit difficult to copy that information from another font due to the way FreeType stores it (it's stored in some font units, not the original call units.) If we have a global object for it, then we'd also have to worry about threading issues if propagating the size. So overall, this may not be the best course of action.

However, that does mean we'd need to find an alternative means of ensuring the missing-glyph warning appears.

QuLogic · 2025-02-25T00:32:39Z

So in the end, I just added a flag to the Last Resort FT2Font to indicate whether to warn or not. That's simpler than trying to reach back into Python somehow to ask if it should.

I don't recall if we concluded that the rcParam was unnecessary, but I've left it in for now.

QuLogic · 2025-02-25T03:42:32Z

I think I'm missing something super-obvious, as I don't know why the What's new plot isn't displaying anything.

QuLogic · 2025-03-25T00:07:36Z

OK, I've figured out why the plot is missing. The warning is preserved here, and since the doc builds set -Werror, this causes an exception in savefig.

This is supposed to be caught:

matplotlib/lib/matplotlib/sphinxext/plot_directive.py

Lines 700 to 713 in a9dc9ac

    
           for fmt, dpi in formats: 
        
               try: 
        
                   figman.canvas.figure.savefig(img.filename(fmt), dpi=dpi) 
        
                   if fmt == formats[0][0] and config.plot_srcset: 
        
                       # save a 2x, 3x etc version of the default... 
        
                       srcset = _parse_srcset(config.plot_srcset) 
        
                       for mult, suffix in srcset.items(): 
        
                           fm = f'{suffix}.{fmt}' 
        
                           img.formats.append(fm) 
        
                           figman.canvas.figure.savefig(img.filename(fm), 
        
                                                        dpi=int(dpi * mult)) 
        
               except Exception as err: 
        
                   raise PlotError(traceback.format_exc()) from err 
        
               img.formats.append(fmt)

and reported:

matplotlib/lib/matplotlib/sphinxext/plot_directive.py

Lines 868 to 875 in a9dc9ac

    
           except PlotError as err: 
        
               reporter = state.memo.reporter 
        
               sm = reporter.system_message( 
        
                   2, "Exception occurred in plotting {}\n from {}:\n{}".format( 
        
                       output_base, source_file_name, err), 
        
                   line=lineno) 
        
               results = [(code, [])] 
        
               errors = [sm]

but I've never seen this output.

Because the exception occurs in savefig (which is outside of the plot directive), there is nothing we can do from inside the plot code itself. So I'm going to just un-error these warnings globally.

QuLogic · 2025-04-03T01:32:30Z

While testing out fallbacks with libraqm, I noticed that these don't work in PDF for some reason. Still need to investigate that a bit closer.

QuLogic · 2025-04-03T06:19:50Z

So instead of using the existing font objects, the PDF backend tracks fonts by filename, and then loads them again individually when writing the final font subsets. What this meant is that a) the Last Resort font was loaded without selecting the right charmap and b) the Last Resort fallback was added to Last Resort if it was explicitly asked for (and only this fallback had the right charmap).

I've modified _get_font so that is more resilient to these cases and always sets the charmap for Last Resort, as well as not duplicating a fallback for it. Also added a small test image for the results just in case we change PDF/SVG.

QuLogic · 2025-04-03T06:30:43Z

Also, I had to bump the GitHub cache key so that test_get_font_names passed; this is because we have a new font in mpl-data/fonts. Does that warrant bumping FontManager.__version__? Or not, because it's only a developer problem?

tacaswell · 2025-04-18T16:44:23Z

-    : ft_glyph_warn(warn), image(), face(nullptr)
+                 FT2Font::WarnFunc warn, bool warn_if_used)
+    : ft_glyph_warn(warn), warn_if_used(warn_if_used), image(), face(nullptr),
+      hinting_factor(hinting_factor_),


Suggested change

hinting_factor(hinting_factor_),

// set a default fontsize 12 pt at 72dpi

hinting_factor(hinting_factor_),

and presumably the comment should be removed below?

No, that had nothing to do with the hinting factor, which is why I left it there.

tacaswell · 2025-04-18T16:44:35Z

-    // set default kerning factor to 0, i.e., no kerning manipulation
-    kerning_factor = 0;
-
    // set a default fontsize 12 pt at 72dpi


Suggested change

// set a default fontsize 12 pt at 72dpi

tacaswell

Left one minor comment about moving a comment.

QuLogic added status: needs tests status: needs documentation labels Dec 20, 2024

github-actions Bot added the topic: rcparams label Dec 20, 2024

QuLogic force-pushed the last-resort-font branch from 24a4b0e to 2a3d7a2 Compare February 24, 2025 21:55

github-actions Bot added the topic: text/fonts label Feb 24, 2025

QuLogic force-pushed the last-resort-font branch from 2a3d7a2 to e95972c Compare February 25, 2025 00:28

QuLogic removed status: needs tests status: needs documentation labels Feb 25, 2025

QuLogic added this to the v3.11.0 milestone Feb 25, 2025

QuLogic marked this pull request as ready for review February 25, 2025 00:28

QuLogic force-pushed the last-resort-font branch from e95972c to b081a80 Compare March 25, 2025 06:43

github-actions Bot added the Documentation: build building the docs label Mar 25, 2025

QuLogic force-pushed the last-resort-font branch 3 times, most recently from 6499c51 to b8b8e5a Compare March 28, 2025 06:18

QuLogic marked this pull request as draft April 3, 2025 01:32

QuLogic force-pushed the last-resort-font branch from b8b8e5a to 6f8cf65 Compare April 3, 2025 05:24

QuLogic marked this pull request as ready for review April 3, 2025 06:19

QuLogic force-pushed the last-resort-font branch 2 times, most recently from 4fe3a07 to c872584 Compare April 17, 2025 19:40

tacaswell reviewed Apr 18, 2025

View reviewed changes

tacaswell approved these changes Apr 18, 2025

View reviewed changes

story645 requested changes Apr 23, 2025

View reviewed changes

Comment thread LICENSE/LICENSE_LAST_RESORT_FONT Resolved

Comment thread doc/conf.py Resolved

Comment thread src/ft2font.cpp Resolved

Add a last resort font for missing glyphs

9abe036

QuLogic force-pushed the last-resort-font branch from c872584 to 9abe036 Compare April 24, 2025 02:36

story645 approved these changes Apr 24, 2025

View reviewed changes

story645 merged commit 4407d1d into matplotlib:main Apr 25, 2025

QuLogic deleted the last-resort-font branch April 25, 2025 19:50

QuLogic added this to Font and text overhaul Jun 12, 2025

github-project-automation Bot moved this to Waiting for other PR in Font and text overhaul Jun 12, 2025

QuLogic moved this from Waiting for other PR to Done in Font and text overhaul Jun 12, 2025

QuLogic mentioned this pull request Jul 1, 2025

BUG: .notdef glyph has to be present in fonts in fontlist #27232

Closed

	hinting_factor(hinting_factor_),
	// set a default fontsize 12 pt at 72dpi
	hinting_factor(hinting_factor_),

Conversation

QuLogic commented Dec 20, 2024

PR summary

PR checklist

Uh oh!

story645 commented Dec 20, 2024

Uh oh!

QuLogic commented Dec 20, 2024

Uh oh!

QuLogic commented Feb 22, 2025

Uh oh!

QuLogic commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

QuLogic commented Feb 25, 2025

Uh oh!

QuLogic commented Mar 25, 2025

Uh oh!

QuLogic commented Apr 3, 2025

Uh oh!

QuLogic commented Apr 3, 2025

Uh oh!

QuLogic commented Apr 3, 2025

Uh oh!

tacaswell Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

QuLogic Apr 22, 2025

Choose a reason for hiding this comment

Uh oh!

tacaswell Apr 18, 2025

Choose a reason for hiding this comment

Uh oh!

tacaswell left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

QuLogic commented Feb 25, 2025 •

edited

Loading