[Doc]: 3.11 text shaping silently reverses the arabic_reshaper + python-bidi workaround #32262

Documentation Link

https://matplotlib.org/devdocs/users/explain/text/text_intro.html

Problem

Since #30000 landed, matplotlib shapes and reorders complex scripts itself. That makes the workaround recommended in essentially every "Arabic text in matplotlib" answer actively wrong:

text = get_display(arabic_reshaper.reshape(text))   # necessary before 3.11, reversed now

The string is already in visual order, matplotlib applies the bidi algorithm again, and the label renders reversed. Nothing raises, and to a reader who does not read Arabic the output still looks like Arabic in the right place, which is why I think it is worth a note rather than leaving it to be discovered.

arabic_reshaper appears nowhere in the docs, so nothing currently tells someone upgrading that the pre-processing should be removed.

Measured on matplotlib 3.11.0, libraqm 0.10.5, macOS, using SFArabic.ttf and the string الإمارات. Mean absolute pixel difference, normalised to 64px height, against a Pillow/Raqm render of the same string in the same font:

One extra detail worth knowing: the reshape collapses lam-alef, so الإمارات goes from 8 codepoints to 7. Where an application stored the pre-shaped string rather than only rendering it, mapping the presentation forms back does not recover the original, because that one codepoint expands in logical order while its neighbours are still in visual order. السلام comes back as السالم, a real but different word.

import matplotlib
matplotlib.use("Agg")
import arabic_reshaper, numpy as np
import matplotlib.pyplot as plt, matplotlib.ft2font as ft
from bidi.algorithm import get_display
from matplotlib import font_manager
from PIL import Image, ImageDraw, ImageFont, features

FONT = "/System/Library/Fonts/SFArabic.ttf"   # any Arabic-capable TTF
TEXT = "الإمارات"
PRE = get_display(arabic_reshaper.reshape(TEXT))
print("matplotlib", matplotlib.__version__, "libraqm", ft.__libraqm_version__)
print("codepoints:", len(TEXT), "->", len(PRE))

fp = font_manager.FontProperties(fname=FONT)

def mpl(text, path):
    fig = plt.figure(figsize=(6, 1.3), dpi=100)
    fig.text(0.04, 0.3, text, fontproperties=fp, fontsize=48)
    fig.savefig(path, facecolor="white"); plt.close(fig)

def pil(text, path, px=67):
    font = ImageFont.truetype(FONT, px, layout_engine=ImageFont.Layout.RAQM)
    img = Image.new("L", (600, 130), 255)
    ImageDraw.Draw(img).text((25, 15), text, font=font, fill=0); img.save(path)

mpl(TEXT, "a_raw.png"); mpl(PRE, "b_preshaped.png"); pil(TEXT, "c_reference.png")

def normalised(path, h=64):
    a = 255 - np.asarray(Image.open(path).convert("L"), float)
    ys, xs = np.nonzero(a > 40)
    im = Image.fromarray((255 - a[ys.min():ys.max()+1, xs.min():xs.max()+1]).astype("uint8"))
    return np.asarray(im.resize((int(im.width * h / im.height), h)), float)

def diff(p, q):
    a, b = normalised(p), normalised(q)
    w = min(a.shape[1], b.shape[1])
    return float(np.abs(a[:, :w] - b[:, :w]).mean())

print("raw       vs reference:", round(diff("a_raw.png", "c_reference.png"), 2))
print("preshaped vs reference:", round(diff("b_preshaped.png", "c_reference.png"), 2))

Suggested improvement

A documentation-only note, in the text guide and in the 3.11 release note: pass the logical string and remove any arabic_reshaper / python-bidi pre-processing, because matplotlib now does the shaping and reordering.

Worth saying plainly that this is unconditional, unlike the equivalent situation in Pillow. I checked extern/meson.build before writing this: libraqm is a required dependency, resolved either from the system or from the vendored static subproject, so there is no build in which the old pre-processing is still correct.

Happy to open that PR if you would like it. I did the equivalent for Pillow in python-pillow/Pillow#9925.

matplotlib Version

3.11.0

Operating system

macOS