Change in OSX Catalina makes matplotlib + multiprocessing crash

I'm getting a crash on macOS Catalina when running some multiprocessing code after doing some plotting with matplotlib.

Looks like this:

libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Couldn't close file
Fatal Python error: Aborted

and using print(psutil.Process().open_files()) I figured out that likely the issue is that this MPL TTF font file is opened multiple times and (I guess) for some reason closing the file occurs only later, from my multiprocessing code, and that fails on macOS Catalina.

open files: [popenfile(path='/Users/deil/anaconda3/envs/gammapy-dev/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf', fd=5), popenfile(path='/Users/deil/anaconda3/envs/gammapy-dev/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSans.ttf', fd=6), popenfile(path='/Users/deil/anaconda3/envs/gammapy-dev/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansDisplay.ttf', fd=7), popenfile(path='/Users/deil/anaconda3/envs/gammapy-dev/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/ttf/DejaVuSansDisplay.ttf', fd=8)]

I tried to extract a minimal test case that doesn't involve our software, but couldn't so far, sorry.
But you should be able to reproduce the crash by copy & pasting these commands:

git clone https://github.com/gammapy/gammapy.git
cd gammapy
conda env create -f environment-dev.yml
pip install -e .
pytest -s -v gammapy/cube

Log: https://gist.github.com/cdeil/a75211856a3bcab751ead707df9708c9#file-gistfile1-txt-L88

I know MPL isn't thread-safe. But it should be possible to do some plotting, and then later after plotting is done to run some multiprocessing code, no?

Is it normal that the file handle for DejaVuSansDisplay.ttf remains open?
Shouldn't MPL load the file content and close the file handle directly?
Maybe that is a bug?

I don't think it matters for this issue, but in case it does -- we are calling plt.close() after all plotting from our tests, because previously we had a problem with too many open figures (see [here](
https://github.com/gammapy/gammapy/blob/d4db11d559b210c0e0d03bba75ddcff9c03e0511/gammapy/utils/testing.py#L209-L230

(gammapy-dev) hfm-1804a:gammapy deil$ python --version
Python 3.7.0
(gammapy-dev) hfm-1804a:gammapy deil$ python -c 'import matplotlib; print(matplotlib.__version__)'
3.1.1
(gammapy-dev) hfm-1804a:gammapy deil$ python -c 'import matplotlib; print(matplotlib.get_backend())'
MacOSX

I think we are holding the ttf open because we do not pull the full font up into ram, but rather go back to the file to pull glyphs out as needed. IIRC, this is done for performance reasons.

It does seem wrong that we have it open twice though (and I also see that behavior).

I also have a vague memory that multi-process behaves differently on OSX than on linux, could that be part of the problem here?

I think we are holding the ttf open because we do not pull the full font up into ram, but rather go back to the file to pull glyphs out as needed. IIRC, this is done for performance reasons.

Do you know who introduced it / which issue? (I couldn't find it with a quick search)
Is it up for discussion to change this?

It does seem wrong that we have it open twice though (and I also see that behavior).

With or without multiprocessing involved?
Is the idea of TTF file handling to open once, and keep it open until the process ends?
Or is there some code to close the file, and maybe that just has to be written to be more deterministic?

My (uneducated -- I don't have a mac to test this) guess is that this is not a change on mpl's side, but rather due to a change in how the newest MacOS handles FILE* pointers after a fork() (which is how multiprocessing is implemented by default on Unices), e.g. what happens if they are close()d in a separate process.

I believe this was done in #5299 and #5410 as part of the 2.0 work.

I get two file handles to each font in without multiprocessing.

What version of mpl and python are you testing with? Poking around at this I found

matplotlib/lib/matplotlib/font_manager.py

Lines 1329 to 1334 in 19d589c

    
           # FT2Font objects cannot be used across fork()s because they reference the same 
        
           # FT_Library object.  While invalidating *all* existing FT2Fonts after a fork 
        
           # would be too complicated to be worth it, the main way FT2Fonts get reused is 
        
           # via the cache of _get_font, which we can empty upon forking (in Py3.7+). 
        
           if hasattr(os, "register_at_fork"): 
        
               os.register_at_fork(after_in_child=_get_font.cache_clear)

which suggest py37 and mpl31 may not have a problem?

Is this a mac only problem?

@anntzer

I've only seen this issue in a very specific case: on my macBook, after upgrade to macOS version 10.15 "Catalina", when running this with Python 3.7 from Anaconda, and MPL 3.1.1:

git clone https://github.com/gammapy/gammapy.git
cd gammapy
conda env create -f environment-dev.yml
pip install -e .
pytest -s -v gammapy/cube

I didn't see the issue on older macOS versions, and we don't see it on Linux or Windows, and I couldn't reproduce the issue with a simpler test case or outside of pytest.
So my guess is that @anntzer above is right - it looks like in that case MPL tries to close a font file from a child process, and that fails for some reason.

Does _get_font.cache_clear close the file?
Then I don't understand why I can't reproduce the crash with a simple test case, doing some plotting and then after doing some multiprocessing.

Probably others will see the same issue as they update to macOS version 10.15 and run Python multiprocessing code (and most not making the connection that MPL trying to close a file from a child process is the issue, because the filename isn't printed).

The problem is that we're really stuck here: the only option I know of is to remove the use of multiprocessing completely from our library (Gammapy) to fix this.

I think it's very exceptional that scientific Python libs keep files open?
Numpy, Scipy, pandas, ... never do this?
And MPL only does it for the font files?

Is changing this behaviour up for debate?
E.g. load full font files in memory and only keep the LRU cache there, but close the files immediately?
That would add ~ 1 MB RAM usage for typical usage, and avoid this issue, make MPL "stateless" wrt. open filehandles, like the other Python libs?

I guess the problem is that cache_clear occurs too late -- it closes the file after the fork(). We could instead do this before the fork but then every new process spawn will result in the font cache being flushed, which seems wrong.
I think fully loading the font file is fine if you can confirm there are no performance issues.

Before we go to far down this route, I think we need to confirm that it actually is mpl's file handles that are the problem not something else in gammapy. That you can only reproduce it in a test suite run via pytest also makes me think that this is not Matplotlib specific, or at a minimum that we do not fully understand the source of the problem, as pytest does very magical things under the hood. I think https://bugs.python.org/issue33725 is relevant and suggests that this is not a Matplotlib specific problem.

the only option I know of is to remove the use of multiprocessing completely from our library (Gammapy) to fix this.

Can you try forcing one of the other multi-process methods?

I don't think holding files open is that odd. h5py is built around holding a file handle open and anything with memory mapping will hold open file handles. Fundamentally, font files are data files so I think it fits the pattern.

IIRC, Mike bench marked that getting the glyphs out of the c api was faster than dictionary lookup. I am also concerned about start up time. It may just be a few extra M of ram, but it is something we would have to read into memory on every import.

I am deeply skeptical of a major overhaul of our font handling system based on the information we currently have.

Can you try forcing one of the other multi-process methods?

Will do, but only in a few days.

I don't think holding files open is that odd. h5py is built around holding a file handle open and anything with memory mapping will hold open file handles. Fundamentally, font files are data files so I think it fits the pattern.

That's not the same. With h5py I can open a file, read my data, close the file, and then do some multiprocessing computations, without any issue. The problem is that matplotlib keeps the font files open unexpectedly, and closes them at a time which can crash on some operating systems (from the child process after forking).

I am deeply skeptical of a major overhaul of our font handling system based on the information we currently have.

Makes sense. I don't know much about multiprocessing and fork/exec and such things, but I'll try to come up with simple test cases of when this crash does and doesn't occur with MPL.

I am having the same issue, but unfortunately won't be able to contribute much to the problem's resolution. :/ I was simply wondering if there is a workaround available until you guys find a solution for this?!

Thanks for any help!

It may be worth trying to track down and close all of the file objects that get opened before you fork.

The other option is to only import Matplotlib after you have launched your multiprocess code (so there are no file handles that can leak between processes).

If I am reading the issues from upstream correctly, moving to py38 may also fix the problem, but you will need to compile many things from source as not a lot of wheels are up yet (ex ours ;) ).

@cdeil

I too had this issue recently; and I too just upgraded my 2015 MacBook Pro to 10.15 Catalina.

I wrote my own object-oriented code from scratch for a physics project, and it uses multiprocessing.Pool; but I copy/pasted a friend's matplotlib configuration scripts. In their code, github.com/dfm/exoplanet, they assign some specific font related rcParams. After using his configuration, I received a warning that DejaVuSans.ttf could not be found [or some such] -- that's what led me down this path.

Every 5th - 10th time that I ran my code, I would receive the error above (libc++abi.dylib: etc etc) and had to crash my terminal to start all over. After searching the internet with the copy/paste of the error output, I found @cdeil's issue page on gammapy, which linked me here.

I'm telling you all of this to describe that after seeing what files were open on @cdeil's machine, I turned off the matplotlib configuration scripts -- which involves minor font selection. I also became over cautious about using matplotlib in the same terminal as my MPI based package; so I would run the MPI package, then save the output, load it into a new terminal just to plot it. Either way it 'worked'; i.e. I never got the error again. It was never deterministic; but I've now run the MPI based code 20+ times with no errors.

I had the feeling that it was 'just this one tff file'; but I can imagine that it's all of them. That is, it feels related to how OSX works with MPL in an MPI framework.

@exowanderer

@exowanderer Are you using both multiprocessing.Pool and mpi and both have problems? Could you try switching how python deals with multiprocessing on OSX?

Can you try whether #15104 helps? (at least the place throwing the exception shouldn't exist anymore...)

If that doesn't help, does

diff --git i/lib/matplotlib/font_manager.py w/lib/matplotlib/font_manager.py
index 6d56ed595..3428e831a 100644
--- i/lib/matplotlib/font_manager.py
+++ w/lib/matplotlib/font_manager.py
@@ -1331,7 +1331,7 @@ _get_font = lru_cache(64)(ft2font.FT2Font)
 # would be too complicated to be worth it, the main way FT2Fonts get reused is
 # via the cache of _get_font, which we can empty upon forking (in Py3.7+).
 if hasattr(os, "register_at_fork"):
-    os.register_at_fork(after_in_child=_get_font.cache_clear)
+    os.register_at_fork(before=_get_font.cache_clear)
 
 
 def get_font(filename, hinting_factor=None):

fix the issue? (Probably this should be done conditionally on the libc++ version to avoid degrading the performance of fork()ing unless needed.) (#15104 would be preferable, if it works.)

As requested on twitter (https://twitter.com/matplotlib/status/1249878438883872768?s=20), here is some code that produces this issue for me on OSX 10.15.4 using Py37:

import numpy as np
from qiskit.tools.parallel import parallel_map

#import matplotlib.pyplot as plt

#x = np.arange(30)
#plt.plot(x, np.sin(x))

def f(x):
    return np.sin(x)/2

for _ in range(10):
    parallel_map(f, np.arange(50))

Here parallel_map is a wrapper around ProcessPoolExecutor: https://github.com/Qiskit/qiskit-terra/blob/master/qiskit/tools/parallel.py

It runs fine with all the MPL stuff commented out. However, after uncommenting and plotting it will fail with the error:

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

and the terminal (Above is running in a notebook) shows:

libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Couldn't close file

We have a similar issue on macOS Catalina, and one workaround that worked for us was upgrading to Python 3.8.2;;

@nonhermitian

@nonhermitian @hisplan Can you please check https://bugs.python.org/issue33725 In particular https://bugs.python.org/issue33725#msg329923 which suggests that adding

mp.set_start_method('forkserver')

will fix this problem on all versions of python3 (the reason upgrading to 3.8.2 works is that this is now the default).

Fix to bug matplotlib/matplotlib#15410

Fix to bug matplotlib/matplotlib#15410 (#88)

@HeyLookItsBrandon

Using the test notebook provided by @HeyLookItsBrandon (#15410 (comment)) I can reproduce the failure with python 3.7.3.

Good news: with @anntzer's #15104, the failure does not occur.

update to package dependencies after nightly build errors

I am going to close this as fixed by:

a) upstream changing its defaults
b) the ability for the user to manually set the multi-process mode so OSX does not behave badly
c) #15104

update to package dependencies after nightly build errors

Don't run image comparison test on MacOS because we need a more recen…

Fix to bug matplotlib/matplotlib#15410 (#88)

Navigation Menu

Repository navigation

Change in OSX Catalina makes matplotlib + multiprocessing crash #15410

22 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	# FT2Font objects cannot be used across fork()s because they reference the same
	# FT_Library object. While invalidating all existing FT2Fonts after a fork
	# would be too complicated to be worth it, the main way FT2Fonts get reused is
	# via the cache of _get_font, which we can empty upon forking (in Py3.7+).
	if hasattr(os, "register_at_fork"):
	os.register_at_fork(after_in_child=_get_font.cache_clear)

Change in OSX Catalina makes matplotlib + multiprocessing crash #15410

Description

Activity

tacaswell commented on Oct 13, 2019

cdeil commented on Oct 14, 2019

anntzer commented on Oct 14, 2019

tacaswell commented on Oct 14, 2019

cdeil commented on Oct 14, 2019

anntzer commented on Oct 14, 2019

tacaswell commented on Oct 14, 2019

cdeil commented on Oct 15, 2019

smeingast commented on Oct 22, 2019

tacaswell commented on Oct 22, 2019

exowanderer commented on Oct 25, 2019

tacaswell commented on Oct 27, 2019

anntzer commented on Oct 27, 2019

22 remaining items

nonhermitian commented on Apr 29, 2020

hisplan commented on May 3, 2020

tacaswell commented on May 7, 2020

efiring commented on May 7, 2020

tacaswell commented on May 21, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions