Repository navigation
Closed
Labels
Milestone
Description
Hi,
The environment is as follows: Ubuntu 19.10, python 3.8.2 managed by pyenv and matplotlib git master version. See the following for the testing steps and results:
$ sudo apt-get build-dep python3-matplotlib python3-matplotlib-venn
$ pip install -e .[dev,extra,all]
$ pytest
========================================== test session starts ==========================================
platform linux -- Python 3.8.2, pytest-5.4.1, py-1.5.4, pluggy-0.13.1
rootdir: /home/werner/Public/hpc/tools/matplotlib.git, inifile: pytest.ini, testpaths: lib
plugins: cov-2.8.1, datadir-1.3.1, regressions-1.0.6, timeout-1.3.4
collected 7591 items
[...]
================================================ FAILURES =================================================
___________________________________________ test_pdflatex[pdf] ____________________________________________
expected = '/home/werner/Public/hpc/tools/matplotlib.git/result_images/test_backend_pgf/pgf_pdflatex-expected.pdf'
actual = PosixPath('/home/werner/Public/hpc/tools/matplotlib.git/result_images/test_backend_pgf/pgf_pdflatex.pdf')
tol = 0
def _raise_on_image_difference(expected, actual, tol):
__tracebackhide__ = True
err = compare_images(expected, actual, tol, in_decorator=True)
if err:
for key in ["actual", "expected"]:
err[key] = os.path.relpath(err[key])
> raise ImageComparisonFailure(
'images not close (RMS %(rms).3f):\n\t%(actual)s\n\t%(expected)s '
% err)
E matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 11.669):
E result_images/test_backend_pgf/pgf_pdflatex_pdf.png
E result_images/test_backend_pgf/pgf_pdflatex-expected_pdf.png
lib/matplotlib/testing/decorators.py:135: ImageComparisonFailure
______________________________________________ test_rcupdate ______________________________________________
@needs_xelatex
@needs_pdflatex
@pytest.mark.skipif(not _has_sfmath(), reason='needs sfmath.sty')
@pytest.mark.style('default')
@pytest.mark.backend('pgf')
def test_rcupdate():
rc_sets = [{'font.family': 'sans-serif',
'font.size': 30,
'figure.subplot.left': .2,
'lines.markersize': 10,
'pgf.rcfonts': False,
'pgf.texsystem': 'xelatex'},
{'font.family': 'monospace',
'font.size': 10,
'figure.subplot.left': .1,
'lines.markersize': 20,
'pgf.rcfonts': False,
'pgf.texsystem': 'pdflatex',
'pgf.preamble': ('\\usepackage[utf8x]{inputenc}'
'\\usepackage[T1]{fontenc}'
'\\usepackage{sfmath}')}]
tol = [6, 0]
for i, rc_set in enumerate(rc_sets):
with mpl.rc_context(rc_set):
create_figure()
> compare_figure('pgf_rcupdate%d.pdf' % (i + 1), tol=tol[i])
lib/matplotlib/tests/test_backend_pgf.py:157:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
fname = 'pgf_rcupdate2.pdf', savefig_kwargs = {}, tol = 0
def compare_figure(fname, savefig_kwargs={}, tol=0):
actual = os.path.join(result_dir, fname)
plt.savefig(actual, **savefig_kwargs)
expected = os.path.join(result_dir, "expected_%s" % fname)
shutil.copyfile(os.path.join(baseline_dir, fname), expected)
err = compare_images(expected, actual, tol=tol)
if err:
> raise ImageComparisonFailure(err)
E matplotlib.testing.exceptions.ImageComparisonFailure: Error: Image files did not match.
E RMS Value: 13.166575390337284
E Expected:
E /home/werner/Public/hpc/tools/matplotlib.git/result_images/test_backend_pgf/expected_pgf_rcupdate2_pdf.png
E Actual:
E /home/werner/Public/hpc/tools/matplotlib.git/result_images/test_backend_pgf/pgf_rcupdate2_pdf.png
E Difference:
E /home/werner/Public/hpc/tools/matplotlib.git/result_images/test_backend_pgf/pgf_rcupdate2_pdf-failed-diff.png
E Tolerance:
E 0
lib/matplotlib/tests/test_backend_pgf.py:63: ImageComparisonFailure
____________________________________________ test_usetex[png] _____________________________________________
expected = '/home/werner/Public/hpc/tools/matplotlib.git/result_images/test_usetex/test_usetex-expected.png'
actual = PosixPath('/home/werner/Public/hpc/tools/matplotlib.git/result_images/test_usetex/test_usetex.png')
tol = 0
def _raise_on_image_difference(expected, actual, tol):
__tracebackhide__ = True
err = compare_images(expected, actual, tol, in_decorator=True)
if err:
for key in ["actual", "expected"]:
err[key] = os.path.relpath(err[key])
> raise ImageComparisonFailure(
'images not close (RMS %(rms).3f):\n\t%(actual)s\n\t%(expected)s '
% err)
E matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 12.877):
E result_images/test_usetex/test_usetex.png
E result_images/test_usetex/test_usetex-expected.png
lib/matplotlib/testing/decorators.py:135: ImageComparisonFailure
========================================= short test summary info =========================================
FAILED lib/matplotlib/tests/test_backend_pgf.py::test_pdflatex[pdf] - matplotlib.testing.exceptions.Imag...
FAILED lib/matplotlib/tests/test_backend_pgf.py::test_rcupdate - matplotlib.testing.exceptions.ImageComp...
FAILED lib/matplotlib/tests/test_usetex.py::test_usetex[png] - matplotlib.testing.exceptions.ImageCompar...
=================== 3 failed, 7517 passed, 60 skipped, 11 xfailed in 630.55s (0:10:30) ====================
Any hints for solving this problems?
Regards
Activity
[-]pytest failed for some tests.[/-][+]Matploglib git master version fails to pass serveral pytest's tests.[/+][-]Matploglib git master version fails to pass serveral pytest's tests.[/-][+]Matplotlib git master version fails to pass serveral pytest's tests.[/+]timhoffm commentedon May 3, 2020
Compare the actual and expected images in the
result_imagesfolder. That tells you what's unexpected and will give a hint what's causing the failure.hongyi-zhao commentedon May 3, 2020
I find all of the failed-diff files as following:
But some of them are "empty" without content, others have something like garbled. I still cannot figure out the reason. See the following for you info:
anntzer commentedon May 3, 2020
What is your version of dvipng? If it is 1.16 there's a known issue about that (#16476 (comment)). Otherwise please delete your ~/.cache/matplotlib/tex.cache and try again.
hongyi-zhao commentedon May 4, 2020
Now, I updated to dvipng 1.17 shipped with TeX Live 2020, see following:
But pytest still throws errors, see following:
tacaswell commentedon May 4, 2020
Almost all of our tests are set to 0 difference thresholds, some of the "empty" images make have only a few pixels off by a few bits which is extremely hard to see by eye in the difference images. That said, the three images in
test_compare_images/are red-herrings, they are testing that things do fail when expected.hongyi-zhao commentedon May 4, 2020
But what's the meaning of this from my testing results:
60 skipped, 11 xfailedRegards
tacaswell commentedon Jun 2, 2020
Sorry for the very slow response
That means 60 tests were skipped (usually because you are missing an optional dependancy) and 11 tests that we expect to fail failed (some are tests that are documenting bugs, at least one is testing that the tests fail when we think they should).
hongyi-zhao commentedon Jun 2, 2020
How to check out the full list for these dependencies?
I install the git master version of matplotlib using the following command in its local repo folder:
$ pip install -e .tacaswell commentedon Jun 2, 2020
Look at the files https://github.com/matplotlib/matplotlib/tree/master/requirements/testing for what we install on CI.
I think we have some platform specific tests so it may not be possible to run every test on your system (this is why we test on windows, linux, and OSX as part of CI).
I suspect you are missing one or more of the GUI frameworks.
hongyi-zhao commentedon Jun 2, 2020
Still no so clear. There are several files and do you mean I should install all the packages listed in these files for doing the tests?
hongyi-zhao commentedon Jun 2, 2020
I installed the requirements list here with the following commands:
Then run pytest and obtain the following results:
tacaswell commentedon Jun 9, 2020
Moved this to 3.4 as I think that the pdf / pgf failures are due to the latex changing under us.
hongyi-zhao commentedon Jun 9, 2020
@tacaswell Have you tested them?
QuLogic commentedon Jun 30, 2020
For
test_usetex, I can always reproduce that failure on Fedora 31+ (it used to be okay in 30, I think.) For the pgf tests, that failure can be reproduce on Fedora Rawhide only, because it has new texlive. Somewhere in the latex stack, the degree glyph was switched from Computer Modern to Latin Modern (which matches the rest of the text), so the line is a bit different now.QuLogic commentedon Jun 3, 2021
Those remaining tests have now got updated images, because we no longer run CI on Ubuntu 16.04 that needed the old ones. So I'm closing this.
[-]Matplotlib git master version fails to pass serveral pytest's tests.[/-][+]Matplotlib git master version fails to pass several pytest's tests[/+]