Repository navigation
2.0.0b4, debian unstable: lots of ImageComparisonFailure #7134
Description
Hello,
while testing 2.0.0b4 on debian unstable, there are a lot of ImageComparisonFailure. it seems that the same test is executed twice, and the first execution always fails, for example:
matplotlib.tests.test_axes.test_canonical.test ... FAIL
matplotlib.tests.test_axes.test_canonical.test ... ok
matplotlib.tests.test_axes.test_canonical.test ... ok
... ... ...
FAIL: matplotlib.tests.test_axes.test_canonical.test
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/build/matplotlib-2.0.0~beta4/build/lib.linux-x86_64-2.7/matplotlib/testing/decorators.py", line 58, in failer
result = f(*args, **kwargs)
File "/build/matplotlib-2.0.0~beta4/build/lib.linux-x86_64-2.7/matplotlib/testing/decorators.py", line 261, in do_test
'(RMS %(rms).3f)'%err)
ImageComparisonFailure: images not close: /build/matplotlib-2.0.0~beta4/result_images/test_axes/canonical.png vs. /build/matplotlib-2.0.0~beta4/result_images/test_axes/canonical-expected.png (RMS 0.003)
(that is the 2.7 failure, but it happens also on 3.5 and on their debug flavor interpreters).
the grand total of the testsuite is FAILED (KNOWNFAIL=69, SKIP=14, failures=887)
please let me know how i can help you debug this
Activity
matthew-brett commentedon Sep 18, 2016
On Sun, Sep 18, 2016 at 1:59 PM, Sandro Tosi notifications@github.com
wrote:
Is it possible you are building against the system freetype, and not the
matplotlib version?
I think the test tolerances depend on a specified freetype version that you
get with
export MPLLOCALFREETYPE=1
sandrotosi commentedon Sep 18, 2016
yes i'm using the system package for freetype (at version 2.6.3): in debian we try hard not to use internal versions of libraries we have available in the distro, and i would like to keep doing this with matplotlib as well. Is there a way to adjust the tests sensibility with a different version of freetype?
matthew-brett commentedon Sep 18, 2016
I know you try hard not to use internal versions :).
Maybe it would need an addition to the
image_comparisondecorator that would allow different tolerances for different freetype versions.tacaswell commentedon Sep 18, 2016
We previously had a default non-zero tolerance on the image comparisons, to account for freetype changing under us. However, this resulted in a couple of cases where we were rendering very wrong glyphs, but the tests still passed due to the tolerance. We still have a handful of tests that need a tolerance (particularly on windows iirc), but overall we should not go back to having non-zero tolerances as a standard practice.
The tests running multiple times is probably the png / svg / pdf tests running in sequence (only the png test uses our bindings to freetype, the others work by converting a vector graphic we have checked into the source against the output by rasterizing both with inkscape/ghostscript respectively).
Probably the best solution is to make the image comparison smart enough to skip the png tests if the freetype version is wrong.
attn @Kojoley
One thing that @astrofrog has with pytest-mpl is to source the baseline images from a URL which will enable use to keep versions of the pngs for multiple versions of freetype, but that would also be a fair amount of work to keep up.
sandrotosi commentedon Sep 18, 2016
thanks for looking into this! just a small note about the baseline retrieval via URLs: in debian we dont allow network access during build, so we would need some other solution to keep those images available during build
rathann commentedon Sep 19, 2016
Same here on Fedora 25/26. We're using freetype-2.6.5 and freetype-2.7, depending on the branch.
Kojoley commentedon Sep 19, 2016
It is definitely png/svg/pdf tests (prior to 903d118 commit multiple baselines/extensions per test was looking as the same test run multiple times in the log)
But freetype is still used for text bbox calculations (actually this is the cause of #6899).
This can be done and we already have
freetype_versiononimage_comparisondecorator for this, but I think it is better to tag baselines with freetype version rather than tag tests code.This will take really huge amount of handwork if we are going to create and support separate baselines for several freetype versions and I think we do not have enough resources for this in the current state. It can be less painful if we could reduce the number of the tests that requires text rendering (most of these tests are mathtext tests).
Alternative solution is to test text rendering without font anti-aliasing/hinting (recent freetype changes are mostly related to hinting).
rathann commentedon Sep 21, 2016
FWIW, relaxing the default tolerance to 0.306 makes all tests pass on x86_64/rawhide (F26) (except test_pngsuite and streamplot_colormap which override tolerance setting). Most differences are under 0.1. I'm considering carrying a patch doing that in Fedora for now as it's too much effort to adjust tolerance for each individual failing test (there are ~900).
matthew-brett commentedon Sep 21, 2016
I feel your pain, but I guess it's possible that some of these tests are showing important differences.
QuLogic commentedon Sep 21, 2016
@rathann Are you building from this copr or some other srpm?
sandrotosi commentedon Sep 21, 2016
@matthew-brett well yeah the fact the tests doesnt pass means there is a substantial difference, but do we have any way to assess what that is? i mean something better than: look at the 2 images and spot the differences?
tacaswell commentedon Sep 21, 2016
@sandrotosi The default tolerance is 0 meaning we get identical RGBA arrays out.
We spent a while thinking about a better way to evaluate images being 'close but not wrong', but could not come up with anything better. The problem seems to boil down to how to tell a character shifted by a pixel or two from a different character being there.
rathann commentedon Sep 22, 2016
@QuLogic I've been doing local mock builds for the last couple of days, but I've just uploaded my latest SRPM there.
QuLogic commentedon Oct 9, 2016
I tried this out on a 32-bit mock for rawhide. Indeed, most of the failures are fairly minor text changes. Strangely, more tests fail on 64-bit, but I've not investigated it.
There are some things that might be just a loss of precision somewhere, e.g., in
test_colorbar/cbar_locationing, the second blue is expected to be#0081ff, but is#0080ff.matthew-brett commentedon Jan 23, 2017
@sandrotosi - did you find a solution for these? We have the same situation over at the 32-bit wheel builds, noted at #6604, as well as intermittent segfaults noted at #6620. At some point we've got to decide whether to drop the 32-bit builds.
2 remaining items
matthew-brett commentedon Jan 24, 2017
Over at #7757 (comment) @dopplershift suggested making the result images into a package. I wonder whether we could use that mechanism to provide a different set of test images for the different architectures, via wheels?
astrofrog commentedon Jan 26, 2017
@tacaswell @matthew-brett to some extent, your two suggestions require the same amount of work. I think it makes more sense to do the first (remote images) because you need to be able to keep reference images for all versions around, so a package might start to become quite large whereas with remote reference images, you just download what you need. I'm not convinced maintaining this would be a huge amount of work, since I think a lot of it could be automated?
sandrotosi commentedon Jan 26, 2017
fetching remote files during build is unacceptable for debian, so we would not be able to use this solution.
i think the main problem is: how can you define if an image is good for a given distribution. in the case of debian, we develop in an ever-changing distribution (unstable) which evolves over a period of time of ~2 years - would generating a snapshot a day work? what about reproducibility? we take snapshots of the upstream projects (that would include also the pngs-for-debian) and release that, it means at a later time if you try to rebuild the same packages that has no diffs in the past, you might get new ones as some other libraries have change in the meantime.
(this might sound stupid, but it might give more ideas) if the main culprit is freetype (so i guess texts) should we try to have images-without-text with a lower tolerance thresholds and images-with-text with a higher threshold? or even "cutting" the image to grab the border (usually with text, like titles, ticks labels, etc) and compare it with a higher threshold that the central part of the image (usually graphs)?
tacaswell commentedon Oct 23, 2017
We do already remove text from many images (that is the default behavior of the image comparison tests!)
We used to just use a threshold by default, but we were outputting mathtext images with the wrong symbols that were still passing the tests.
@sandrotosi I suspect it would end up looking like a bunch of (versioned) tarballs of
test_images-MPLVER-FTVER.tar.gzso I think debian would just have a 'mpl-test-data' package that just is those images put into the correct place in the file system that is a build-time dependency of Matplotlib.Punting to 2.2 as we are not going to do anything this drastic in a micro release.
sandrotosi commentedon May 18, 2018
JFYI i decided in debian to not make tests fail if the images are not close: this way we can catch actual failures that would otherwise pass un-noticed in the storm of ImageComparisonFailure failures
QuLogic commentedon Sep 24, 2020
I'm going to close this as 2.0.0 is unsupported, and you have #11324 about more recent versions.