Repository navigation
PDF file generation is not deterministic - results in different outputs on the same input #6317
Description
Suppose you want to generate a pdf file with matplotlib and save it.
genfigure.py:
import matplotlib.pyplot as plt
import sys
plt.plot([0, 1], [0, 1])
plt.savefig(sys.argv[1])
Run the script from the command line
$ python genfigure.py 1.pdf
$ python genfigure.py 2.pdf
Given that we are saving the same figure, we would expect the output to be the same. However, after looking at the file hashes, they appear to be different. In my particular case:
$ md5sum 1.pdf 2.pdf
e54cdbd65a6baaa5152d90743d800039 1.pdf
4b0ac2a7c046c4813114c63f3c4d27e7 2.pdf
On the other hand, no such issue exists when saving png files.
$ python genfigure.py 1.png
$ python genfigure.py 2.png
The two png files are exactly the same
$ md5sum 1.png 2.png
5d22187827337cd9262ee248550fab6f 1.png
5d22187827337cd9262ee248550fab6f 2.png
It appears that pdf saving has some source of non-determinism.
Is there a way to ensure that saving the same figure multiple times, results in exactly the same pdf file?
Reacted by Danny Hermes, Nicolai Weitkemper and blaiseli
Activity
drufat commentedon Apr 19, 2016
Looking inside the generated pdf file, it appears that matplotlib inserts a creation date
but that is not the only difference. There appear to be other differences as well.
WeatherGod commentedon Apr 19, 2016
Which version of matplotlib are you using? I think some work has been done
recently to make sure that outputs are deterministic.
On Mon, Apr 18, 2016 at 9:10 PM, Dzhelil Rufat notifications@github.com
wrote:
drufat commentedon Apr 19, 2016
drufat commentedon Apr 19, 2016
It seems like the issue is in the
backends/backend_pdf.pyfile. It is using a lot ofdicts when it should be usingOrderedDictin order to ensure that iterations are executed deterministically.WeatherGod commentedon Apr 19, 2016
matplotlib version 1.5.1 still supports python 2.6, so ordered dicts can't
be used yet. But, version 2.0 will drop support for python 2.6. I did a
little bit of digging, and the deterministic output work that I was
thinking of was for SVG output, not PDF/PS output:
#4434
However, the solution there actually might be applicable here too.
On Mon, Apr 18, 2016 at 10:11 PM, Dzhelil Rufat notifications@github.com
wrote:
drufat commentedon Apr 19, 2016
Even the svg backend is not completely deterministic
WeatherGod commentedon Apr 19, 2016
now that is weird... I am fairly certain we added unit tests to check that
the SVG output remains deterministic...
On Mon, Apr 18, 2016 at 10:40 PM, Dzhelil Rufat notifications@github.com
wrote:
WeatherGod commentedon Apr 19, 2016
Ah, which version of matplotlib are you using? This might only be on master
right now.
On Tue, Apr 19, 2016 at 9:11 AM, Benjamin Root ben.v.root@gmail.com wrote:
WeatherGod commentedon Apr 19, 2016
and, reading it further, it looks like by default, it will not be
completely deterministic. One would have to set the svg.hashsalt parameter.
But, the use of OrderedDicts is in master, so I would suggest trying out
master to see if the problem still persists for PDFs.
On Tue, Apr 19, 2016 at 9:14 AM, Benjamin Root ben.v.root@gmail.com wrote:
efiring commentedon Apr 19, 2016
I don't see any instances of OrderedDict in backend_pdf.py in master. @drufat, would you like to submit a PR for this? @jkseppan, do you see any problems with this approach?
jkseppan commentedon Apr 19, 2016
I don't see why you'd need to use OrderedDict. You could just change the dict branch of
pdfReprto iterate the keys in sorted order. Then there's at leastCreationDatein the information dict, and possibly other sources of nondeterminism.May I ask why you'd want to make pdf files be exactly the same?
drufat commentedon Apr 19, 2016
@jkseppan There are many reasons why determinism is important - functions are generally expected to return the same output if given the same input - it makes debugging much easier. If changes in the saved plot files are introduced by genuine changes in the plot data, rather than spurious change introduced by how matplotlib saves the file, one can easily detect any mistakes. Suppose you are working on a large code base, and you are saving some function plots. When you make changes in one area of the code you want to ensure you have not broken other parts, and you do that by comparing the generated plot files, and ensuring they have not changed. Since you cannot visually inspect every single one of them, it is easier to check that they have not changed by comparing their actual content or md5sum. With the current nondeterminism in pdf saving it is impossible to compare two pdf files.
Also I do not see the need to save metadata in the pdf files such as
CreationDate- what purpose could that possibly serve? When you callsinfromcmath, you do not expect it to return aComputationDateembedded in the result. Why should pdf generation be any different? For those that care about such information the file system already provides modification and creation times anyway.@efiring I will try to look at it later this weekend, but I am no expert in pdf generation.
jkseppan commentedon Apr 19, 2016
The matplotlib test suite checks that the pdf backend produces the same output as earlier, by rendering the output file into pixels and comparing the results. This works across versions and big changes at all levels of matplotlib.
A pdf file is fundamentally different from a png file or a matrix of numbers: it is an encoding of a drawing process into a language (essentially a very restricted form of PostScript), and a further rendering step is needed to produce an image on some output medium. There are always going to be many ways of encoding the process to create a given output, and the ordering of dicts at the bottom-most level is just the most obvious kind of variation. We can certainly sort the dictionaries repeatably and add an option to remove the creation date, but next you're likely to find that small changes in your script or small updates to matplotlib will cause changes in the ordering of higher-level objects, or coordinates that differ in the 10th decimal, or some similar changes that have no semantic meaning but will result in a different hash of the output file.
One more source of nondeterminism: font subsetting will add random tags in the font name in any pdf-generating program, so that when different subsets of the same font end up getting included in the same print job, you avoid the printer using a cached font that is missing some random characters. If the software that combines pdf files is written diligently, it should recurse through all included pdf files and alter any font names that do not refer to the same exact font subset, but using a random tag is a very simple way to avoid this problem even if the combining program fails to do this.
WeatherGod commentedon Apr 19, 2016
I am going to take it as a given that determinism certainly has its
benefits. We were already convinced as such to do it for SVG, so I don't
think we need to rehash that discussion for PDFs. The real question is how
much determinism is desired? Obviously, matplotlib has come up with its own
solution for unit testing, but it only serves our own purpose (checking the
rendered results) and really isn't sufficient for people who may be taking
advantage of PdfPages and other tools.
The goal for the determinism for SVG output was to make it easier to
compare subsequent runs, rather than for making it possible to compare
across different versions of matplotlib, or different major versions of the
generating script. We do not strive to ensure that the structure will
remain the same, which is why the default for SVGs is that the hashsalt is
a uuid4.
Now, there are plenty of reasons to include a creation date in the pdf
metadata. The creation date is conceptually different from the file system
timestamps. It is static across many file copies and transfers from one
system to another. It is just good data management practice to include such
metadata in your products. Of course, this can make differencing a bit
difficult, but at least it isn't a surprise.
On Tue, Apr 19, 2016 at 1:42 PM, Jouni K. Seppänen <notifications@github.com
dopplershift commentedon Apr 19, 2016
Determinism is also nice for auto-generated stuff that's going into git--it's nice to have things change only if they actually changed. I don't think this is a supreme goal, but there's nothing wrong with working towards it if there are simple changes to be made (I consider ordereddict to be simple).
As far as creation date goes, is there a way for the user to set that? If I was doing this as in some automated process, and the creation date is guaranteed to be in the metadata of the output file, I'd want some way to be able to control it. In this case I'd want the PDF to be the last modification time of the generating script, not just when I happened to run the script.
tacaswell commentedon Apr 20, 2016
This thread is getting a bit testier than strictly needed, we are all on the same side here 😄 I think the main contention is what 'the same' means. The meaning mpl has settled on (long ago) is 'looks the same' and the internal details of the vector backends is 'an implementation detail' and not part of the stable public API.
To summarize the svg discussion:
The compromise was to add the
svg.hashsaltrcparam to opt-into the byte-for-byte deterministic output, but we are only guaranteeing determinism run-to-run on the same commit of mpl (ex, the byte output is not part of the API, only the rasterized output is). We were not willing to make any guarantees about the stability of the pre-rendered output, that has never been part of what we consider the 'public' API.It seems reasonable to do the same for the pdf backend:
@dopplershift That comes back to the more general problem of passing backend-specific kwargs/meta-data through
savefig.I can see three positions on including the date:
@drufat If you want to use pdfs for testing regressions I strongly suggest you use the render-and-compare approach we use internally. For example, changing the z-order (or even just plot order) of two non-overlapping lines will change the pdf representation, but not change your output in any meaningful way.
I also would consider the fact that the png backend produces bit identical outputs a coincidence and not to be part of the public API. That hash value may change on a micro-version, may not be the same across platforms. Again, if you are using this for regressions, open the images and compare them.
jkseppan commentedon Apr 20, 2016
I still think it's simpler to iterate dicts in order than to replace them with OrderedDicts. The
pdfReprfunction is a chokepoint through which all page content passes, so it should be enough to change the iteration order there. I don't have time right now to test and submit a patch, but it should be a one- or two-line change.IMHO we should add separate options to set the creation date to a user-supplied value (or to remove it) and to disable font subsetting (the pdf spec requires the addition of tags for subsets, so not adding them makes the resulting file non-compliant), and any other sources of nondeterminism that we discover. I think it's likely that there are such sources that we may have missed so far.
drufat commentedon Apr 27, 2016
Using sorted instead of OrderedDict in your for loops could affect the time complexity. Sorting is an O(N log N) operation, whereas OrderedDict can be iterated in linear time O(N).
jkseppan commentedon Apr 27, 2016
But then you just move the sort operation to dict construction time instead of iteration time? Both happen once, sometimes with a few inserts in between. In any case, I will be very surprised if you can measure the performance difference in outputting a pdf document. The dicts typically have a few keys, a few tens at the most.
Output pdf dictionaries in deterministic order
Output pdf dictionaries in deterministic order
QuLogic commentedon Dec 9, 2016
Fixed by #6597.
dhermes commentedon Jan 18, 2018
For future visitors https://matplotlib.org/2.1.1/users/whats_new.html#reproducible-ps-pdf-and-svg-output is a helpful resource
seekstar commentedon May 11, 2025
It seems that now we should call savefig with
metadata={'CreationDate': None}instead ofcreationDate