Repository navigation
Performance degradation in 2.1.3 #605
Description
Hi,
The recent commit f1a82e4 has caused a significant performance
degradation in the use of GitPython. As an example, one unit
test that we have which performs operations on about 42
repositories normally runs in 6.7 seconds, but with the two new
gc.collect() calls in f1a82e4 it now takes 24.2 seconds.
The gitdb.util.mman.collect() call does not seem to be expensive,
only gc.collect().
It's worth noting that most objects in python are freed by use of
reference counting, while gc.collect() invokes the garbage
collector, which is only needed to free objects with reference
cycles. Normally, the garbage collector runs periodically and
does not need to be explicitly invoked.
Since the garbage collector should only be effective on objects
which no longer have references from active frames, it's not
clear why these two calls to gc.collect() are required. It would
be nice to know if there are any other ways to address what they
are attempting to fix. If they truly are required in some
circumstances, it would be good to know what those are and if
there are cases where we do not need to invoke the expense. And
if they are not required, we should remove them.
Thanks!
Activity
Cap GitPython at 2.1.1 due to performance degradation
Cap GitPython at 2.1.1 due to performance degradation
Cap GitPython at 2.1.1 due to performance degradation
Updated openstack/openstack
Cap GitPython at 2.1.1 due to performance degradation
zuul: pin GitPython <2.1.2
ankostis commentedon Mar 17, 2017
Actually all objects were freed on ref-counting, till Python-3.5; since 3.5 that is not the case.
They are freed later, by a GC thread (I guess).
You rarely get to notice that in Linux - you can still delete files that are open.
But you bump into this every time you create a temp-file/folder on Windows, and then try to clean it up, because dead but non-GCed objects still prevent files from deleting.
Now, the "correct" way is to use WeakRef finalizers.
You can see all efforts into these issues by searching for
tag.leaks.Any improvement would be gladly accepted.
Byron commentedon Apr 9, 2017
Indeed, this degradation was introduced in order to more promptly release file handles on windows. A possible fix could be to just enforce
gc.collect()on windows, even though it appears like a hack on top of a hack.@jeblair Would that help?
@ankostis Do you think doing so would be acceptable?
ankostis commentedon Apr 12, 2017
I expect all TCs to remain the same, it should be fine.
zuul: pin GitPython <2.1.2
zuul: pin GitPython <2.1.2