Repository navigation
enumerate's are slow #121380
Description
Performance Improvement
Using enumerate(...) in for loops are slow and there are at least +250 enumerate's in CPython
Let's take for example this for loop with enumerate:
Lines 116 to 118 in 5f660e8
If you remove enumerate, this method will run 5% to 7% faster:
i = 0
for c in s1:
if c != s2[i]:
return s1[:i]
i += 1Complete performance test script, basically I copied twice the original commonprefix(m) and then I modified the last one:
import os
import time
# From file CPython/Lib/genericpath.py
# Return the longest prefix of all list elements.
def old_commonprefix(m):
"Given a list of pathnames, returns the longest common leading component"
if not m: return ''
# Some people pass in a list of pathname parts to operate in an OS-agnostic
# fashion; don't try to translate in that case as that's an abuse of the
# API and they are already doing what they need to be OS-agnostic and so
# they most likely won't be using an os.PathLike object in the sublists.
if not isinstance(m[0], (list, tuple)):
m = tuple(map(os.fspath, m))
s1 = min(m)
s2 = max(m)
for i, c in enumerate(s1):
if c != s2[i]:
return s1[:i]
return s1
# Return the longest prefix of all list elements.
def commonprefix(m):
"Given a list of pathnames, returns the longest common leading component"
if not m: return ''
# Some people pass in a list of pathname parts to operate in an OS-agnostic
# fashion; don't try to translate in that case as that's an abuse of the
# API and they are already doing what they need to be OS-agnostic and so
# they most likely won't be using an os.PathLike object in the sublists.
if not isinstance(m[0], (list, tuple)):
m = tuple(map(os.fspath, m))
s1 = min(m)
s2 = max(m)
i = 0
for c in s1:
if c != s2[i]:
return s1[:i]
i += 1
return s1
def perfomance(func):
start = time.time()
for _ in range(1000000):
func(["/usr/lib", "/usr/local/lib", "/usr/local/bin", "/usr/bin", "/usr/local/share"])
end = time.time()
return end - start
old_perfomance = perfomance(old_commonprefix)
new_perfomance = perfomance(commonprefix)
perc = (old_perfomance / new_perfomance) * 100
print(f"with enumerate: {old_perfomance} secs")
print(f"without enumerate: {new_perfomance} secs")
print(f"New version is {perc:.2f}% faster")You will get this performance improvement:
with enumerate: 0.8863015174865723 secs
without enumerate: 0.8414063453674316 secs
New version is 105.34% faster
Running this commands, you can count and find all for loops with enumerate in /Lib for example
$ grep -r "in enumerate" ./Lib | wc -l
$ grep -rl "in enumerate" ./LibIf this is useful for CPython, I can create specific issues+PR replacing enumerate(...)'s.
Has this already been discussed elsewhere?
No response given
Links to previous discussion of this feature:
No response
Activity
nineteendo commentedon Jul 4, 2024
This sounds like an issue for https://github.com/faster-cpython/ideas. Someone clever might optimise
enumerate().gaogaotiantian commentedon Jul 4, 2024
Yes
enumeratewill introduce some overhead. In your example, the one-time overhead to create theenumerateobject is also significant. If you increase the common prefix, the diff would be ~2% to 3%. Yes it's "slower" than incrementing an index by yourself, but sometimes that's the cost of making the code look good. Basically under the hood, a new tuple is created and unpacked.An important thing to notice is that your loop is basically an "empty loop", which makes the cost of the loop itself more significant. In more common cases, where the loop has more code to execute, the impact will be smaller, possibly negligible.
No, you should not replace all the
enumerates in CPython. Like I mentioned above, it's not that slow, in many cases you won't be able to get the difference. More importantly, performance is not the only golden rule for CPython. Code readability is critical for maintenance and that's the reason we haveenumerate.rhettinger commentedon Jul 5, 2024
The timings bounce around from version to version:
Timing code:
As noted above, readability is the primary goal. Marking as closed.