GH-113225: Speed up pathlib.Path.walk(top_down=False)#113693
Merged
Conversation
Use `_make_child_entry()` rather than `_make_child_relpath()` to retrieve path objects for directories to visit. This saves the allocation of one path object per directory in user subclasses of `PathBase`, and avoids a second loop. This trick does not apply when walking top-down, because users can affect the walk by modifying *dirnames* in-place. A side effect of this change is that, in bottom-up mode, subdirectories of each directory are visited in reverse order, and that this order doesn't match that of the names in *dirnames*. I suspect this is fine as the order is arbitrary anyway.
barneygale
added
performance
barneygale
changed the title
pathlib._abc.PathBase.walk(top_down=False)pathlib.Path.walk(top_down=False)
Contributor
Author
|
About 2% faster: $ ./python -m timeit -s "from pathlib import Path" "list(Path().walk(top_down=False))"
10 loops, best of 5: 27.5 msec per loop # before
10 loops, best of 5: 27 msec per loop # after
$ ./python -m timeit -s "from pathlib import Path" "list(Path.cwd().walk(top_down=False))"
10 loops, best of 5: 28.2 msec per loop # before
10 loops, best of 5: 27.5 msec per loop # after |
barneygale
enabled auto-merge (squash)
January 20, 2024 02:47
kulikjak
pushed a commit
to kulikjak/cpython
that referenced
this pull request
Jan 22, 2024
…#113693) Use `_make_child_entry()` rather than `_make_child_relpath()` to retrieve path objects for directories to visit. This saves the allocation of one path object per directory in user subclasses of `PathBase`, and avoids a second loop. This trick does not apply when walking top-down, because users can affect the walk by modifying *dirnames* in-place. A side effect of this change is that, in bottom-up mode, subdirectories of each directory are visited in reverse order, and that this order doesn't match that of the names in *dirnames*. I suspect this is fine as the order is arbitrary anyway.
aisk
pushed a commit
to aisk/cpython
that referenced
this pull request
Feb 11, 2024
…#113693) Use `_make_child_entry()` rather than `_make_child_relpath()` to retrieve path objects for directories to visit. This saves the allocation of one path object per directory in user subclasses of `PathBase`, and avoids a second loop. This trick does not apply when walking top-down, because users can affect the walk by modifying *dirnames* in-place. A side effect of this change is that, in bottom-up mode, subdirectories of each directory are visited in reverse order, and that this order doesn't match that of the names in *dirnames*. I suspect this is fine as the order is arbitrary anyway.
Glyphack
pushed a commit
to Glyphack/cpython
that referenced
this pull request
Sep 2, 2024
…#113693) Use `_make_child_entry()` rather than `_make_child_relpath()` to retrieve path objects for directories to visit. This saves the allocation of one path object per directory in user subclasses of `PathBase`, and avoids a second loop. This trick does not apply when walking top-down, because users can affect the walk by modifying *dirnames* in-place. A side effect of this change is that, in bottom-up mode, subdirectories of each directory are visited in reverse order, and that this order doesn't match that of the names in *dirnames*. I suspect this is fine as the order is arbitrary anyway.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use
_make_child_entry()rather than_make_child_relpath()to retrieve path objects for directories to visit.In user subclasses of
PathBase, this saves the allocation of one path object per directory. InPathit saves a small amount path string generation.This trick does not apply when walking top-down, because users can affect the walk by modifying dirnames in-place.
A side effect of this change is that, in bottom-up mode, subdirectories of each directory are visited in reverse order, and that this order doesn't match that of the names in dirnames. I suspect this is fine as the order is arbitrary anyway.
pathlib.Path.iterdir()and friends by usingos.DirEntry.path#113225