Repository navigation
Closed
Labels
3.10 (EOL)end of lifeend of lifestdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancementA feature request or enhancement
Description
| BPO | 38938 |
|---|---|
| Nosy | @tim-one, @rhettinger, @serhiy-storchaka, @bharel, @DimitrisJim, @bbayles, @sweeneyde |
| PRs | |
| Files |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = 'https://github.com/rhettinger'
closed_at = None
created_at = <Date 2019-11-29.02:44:20.990>
labels = ['type-feature', 'library', '3.10']
title = 'Possible performance improvement for heapq.merge()'
updated_at = <Date 2022-01-19.20:29:35.667>
user = 'https://github.com/sweeneyde'bugs.python.org fields:
activity = <Date 2022-01-19.20:29:35.667>
actor = 'bar.harel'
assignee = 'rhettinger'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2019-11-29.02:44:20.990>
creator = 'Dennis Sweeney'
dependencies = []
files = ['49160', '49164', '49166', '49167', '49170', '49180', '49184', '49198']
hgrepos = []
issue_num = 38938
keywords = ['patch']
message_count = 23.0
messages = ['357631', '357632', '357664', '357665', '358968', '364160', '364198', '364201', '364205', '364260', '364279', '364297', '368934', '368989', '369012', '369064', '369115', '369145', '369198', '369559', '369667', '370176', '370413']
nosy_count = 7.0
nosy_names = ['tim.peters', 'rhettinger', 'serhiy.storchaka', 'bar.harel', 'Jim Fasarakis-Hilliard', 'bbayles', 'Dennis Sweeney']
pr_nums = ['17422', '17729', '18427', '20550']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue38938'
versions = ['Python 3.10']Metadata
Metadata
Assignees
Labels
3.10 (EOL)end of lifeend of lifestdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancementA feature request or enhancement
Activity
sweeneyde commentedon Nov 29, 2019
Although the implementation of the heapq.merge function uses an underlying heap structure, its behavior centers on iterators. For this reason, I believe there should either be an alias to this function in the itertools module or at least a recipe in the itertools docs describing the use of heapq.merge.
Furthermore, I have an implementation (attached) of heapq.merge that is twice as fast for two iterables (as used in mergesort and other common cases), and is faster for up to 6 or 7 iterables. I think it would be nice to special-case this for small numbers of iterables for this significant speedup.
sweeneyde commentedon Nov 29, 2019
The following seems like it is a short, readable recipe for itertools.
sweeneyde commentedon Nov 30, 2019
Disregard merge_recipe.py: it would skip over a value that had already been retrieved from the iterator when the loop finished.
rhettinger commentedon Nov 30, 2019
Several thoughts:
heapq.merge() could have plausibly been in the itertools module instead of the heapq module. However, since it already has a home, there is no reason to move it or to create duplication with an intermodule dependency. We certainly don't have a rule that the itertools module must house everything accepts an iterator as an input and returns an iterator in the output.
The iter_merge() recipe is a bit sprawling and IMHO not a useful teaching aid (a primary goal of the itertools recipe), so it doesn't have much documentation value. Instead, consider submitting this to the "more-itertools" project which is referenced by the docs. There it would likely be used directly rather than as a recipe.
Over the next few days, I'll take a closer look at the binary tree approach versus a heap approach. I like how it intrinsically avoids the overhead of the 4-tuple while maintaining order, but I want to test how well it does with slow comparison keys (i.e. does the total number of comparisons get worse). Am not sure whether the tree needs to be rebalanced as some of the input streams become exhausted. I have a vague recollection that Knuth's TAOCP deemed the heap approach to be algorithmically superior, but need to dig out my books to be verify that.
We could just build-out the current heapq.merge() code to include a special case for only two input iterables. That would speed-up a common case by avoiding the overhead of tracking a 4-tuple for each element. If you want to submit a PR for that, I would be happy to take a close look and verify the timings.
It might be interesting to time the pure python variants against "sorted(chain(a, b, c, d, e))". It isn't quite an apples-to-apples comparison because the latter pulls all the data in memory and it doesn't the runs pre-segregated, but it might give a sense of how well merge() could do if we decided to go gonzo on optimizations. Until now, we've been satisfied with letting the heap structure minimize the number of comparisons and there hasn't been a focus on reducing the constant factor overhead.
[-]Iterable merge performance and inclusion in itertools[/-][+]Possible performance improvement for heaqq.merge()[/+]sweeneyde commentedon Dec 28, 2019
PR 17729 is a C implementation of a non-recursive "flattening" of the the recursive-lazy-mergesort algorithm into a tournament whose state is a tree of losers of comparisons.
serhiy-storchaka commentedon Mar 14, 2020
This is a large change. And making heaqq.merge a class looks unrelated to the declared goal. I afraid it may add significant overhead.
Since this is an optimization, could you please provide any benchmarks which we can use to check the performance boost?
7 remaining items
rhettinger commentedon May 16, 2020
The nested generators approach looks promising. I hope you keep working on that :-)
I'm not too keen on PR 18427 because it is a massive explosion in code volume and complexity.
With some continued effort, I expect we'll get to something much simpler and more maintainable. The existing code already performs well for typical applications, so there is no need to "go gonzo" and throw a massive wall of code at the problem. The ensuing maintenance costs would be too high.
To save time and effort, please hold-off on a C implementation until we've agreed to a pure python version of the algorithm.
For timings, try using strings, tuples, or dataclass instances. Timing integer inputs isn't representative of how merge() is used and it tends to exaggerate improvements. For interesting merges, the dominant cost is the comparison step.
By switching to a binary tree arrangement, the payoff we're aiming for is to avoid the double comparison in the tuple inequality logic — "('a',1)<('b',2)" tests both 'a'=='b' and 'a'<='b'. I suggest aiming for the simplest possible code that avoids the double test.
sweeneyde commentedon May 16, 2020
The attached recursive_merge.py should be much less ugly and still somewhat performant.
It should be the same algorithm as that PR, just written recursively rather than iteratively.
I got some text files from http://www.gwicks.net/dictionaries.htm and tried merging them line-by-line:
py -3.9 -m pyperf timeit -s "from heapq import merge; from collections import deque" "deque(merge(open('english.txt'), open('dutch.txt'), open('french.txt'), open('german.txt'), open('italian.txt')), maxlen=0)"
py -3.9 -m pyperf timeit -s "from recursive_merge import merge; from collections import deque" "deque(merge(open('english.txt'), open('dutch.txt'), open('french.txt'), open('german.txt'), open('italian.txt')), maxlen=0)"
Perhaps that's a more real-world benchmark.
rhettinger commentedon May 16, 2020
For comparison, I'm attaching an iterative version that manipulates the tree nodes with all local variables. Try it out and see what you think.
sweeneyde commentedon May 17, 2020
It seems to me that the code sprawl mostly comes from the separate handling of the four keyed/unkeyed and forward/reverse cases, which as far as I can tell requires a branch in the innermost loop if not unrolled into separate cases. I think recursive_merge.py is about as concise as possible while still efficiently handling all four cases.
Is there any issue with recursion in this application? Even if there are 2**64-1 iterables, this would only mean 64 nested next() calls, which should be within the stack on most machines, no?
I did the following benchmark:
py -3.9 -m pyperf timeit -s "from [FILENAME] import merge; from collections import deque" "deque(merge(open('english.txt'), open('dutch.txt'), open('french.txt'), open('german.txt'), open('italian.txt')), maxlen=0)"
I can look at some more diverse benchmarks (types, iterable length imbalance, lengths of an iterator's win-streak, etc.) soon.
rhettinger commentedon May 17, 2020
Am leaning toward the iterative approach in new_merge.py because it most directly implements the core concept of storing the data in a binary tree.
Merits: no recursion, no nested generators, avoids the clutter of left-empty and right-empty checks, easy to understand, simple and fast loops, all local variables or instance variables, no calls to memory allocator after then initial tree construction, runs well on PyPy, and easily Cythonized.
sweeneyde commentedon May 18, 2020
I mostly like new_merge.py too, especially the dynamic reduction of the tree.
However, it looks like
list(merge([2],[1],[1]))currently fails, and I think what's missing is the following in the sibling-promotion:+ if sibling.left is not None:
+ sibling.left.parent = sibling.right.parent = parent
Also for what it's worth, I think both appearances of "c1 if c1.value < c2.value else c2" should become "c2 if c2.value < c1.value else c1" for stability.
I've attached winners.py, which is similar to new_merge.py, but replaces the "while node.left is not None: node = node.source" traversal with a single "node = node.leaf" call and adds a fast yield-from for the last iterator. I don't think it's much more complex either.
sweeneyde commentedon May 22, 2020
key_and_reverse.py employs the same strategy as winners.py, but uses lists as the nodes of the tree rather than using Node instances. It also eliminates the recursion of treeify, and adds (with neither much of a performance hit nor much code duplication) support for
key=some_funcand/orreverse=Truekeyword arguments, so it now passes the test_heapq suite.rhettinger commentedon May 22, 2020
Thanks. I'll look at this shortly. We're getting much closer.
Just so that I don't lose other work I've done, am uploading new_merge2.py with in-line iterative tree construction of the initial tree.
sweeneyde commentedon May 28, 2020
less_movement.py is my favorite so far. It still handles key and reverse,
but using instance attributes instead of the list indices I tried before.
It does this by only advancing the "key" and "leaf" attributes up toward
the root (where the key is just the value if key is None), while the value
is stored only in the leaf. Since the "value" attribute is no longer used
except for at the leaves, we can pack a leaf's value into its left slot
and reduce the number of slots.
This seems to run very well on pypy and it looks like it would be pretty
receptive to a c implementation, which I would be happy to work on.
[-]Possible performance improvement for heaqq.merge()[/-][+]Possible performance improvement for heapq.merge()[/+]sweeneyde commentedon May 31, 2020
PR 20550 uses a linked structure like what we've been talking about.