Skip to content

bpo-34151: Improve performance of some list operations #8332

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zhangyangyu merged 7 commits into from
Aug 11, 2018
Merged

bpo-34151: Improve performance of some list operations #8332

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
efce592
bpo-34151: Improved performance of some list operations by using mall…
sir-sigurd Jul 18, 2018
b5da6db
Made requested changes.
sir-sigurd Jul 19, 2018
50f71bd
bugfix: set size for single item list repeat.
sir-sigurd Jul 19, 2018
c38c50c
Update listobject.c
methane Jul 25, 2018
604ae8b
Update listobject.c
methane Jul 25, 2018
5a6de07
Added NEWS entry.
sir-sigurd Jul 25, 2018
3d685b1
Added assertion.
sir-sigurd Aug 1, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Core and Builtins/2018-07-25-20-26-02.bpo-34151.Q2pK9Q.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Performance of list concatenation, repetition and slicing operations is
slightly improved. Patch by Sergey Fedoseev.
51 changes: 36 additions & 15 deletions Objects/listobject.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,23 @@ PyList_New(Py_ssize_t size)
return (PyObject *) op;
}

static PyObject *

@vstinner vstinner Jul 31, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add a short comment to explain when you consider that this function is more appropriate than PyList_New()? I understand that it should be preferred when the caller will always immediately fill all allocated list items

list_new_prealloc(Py_ssize_t size)
{
PyListObject *op = (PyListObject *) PyList_New(0);
if (size == 0 || op == NULL) {
return (PyObject *) op;
}
assert(op->ob_item == NULL);
op->ob_item = PyMem_New(PyObject *, size);
if (op->ob_item == NULL) {
Py_DECREF(op);
return PyErr_NoMemory();
}

@vstinner vstinner Jul 31, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From this point, it is possible that a GC collection will access the list. It would be safer to not call _PyObject_GC_TRACK(op) before the list is fully initialized. I suggest to modify list_new_prealloc() to avoid _PyObject_GC_TRACK(op), but call in the caller.

Maybe you can rewrite the code so PyList_New() calls list_new_prealloc(), and not the opposite?

@sir-sigurd sir-sigurd Jul 31, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you elaborate on what do you mean exactly by not "fully initialized" here?

I think I could rewrite the whole patch using existing list_resize() function (I didn't it initially because unlike PyList_New() it preallocates more space than needed), since its usage doesn't raise such questions.

@vstinner vstinner Jul 31, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

list_dealloc() will crash if it's called on the list just after list_new_prealloc(), since items are uninitialized, whereas list_dealloc() calls Py_XDECREF() on each item. But the main issue is list_traverse(). If the garbage collection is triggered, list_traverse() will dereference uninitialized object pointers and so crash.

@methane methane Jul 31, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

list_traverse doesn’t touch preallocated area.
Unless Py_SIZE(list) is updated before filling objects, there are no dangers, I suppose.

op->allocated = size;
return (PyObject *) op;
}

Py_ssize_t
PyList_Size(PyObject *op)
{
Expand Down Expand Up @@ -438,7 +455,7 @@ list_slice(PyListObject *a, Py_ssize_t ilow, Py_ssize_t ihigh)
else if (ihigh > Py_SIZE(a))
ihigh = Py_SIZE(a);
len = ihigh - ilow;
np = (PyListObject *) PyList_New(len);
np = (PyListObject *) list_new_prealloc(len);
if (np == NULL)
return NULL;

Expand All @@ -449,6 +466,7 @@ list_slice(PyListObject *a, Py_ssize_t ilow, Py_ssize_t ihigh)
Py_INCREF(v);
dest[i] = v;
}
Py_SIZE(np) = len;
return (PyObject *)np;
}

Expand Down Expand Up @@ -479,7 +497,7 @@ list_concat(PyListObject *a, PyObject *bb)
if (Py_SIZE(a) > PY_SSIZE_T_MAX - Py_SIZE(b))
return PyErr_NoMemory();
size = Py_SIZE(a) + Py_SIZE(b);
np = (PyListObject *) PyList_New(size);
np = (PyListObject *) list_new_prealloc(size);
if (np == NULL) {
return NULL;
}
Expand All @@ -497,6 +515,7 @@ list_concat(PyListObject *a, PyObject *bb)
Py_INCREF(v);
dest[i] = v;
}
Py_SIZE(np) = size;
return (PyObject *)np;
#undef b
}
Expand All @@ -516,28 +535,30 @@ list_repeat(PyListObject *a, Py_ssize_t n)
size = Py_SIZE(a) * n;
if (size == 0)
return PyList_New(0);
np = (PyListObject *) PyList_New(size);
np = (PyListObject *) list_new_prealloc(size);
if (np == NULL)
return NULL;

items = np->ob_item;
if (Py_SIZE(a) == 1) {
items = np->ob_item;
elem = a->ob_item[0];
for (i = 0; i < n; i++) {
items[i] = elem;
Py_INCREF(elem);
}
return (PyObject *) np;
}
p = np->ob_item;
items = a->ob_item;
for (i = 0; i < n; i++) {
for (j = 0; j < Py_SIZE(a); j++) {
*p = items[j];
Py_INCREF(*p);
p++;
}
else {
p = np->ob_item;
items = a->ob_item;
for (i = 0; i < n; i++) {
for (j = 0; j < Py_SIZE(a); j++) {
*p = items[j];
Py_INCREF(*p);
p++;
}
}
}
Py_SIZE(np) = size;
return (PyObject *) np;
}

Expand Down Expand Up @@ -2738,7 +2759,7 @@ list_subscript(PyListObject* self, PyObject* item)
return list_slice(self, start, stop);
}
else {
result = PyList_New(slicelength);
result = list_new_prealloc(slicelength);
if (!result) return NULL;

src = self->ob_item;
Expand All @@ -2749,7 +2770,7 @@ list_subscript(PyListObject* self, PyObject* item)
Py_INCREF(it);
dest[i] = it;
}

Py_SIZE(result) = slicelength;
return result;
}
}
Expand Down