I'd give this function a name that makes it clear that the memory will not be initialised. Traditionally, [builtin]_new is the name of the tp_new() slot function, which this isn't.

something like: list_new_uninitialized()? Or maybe list_new_prealloc()?

I prefer list_new_uninitialized over list_new_prealloc. prealloc makes me think the function is to allocate more memory than needed now for later usage but it is not.

something like: list_new_uninitialized()? Or maybe list_new_prealloc()?

You should check if size*sizeof() overflow or not.

You might allocate ob_item before creating op, since I dislike calling Py_DECREF() on a just created object (is it fully initialized or not?).

Especially, in this case deallocating the list will decref uninitialized list items...

@vstinner

You might allocate ob_item before creating op, since I dislike calling Py_DECREF() on a just created object (is it fully initialized or not?).

Could you explain the difference?

Especially, in this case deallocating the list will decref uninitialized list items

No, it won't. Either the PyMem_Malloc() fails, then ob_item is NULL and the list size is 0, which is consistent and can be cleaned up safely. Or, if PyMem_Malloc() succeeds, then ob_item points to the item array, which will be filled up right afterwards.

I think this code is ok as it is.

I'm not sure about this line. I would feel more confortable if it's done by the caller to always keep the list object consistent. For example, the caller could only set the size once the list is fully initialized. Otherwise, Py_DECREF(op) can crash on a half initialized list.

I partly agree with this, but also note that any subsequent list array filling will necessarily incref the items along the way, so after the item array allocation and while the list isn't fully filled yet, it will necessarily be in an inconsistent and unsafe state that cannot simply be cleaned up via Py_DECREF(op). Thus the careful selection of list operations to which this is applied: only those that do not risk raising exceptions within this state transition.

Letting the caller do the size adjustment does not save us here. But I agree that it slightly reduces the chance of programmer errors. So, +1. And that means we should also have a comment at this function that states this contract.

I have the same feeling here with @scoder . Putting the size setting on the caller's side doesn't save us. If there is any exception raising cases inside the content setting logic, it's still painful since you have to maintain the current size. And the new helper function doesn't do all metadata setting now. I have not better solution now but I'd suggest add this note to the comment of the new helper function no matter which way it is in.

Please add a comment why you don't use calloc() here, with a reference to the bpo number.

Can you please add a short comment to explain when you consider that this function is more appropriate than PyList_New()? I understand that it should be preferred when the caller will always immediately fill all allocated list items

From this point, it is possible that a GC collection will access the list. It would be safer to not call _PyObject_GC_TRACK(op) before the list is fully initialized. I suggest to modify list_new_prealloc() to avoid _PyObject_GC_TRACK(op), but call in the caller.

Maybe you can rewrite the code so PyList_New() calls list_new_prealloc(), and not the opposite?

Could you elaborate on what do you mean exactly by not "fully initialized" here?

I think I could rewrite the whole patch using existing list_resize() function (I didn't it initially because unlike PyList_New() it preallocates more space than needed), since its usage doesn't raise such questions.

list_dealloc() will crash if it's called on the list just after list_new_prealloc(), since items are uninitialized, whereas list_dealloc() calls Py_XDECREF() on each item. But the main issue is list_traverse(). If the garbage collection is triggered, list_traverse() will dereference uninitialized object pointers and so crash.

list_traverse doesn’t touch preallocated area.
Unless Py_SIZE(list) is updated before filling objects, there are no dangers, I suppose.

list_dealloc() will crash if it's called on the list just after list_new_prealloc(), since items are uninitialized, whereas list_dealloc() calls Py_XDECREF() on each item. But the main issue is list_traverse(). If the garbage collection is triggered, list_traverse() will dereference uninitialized object pointers and so crash.