Skip to content

generic and efficiency way to consume iterable object - #2284

Merged
youknowone merged 4 commits into
RustPython:masterfrom
qingshi163:dev4
Oct 21, 2020
Merged

youknowone merged 4 commits into
RustPython:masterfrom
qingshi163:dev4

Conversation

@qingshi163

Copy link
Copy Markdown
Contributor

No description provided.

Comment thread vm/src/pyobject.rs Outdated
Comment on lines +902 to +887
pub fn try_iterable_map<F, T, R>(
vm: &VirtualMachine,
obj: &PyObjectRef,
mut f: F,
) -> PyResult<Vec<R>>
where
T: TryFromObject,
F: FnMut(T) -> PyResult<R>,
{

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better name?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try_map_iter?

@coolreader18

Copy link
Copy Markdown
Member

Looks good! I think it would make sense to put this as a method on VirtualMachine, and then you could also define extract_elements in terms of this

@qingshi163

qingshi163 commented Oct 13, 2020 •

Copy link
Copy Markdown
Contributor Author

I think I stacked here, CPython allows create the bytes from an iterable object like a list, and inside the list you could put the object that has __index__(), but when this __index__() been called it allows the list to be modified!

This cause the problem if we map the list so the list is borrowed, and when __index__() try to modity the list we are in deadlock.

I think we may have to go back to extract all the elements from the list before try to convert the items. (CPython is doing like lazy-evaluation).

Or any idea that help?

@qingshi163

qingshi163 commented Oct 13, 2020 •

Copy link
Copy Markdown
Contributor Author

Or we need to do the map manually, so we clone the item out, free the lock, do the convert.
but this did not sound thread safe, is ok to go this way?
@coolreader18

@youknowone

youknowone commented Oct 15, 2020 •

Copy link
Copy Markdown
Member

When there is difference, following CPython way is almost the best.
Copying here doesn't seem to solve the compatibility problem regardless threading. If __index__ edits the content of the list, it will not affect the copied content.
So, I think tuple can take this way but list is not. right?

@qingshi163

Copy link
Copy Markdown
Contributor Author

I moved the mutable function from bytesinner to bytearray for better control of the borrowing.

Comment thread vm/src/vm.rs Outdated
Comment on lines +990 to +996
pub fn map_elements<F, T, R>(&self, obj: &PyObjectRef, mut f: F) -> PyResult<PyResult<Vec<R>>>
where
T: TryFromObject,
F: FnMut(T) -> PyResult<R>,
{
self.map_iterable_object(obj, |elem| f(T::try_from_object(self, elem)?))
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function not been use yet, maybe remove it?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally, yes. having unused code is a sort of cost. Do you have plan to use it in other patch?

@qingshi163

Copy link
Copy Markdown
Contributor Author

#2125 fixed

@coolreader18 coolreader18 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread vm/src/iterator.rs Outdated
{
let cap = length_hint(vm, iter_obj.clone())?.unwrap_or(0);
// TODO: fix extend to do this check (?), see test_extend in Lib/test/list_tests.py,
// https://github.com/python/cpython/blob/master/Objects/listobject.c#L934-L940

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

link to cpython master will be broken one day. link to specific tag will be better.

Comment thread vm/src/iterator.rs
let cap = length_hint(vm, iter_obj.clone())?.unwrap_or(0);
// TODO: fix extend to do this check (?), see test_extend in Lib/test/list_tests.py,
// https://github.com/python/cpython/blob/master/Objects/listobject.c#L934-L940
if cap >= isize::max_value() as usize {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is sys.maxsize but just found there is no way to access to the const in rust side.

@youknowone
youknowone merged commit c319b16 into RustPython:master Oct 21, 2020
@qingshi163
qingshi163 deleted the dev4 branch October 22, 2020 06:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants