Skip to content

Add tests for the C tokenizer and expose it as a private module#27924

Merged
pablogsal merged 5 commits intopython:mainfrom
pablogsal:test_c_tokenizer
Aug 24, 2021
Merged

Add tests for the C tokenizer and expose it as a private module#27924
pablogsal merged 5 commits intopython:mainfrom
pablogsal:test_c_tokenizer

Conversation

@pablogsal
Copy link
Copy Markdown
Member

@pablogsal pablogsal commented Aug 24, 2021

No description provided.

@pablogsal pablogsal requested a review from a team as a code owner August 24, 2021 00:09
@pablogsal pablogsal requested a review from isidentical August 24, 2021 00:10
@pablogsal pablogsal force-pushed the test_c_tokenizer branch 7 times, most recently from 9d34dc1 to b11828c Compare August 24, 2021 12:08
isidentical
isidentical approved these changes Aug 24, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@isidentical isidentical left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (a couple PEP 7 nits, and one question about the condition)

@isidentical
Copy link
Copy Markdown
Member

isidentical commented Aug 24, 2021

Running the buildbots (for refleaks, if there are any)

@isidentical isidentical added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 24, 2021
@bedevere-bot
Copy link
Copy Markdown

bedevere-bot commented Aug 24, 2021

🤖 New build scheduled with the buildbot fleet by @isidentical for commit deee43b 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

@bedevere-bot bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Aug 24, 2021
@pablogsal
Copy link
Copy Markdown
Member Author

pablogsal commented Aug 24, 2021 

❯ ./python -m test test_tokenize -R :
0:00:00 load avg: 0.52 Run tests sequentially
0:00:00 load avg: 0.52 [1/1] test_tokenize
beginning 9 repetitions
123456789
.........

== Tests result: SUCCESS ==

1 test OK.

Total duration: 12.7 sec
Tests result: SUCCESS

@pablogsal pablogsal merged commit a24676b into python:main Aug 24, 2021
@pablogsal pablogsal deleted the test_c_tokenizer branch August 24, 2021 16:50
ambv
ambv reviewed Aug 24, 2021
View reviewed changes
Python/Python-tokenize.c
Comment on lines +24 to +25
[clinic start generated code]*/
/*[clinic end generated code: output=da39a3ee5e6b4b0d input=96d98ee2fef7a8bc]*/
Copy link
Copy Markdown
Contributor

@ambv ambv Aug 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this expected?

Copy link
Copy Markdown
Member Author

@pablogsal pablogsal Aug 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, check other modules that have classes created with the argument clinic. Maybe I am missing what you mean though 😅

Python/Python-tokenize.c
if (self == NULL) {
return NULL;
}
PyObject* filename = PyUnicode_FromString("<string>");
Copy link
Copy Markdown
Contributor

@ambv ambv Aug 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This returns a new reference, don't you have to decref it in line 53?

Copy link
Copy Markdown
Member Author

@pablogsal pablogsal Aug 24, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, the tokenizer free function does it

Ah, I misread the comment. Yeah, we are missing a decref in the error path

Copy link
Copy Markdown
Member Author

@pablogsal pablogsal Aug 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #27935

Python/Python-tokenize.c
{Py_tp_getattro, PyObject_GenericGetAttr},
{Py_tp_iter, PyObject_SelfIter},
{Py_tp_iternext, tokenizeriter_next},
{0, NULL},
Copy link
Copy Markdown
Contributor

@ambv ambv Aug 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are some of the initializers 8-char indented while others are 4-char indented?

Copy link
Copy Markdown
Member Author

@pablogsal pablogsal Aug 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hummm, I think is because I messed up with the format. All should be 4 char. Will correct this in another pR unless you manage to do it before

Copy link
Copy Markdown
Member Author

@pablogsal pablogsal Aug 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in #27935

Python/Python-tokenize.c
};

static PyModuleDef_Slot tokenizemodule_slots[] = {
{Py_mod_exec, tokenizemodule_exec},
Copy link
Copy Markdown
Contributor

@ambv ambv Aug 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Example of a 4-indented initializer.

@sunmy2019 sunmy2019 mentioned this pull request Apr 12, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ambv ambv ambv left review comments

@isidentical isidentical isidentical approved these changes

Assignees

No one assigned

Labels

skip issue skip news

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@pablogsal @isidentical @bedevere-bot @ambv @the-knights-who-say-ni