Skip to content

bpo-37335: Fix unexpected ASCII aliases in locale coercion tests #14443

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kulikjak wants to merge 2 commits into from
Closed

bpo-37335: Fix unexpected ASCII aliases in locale coercion tests #14443

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1ef7967
bpo-37335: Fix unexpected ASCII aliases in locale coercion tests
kulikjak Jun 28, 2019
802ef53
Add news entry
kulikjak Jun 28, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 6 additions & 19 deletions Lib/test/test_c_locale_coercion.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,11 +100,11 @@ def _set_locale_in_subprocess(locale_name):
class EncodingDetails(_EncodingDetails):
# XXX (ncoghlan): Using JSON for child state reporting may be less fragile
CHILD_PROCESS_SCRIPT = ";".join([
"import sys, os",
"print(sys.getfilesystemencoding())",
"print(sys.stdin.encoding + ':' + sys.stdin.errors)",
"print(sys.stdout.encoding + ':' + sys.stdout.errors)",
"print(sys.stderr.encoding + ':' + sys.stderr.errors)",
"import sys, os, codecs",
"print(codecs.lookup(sys.getfilesystemencoding()).name)",
"print(codecs.lookup(sys.stdin.encoding).name + ':' + sys.stdin.errors)",
"print(codecs.lookup(sys.stdout.encoding).name + ':' + sys.stdout.errors)",
"print(codecs.lookup(sys.stderr.encoding).name + ':' + sys.stderr.errors)",

@vstinner vstinner Jun 28, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh wait, I added _PyUnicode_InitEncodings() in Python 3.8: filesystem and stdio encodings are now always normalized. https://bugs.python.org/issue36775 :

commit 43fc3bb7cf0278735eb0010d7b3043775a120cb5
Author: Victor Stinner <vstinner@redhat.com>
Date:   Thu May 2 11:54:20 2019 -0400

    bpo-36775: Add _PyUnicode_InitEncodings() (GH-13057)
    
    Move get_codec_name() and initfsencoding() from pylifecycle.c to
    unicodeobject.c.
    
    Rename also "init" functions in pylifecycle.c.

So _handle_output_variations() can simply be removed from 3.8, without having to add codecs.lookup().name here.

This change makes sense to Python 3.7. Would you mind to first create a new PR for master, backport it to 3.8, and then retarget this PR to 3.7?

@kulikjak kulikjak Jun 28, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that will be the reason why Python 3.8 no longer has this problem. I will look into it.

Is there a way how to retarget this to 3.7? When we tried it previously, Github tried to backport the whole master into 3.7... Or do I have to create new PR for that as well?

@vstinner vstinner Jun 28, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, maybe creating a new PR for 3.7 is simpler :-)

"print(os.environ.get('LANG', 'not set'))",
"print(os.environ.get('LC_CTYPE', 'not set'))",
"print(os.environ.get('LC_ALL', 'not set'))",
Expand All @@ -128,19 +128,6 @@ def get_expected_details(cls, coercion_expected, fs_encoding, stream_encoding, e
env_info = expected_lang, expected_lc_ctype, expected_lc_all
return dict(cls(fs_encoding, *stream_info, *env_info)._asdict())

@staticmethod
def _handle_output_variations(data):
"""Adjust the output to handle platform specific idiosyncrasies

* Some platforms report ASCII as ANSI_X3.4-1968
* Some platforms report ASCII as US-ASCII
* Some platforms report UTF-8 instead of utf-8
"""
data = data.replace(b"ANSI_X3.4-1968", b"ascii")
data = data.replace(b"US-ASCII", b"ascii")
data = data.lower()
return data

@classmethod
def get_child_details(cls, env_vars):
"""Retrieves fsencoding and standard stream details from a child process
Expand All @@ -160,7 +147,7 @@ def get_child_details(cls, env_vars):
if not result.rc == 0:
result.fail(py_cmd)
# All subprocess outputs in this test case should be pure ASCII
adjusted_output = cls._handle_output_variations(result.out)
adjusted_output = result.out.lower()
stdout_lines = adjusted_output.decode("ascii").splitlines()
child_encoding_details = dict(cls(*stdout_lines)._asdict())
stderr_lines = result.err.decode("ascii").rstrip().splitlines()
Expand Down
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Tests/2019-06-28-13-55-58.bpo-37335.LLzOx8.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Improve locale coercion tests by using codec lookup instead of more fragile
replace().