Skip to content

bpo-33578: Add getstate/setstate for CJK codec#6984

Merged
miss-islington merged 1 commit into
python:masterfrom
libcthorne:fix-multibyte-state
Nov 1, 2018
Merged

bpo-33578: Add getstate/setstate for CJK codec#6984
miss-islington merged 1 commit into
python:masterfrom
libcthorne:fix-multibyte-state

Conversation

@libcthorne

@libcthorne libcthorne commented May 19, 2018 •
edited by methane
Loading

Copy link
Copy Markdown
Contributor

This implements getstate and setstate for the cjkcodecs multibyte incremental encoders/decoders, primarily to fix issues with seek/tell.

The encoder getstate/setstate is slightly tricky as the "state" is pending bytes + MultibyteCodec_State but only an integer can be returned. The approach I've taken is to encode this data into a long, similar to how .tell() encodes a "cookie_type" as a long.

https://bugs.python.org/issue33578

@the-knights-who-say-ni

This comment has been minimized.

Sign in to view
@libcthorne

libcthorne commented May 19, 2018

Copy link
Copy Markdown
Contributor Author

(Added GitHub name to bugs.python.org. CLA is signed already)

pablogsal
pablogsal previously requested changes May 30, 2018
View reviewed changes
Comment thread Modules/cjkcodecs/multibytecodec.c Outdated

@pablogsal pablogsal May 30, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to refactor all error paths under one "error" label section and substitute these with gotosas indicated here so there is only one exit point from the function?

Comment thread Modules/cjkcodecs/multibytecodec.c Outdated

This comment was marked as resolved.

Sign in to view

Comment thread Modules/cjkcodecs/multibytecodec.c Outdated

This comment was marked as resolved.

Sign in to view

@libcthorne
libcthorne force-pushed the fix-multibyte-state branch from 8b5ee96 to db1a055 Compare June 3, 2018 16:12
@libcthorne

libcthorne commented Jun 3, 2018

Copy link
Copy Markdown
Contributor Author

Thanks for the review @pablogsal, I've made the changes. Refactoring to use an error label helped catch a missing decref 👍

pablogsal
pablogsal reviewed Jun 3, 2018
View reviewed changes
Comment thread Modules/cjkcodecs/multibytecodec.c Outdated
@libcthorne
libcthorne force-pushed the fix-multibyte-state branch from db1a055 to 6f9869d Compare June 3, 2018 16:36
pablogsal
pablogsal reviewed Jun 3, 2018
View reviewed changes
Comment thread Lib/test/test_io.py Outdated

@pablogsal pablogsal Jun 3, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is leaking a file descriptor:

ResourceWarning: unclosed file <_io.TextIOWrapper name='@test_22392_tmp' mode='r' encoding='jisx0213'>

@libcthorne libcthorne Jun 3, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, missed the last .close, thanks.

pablogsal
pablogsal reviewed Jun 3, 2018
View reviewed changes
Comment thread Lib/test/test_io.py Outdated

This comment was marked as resolved.

Sign in to view

This comment was marked as resolved.

Sign in to view

This comment was marked as resolved.

Sign in to view

This comment was marked as resolved.

Sign in to view

This comment was marked as resolved.

Sign in to view

@libcthorne
libcthorne force-pushed the fix-multibyte-state branch from 6f9869d to 0af0cc7 Compare June 3, 2018 17:30
methane
methane reviewed Jun 4, 2018
View reviewed changes
Comment thread Lib/test/test_io.py Outdated

@methane methane Jun 4, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use "euc_jis_2004" instead of "jisx0213" alias.
"JIS X 0213:2004" defines charset. There are three encoding for it; "ISO-2022-JP-2004", "EUC-JIS-2004", and "Shift-JIS 2004".
Additionally, there is an old version of it; "JIS X 0213:2000".

I don't know why "jisx0213" alias for "euc_jis_2004" was added, but it seems not predictable alias.

@libcthorne libcthorne Jun 4, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I've changed it to use euc_jis_2004. Also replaced euc-jp with euc_jp.

@libcthorne
libcthorne force-pushed the fix-multibyte-state branch from 0af0cc7 to 44727e7 Compare June 4, 2018 20:27
methane
methane reviewed Jun 5, 2018
View reviewed changes
Comment thread Modules/cjkcodecs/multibytecodec.c Outdated
@libcthorne
libcthorne force-pushed the fix-multibyte-state branch 2 times, most recently from be4dc14 to 3c18d61 Compare June 7, 2018 23:30
libcthorne
libcthorne commented Jun 7, 2018
View reviewed changes
Comment thread Modules/cjkcodecs/multibytecodec.c Outdated
@libcthorne

libcthorne commented Jun 8, 2018 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Need to fix the following warnings:

2018-06-07T23:35:38.9073811Z /opt/vsts/work/1/s/Modules/cjkcodecs/multibytecodec.c: In function ‘_multibytecodec_MultibyteIncrementalEncoder_setstate’:
2018-06-07T23:35:38.9095425Z /opt/vsts/work/1/s/Modules/cjkcodecs/multibytecodec.c:947:40: warning: passing argument 2 of ‘_PyLong_AsByteArray’ from incompatible pointer type [-Wincompatible-pointer-types]
2018-06-07T23:35:38.9112219Z      if (_PyLong_AsByteArray(statelong, &state,
2018-06-07T23:35:38.9128553Z                                         ^
2018-06-07T23:35:38.9142991Z In file included from ./Include/Python.h:81:0,
2018-06-07T23:35:38.9157853Z                  from /opt/vsts/work/1/s/Modules/cjkcodecs/multibytecodec.c:8:
2018-06-07T23:35:38.9175164Z ./Include/longobject.h:163:17: note: expected ‘unsigned char *’ but argument is of type ‘MultibyteIncrementalEncoderState * {aka struct <anonymous> *}’
2018-06-07T23:35:38.9191247Z  PyAPI_FUNC(int) _PyLong_AsByteArray(PyLongObject* v,

and Travis has picked up a checksum mismatch:

Error in file "./Modules/cjkcodecs/multibytecodec.c" on line 936:
Checksum mismatch!
Expected: 2ef45090fc274f1e
Computed: ded04a702b17468c

Update: done

@libcthorne
libcthorne force-pushed the fix-multibyte-state branch from 3c18d61 to f60f50c Compare June 8, 2018 22:58
methane
methane requested changes Jun 11, 2018
View reviewed changes
Comment thread Modules/cjkcodecs/multibytecodec.c Outdated

@methane methane Jun 11, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use S for better error message..
https://docs.python.org/3/library/codecs.html#codecs.IncrementalDecoder.getstate

Comment thread Modules/cjkcodecs/multibytecodec.c Outdated

@methane methane Jun 11, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyArg_ParseTuple will raise TypeError. So this check may be redundant.

@libcthorne libcthorne Jun 11, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyArg_ParseTuple is enough but the error message is not very user friendly:

>>> import codecs
>>> decoder = codecs.getincrementaldecoder('euc_jp')()
>>> decoder.setstate(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: new style getargs format but argument is not a tuple

@methane methane Jun 12, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then, you can use ArgumentClinic again. object(subclass_of='&PyTuple_Type').

@libcthorne libcthorne Jun 12, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that works nicely :)

>>> decoder.setstate(0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: setstate() argument must be tuple, not int

Comment thread Modules/cjkcodecs/multibytecodec.c Outdated

@methane methane Jun 11, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Split } and else { in two lines.

Comment thread Modules/cjkcodecs/multibytecodec.c Outdated

@methane methane Jun 11, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to check buffersize <= MAXDECPENDING

@methane methane Jun 11, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And test cases for possible wrong inputs are needed too.

Comment thread Modules/cjkcodecs/multibytecodec.c Outdated

@methane methane Jun 11, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use PyBytes_FromStringAndSize.

@bedevere-bot

bedevere-bot commented Jun 11, 2018

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

methane
methane reviewed Jun 11, 2018
View reviewed changes
Comment thread Modules/cjkcodecs/multibytecodec.c Outdated

@methane methane Jun 11, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is make_tuple() in this file. It can be used instead.

methane
methane reviewed Jun 11, 2018
View reviewed changes
Comment thread Modules/cjkcodecs/multibytecodec.h Outdated

@methane methane Jun 11, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This struct is used from only two place in single source file.
Move this to the file.

@libcthorne
libcthorne force-pushed the fix-multibyte-state branch from c97199f to edf0b4e Compare July 7, 2018 18:25
@libcthorne

libcthorne commented Jul 7, 2018

Copy link
Copy Markdown
Contributor Author

Changes to address feedback from @doerwalter have been pushed.

  • MultibyteCodec_State has been changed to a struct with single field and usage updated appropriately
  • MultibyteIncrementalEncoderState.pendingsize has been changed to an unsigned char to make its representation independent of endianness and 32/64 bit size
  • PyLong_FromByteArray and PyLong_ToByteArray calls are passed constant of 1 for little_endian parameter to ensure same value is returned on big/little endian systems
  • test_multibytecodec has been updated to match endian related changes and also has some readability improvements

methane
methane reviewed Jul 9, 2018
View reviewed changes
Comment thread Modules/cjkcodecs/multibytecodec.c Outdated

@methane methane Jul 9, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Struct layout is different by compiler.
To build portable integer, you need to use just bytearray:

unsigned char state[1 + MAXPENDING*4 + 8];
...
state[0] = peindingsize;
memcpy(state+1, pendingbuffer, pendingsize);
int pos = 1 + pendingsize;
memcpy(state+pos, self->state, 8);

@libcthorne libcthorne Jul 9, 2018 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @methane. state should now be independent of x32/64, endianness, and compiler.

Latest commit changes:

  • Replace MultibyteIncrementalEncoderState with byte array
  • Update test_multibytecodec to match new byte array format (difference: pendingsize moved to the first byte + only pendingsize bytes stored in buffer position)

@libcthorne libcthorne Jul 9, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also added test_getstate_returns_expected_value to test_multibytecodec so that the consistency of the state byte array structure is validated by the test suite.

@libcthorne
libcthorne force-pushed the fix-multibyte-state branch 5 times, most recently from e43fb6f to b36149f Compare July 9, 2018 23:06
@methane
methane dismissed pablogsal’s stale review July 11, 2018 01:20

all resolved

@zhangyangyu
zhangyangyu self-requested a review July 12, 2018 04:11
@vstinner

vstinner commented Oct 19, 2018

Copy link
Copy Markdown
Member

@methane: you approved the change, but you didn't merge it. Do you want another core dev to approve the change, or did you just forget to merge it?

@methane

methane commented Oct 20, 2018

Copy link
Copy Markdown
Member

@doerwalter Would you review this again before merging this?

doerwalter
doerwalter reviewed Oct 22, 2018
View reviewed changes
Comment thread Modules/cjkcodecs/_codecs_cn.c

#define STATE_OFFSET 0

/*

@doerwalter doerwalter Oct 22, 2018 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the meaning of this #define?

@libcthorne libcthorne Oct 22, 2018 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I felt c[STATE_OFFSET] was more explicit than c[0], but don't mind inlining it if you prefer. Alternatively, STATE_POSITION may be a better name, as I wanted to get across that only byte 0 is relevant in state->c for the CN codec.

Perhaps even CN_STATE_POSITION?

@doerwalter doerwalter Oct 22, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I get it: c[8] is the merged version of the previous union, so it serves two purposes: it's a character array and a (very) small integer. IMHO this (and how it's used) should be documented where MultibyteCodec_State is defined.

@libcthorne libcthorne Oct 22, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I were to document how I see MultibyteCodec_State now, I would say something along the lines of:

A union that provides 8 bytes of state for multibyte codecs. Codecs
are free to use this how they want. Note: if you need to add a new
member to this union, ensure that its byte order is independent of CPU
endianness so that the return value of `getstate` doesn't differ
between little and big endian CPUs.

Should I add this as a comment next to the union?

@doerwalter doerwalter Oct 23, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me. However MultibyteCodec_State no longer is a union, "if you need to add a new member to this union" should be phrased differently.

And STATE_OFFSET would have a comment like: "Codecs that have to store a small integer in MultibyteCodec_State store it at the following offset".

And yes, multibytecodec.h is a little light on documentation, but that's probably another patch.

@libcthorne libcthorne Oct 23, 2018

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added comments next to STATE_OFFSET (now CN_STATE_OFFSET for clarity) and MultibyteCodec_State. Please let me know if the explanations make sense.

@doerwalter doerwalter Oct 23, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, but I'd like @methane to make the final decision about merging this PR.

@methane methane changed the title bpo-33578: Implement multibyte encoder/decoder state methods bpo-33578: Add getstate/setstate for multibyte codec Nov 1, 2018
@methane methane changed the title bpo-33578: Add getstate/setstate for multibyte codec bpo-33578: Add getstate/setstate for CJK codec Nov 1, 2018
@miss-islington
miss-islington merged commit ac22f6a into python:master Nov 1, 2018
@bedevere-bot

bedevere-bot commented Nov 1, 2018

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x SLES 3.x has failed when building commit ac22f6a.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/16/builds/1829) and take a look at the build logs.
  4. Check if the failure is related to this commit (ac22f6a) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/16/builds/1829

Click to see traceback logs 
From https://github.com/python/cpython
 * branch            master     -> FETCH_HEAD
Reset branch 'master'

/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Modules/_testcapimodule.c:5062:5: warning: initialization from incompatible pointer type [enabled by default]
     {"bad_get", bad_get, METH_FASTCALL},
     ^
/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Modules/_testcapimodule.c:5062:5: warning: (near initialization for ‘TestMethods[173].ml_meth’) [enabled by default]

test_devpoll skipped -- test works only on Solaris OS family
test_tix skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_sqlite skipped -- No module named '_sqlite3'
test_ttk_guionly skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_startfile skipped -- object <module 'os' from '/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/os.py'> has no attribute 'startfile'
test_kqueue skipped -- test works only on BSD
test_readline skipped -- No module named 'readline'
python: /home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Modules/cjkcodecs/_codecs_iso2022.c:515: iso2022_decode: Assertion `dsg->mark != '\0'' failed.
Fatal Python error: Aborted

Current thread 0x000003ffb85f9710 (most recent call first):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_codecs.py", line 72 in check_state_handling_decode
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_codecs.py", line 2135 in test_decoder_state
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 610 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 658 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/runner.py", line 176 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/support/__init__.py", line 1928 in _run_suite
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/support/__init__.py", line 2022 in run_unittest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 124 in runtest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest_mp.py", line 68 in run_tests_worker
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 587 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 571 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 627 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/regrtest.py", line 46 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/regrtest.py", line 50 in <module>
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/runpy.py", line 85 in _run_code
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/runpy.py", line 192 in _run_module_as_main
stty: 'standard input': Inappropriate ioctl for device
test_winconsoleio skipped -- test only relevant on win32
test_winreg skipped -- No module named 'winreg'
test_msilib skipped -- No module named '_msi'
test_ioctl skipped -- Unable to open /dev/tty
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
test_ossaudiodev skipped -- [Errno 2] No such file or directory: '/dev/dsp'
test_winsound skipped -- No module named 'winsound'
test_tk skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
python: /home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Modules/cjkcodecs/_codecs_iso2022.c:515: iso2022_decode: Assertion `dsg->mark != '\0'' failed.
Fatal Python error: Aborted

Current thread 0x000003ffb4f79710 (most recent call first):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_codecs.py", line 72 in check_state_handling_decode
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_codecs.py", line 2135 in test_decoder_state
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 610 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 658 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/runner.py", line 176 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/support/__init__.py", line 1928 in _run_suite
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/support/__init__.py", line 2022 in run_unittest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/runtest.py", line 134 in runtest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 301 in rerun_failed_tests
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 610 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 571 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/libregrtest/main.py", line 627 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/__main__.py", line 2 in <module>
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/runpy.py", line 85 in _run_code
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/runpy.py", line 192 in _run_module_as_main
make: *** [buildbottest] Aborted (core dumped)

@bedevere-bot

bedevere-bot commented Nov 1, 2018

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL 3.x has failed when building commit ac22f6a.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/21/builds/1850) and take a look at the build logs.
  4. Check if the failure is related to this commit (ac22f6a) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/21/builds/1850

Click to see traceback logs 
From https://github.com/python/cpython
 * branch            master     -> FETCH_HEAD
Reset branch 'master'

/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Modules/_testcapimodule.c:5062:5: warning: initialization from incompatible pointer type [enabled by default]
     {"bad_get", bad_get, METH_FASTCALL},
     ^
/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Modules/_testcapimodule.c:5062:5: warning: (near initialization for ‘TestMethods[173].ml_meth’) [enabled by default]
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s
/usr/bin/ld: skipping incompatible /usr/local/lib/libgcc_s.so when searching for -lgcc_s

test_ioctl skipped -- Unable to open /dev/tty
python: /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Modules/cjkcodecs/_codecs_iso2022.c:515: iso2022_decode: Assertion `dsg->mark != '\0'' failed.
Fatal Python error: Aborted

Current thread 0x000003fffd12f700 (most recent call first):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/test_codecs.py", line 72 in check_state_handling_decode
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/test_codecs.py", line 2135 in test_decoder_state
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/case.py", line 610 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/case.py", line 658 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/runner.py", line 176 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/support/__init__.py", line 1928 in _run_suite
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/support/__init__.py", line 2022 in run_unittest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/runtest.py", line 124 in runtest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/runtest_mp.py", line 68 in run_tests_worker
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/main.py", line 587 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/main.py", line 571 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/main.py", line 627 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/regrtest.py", line 46 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/regrtest.py", line 50 in <module>
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/runpy.py", line 85 in _run_code
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/runpy.py", line 192 in _run_module_as_main
test_winreg skipped -- No module named 'winreg'
test_startfile skipped -- object <module 'os' from '/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/os.py'> has no attribute 'startfile'
test_winsound skipped -- No module named 'winsound'
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
test_devpoll skipped -- test works only on Solaris OS family
test_winconsoleio skipped -- test only relevant on win32
test_kqueue skipped -- test works only on BSD
test_msilib skipped -- No module named '_msi'
test_ossaudiodev skipped -- [Errno 2] No such file or directory: '/dev/dsp'
test_ttk_guionly skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
stty: standard input: Inappropriate ioctl for device
test_tix skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_tk skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
python: /home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Modules/cjkcodecs/_codecs_iso2022.c:515: iso2022_decode: Assertion `dsg->mark != '\0'' failed.
Fatal Python error: Aborted

Current thread 0x000003fffd3f3700 (most recent call first):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/test_codecs.py", line 72 in check_state_handling_decode
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/test_codecs.py", line 2135 in test_decoder_state
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/case.py", line 610 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/case.py", line 658 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/unittest/runner.py", line 176 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/support/__init__.py", line 1928 in _run_suite
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/support/__init__.py", line 2022 in run_unittest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/runtest.py", line 134 in runtest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/main.py", line 301 in rerun_failed_tests
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/main.py", line 610 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/main.py", line 571 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/libregrtest/main.py", line 627 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/test/__main__.py", line 2 in <module>
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/runpy.py", line 85 in _run_code
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel-z/build/Lib/runpy.py", line 192 in _run_module_as_main
make: *** [buildbottest] Aborted

@bedevere-bot

bedevere-bot commented Nov 1, 2018

Copy link
Copy Markdown

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Debian 3.x has failed when building commit ac22f6a.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/13/builds/1863) and take a look at the build logs.
  4. Check if the failure is related to this commit (ac22f6a) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/13/builds/1863

Click to see traceback logs 
From https://github.com/python/cpython
 * branch                  master     -> FETCH_HEAD
Reset branch 'master'

/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Modules/_testcapimodule.c:5062:17: warning: initialization from incompatible pointer type [-Wincompatible-pointer-types]
     {"bad_get", bad_get, METH_FASTCALL},
                 ^~~~~~~
/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Modules/_testcapimodule.c:5062:17: note: (near initialization for ‘TestMethods[173].ml_meth’)

test_tix skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_winreg skipped -- No module named 'winreg'
test_startfile skipped -- object <module 'os' from '/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/os.py'> has no attribute 'startfile'
test_winsound skipped -- No module named 'winsound'
python: /home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Modules/cjkcodecs/_codecs_iso2022.c:515: iso2022_decode: Assertion `dsg->mark != '\0'' failed.
Fatal Python error: Aborted

Current thread 0x000003ffa11f2700 (most recent call first):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_codecs.py", line 72 in check_state_handling_decode
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_codecs.py", line 2135 in test_decoder_state
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 610 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 658 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/runner.py", line 176 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/support/__init__.py", line 1928 in _run_suite
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/support/__init__.py", line 2022 in run_unittest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 124 in runtest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest_mp.py", line 68 in run_tests_worker
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 587 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 571 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 627 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/regrtest.py", line 46 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/regrtest.py", line 50 in <module>
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/runpy.py", line 85 in _run_code
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/runpy.py", line 192 in _run_module_as_main
stty: 'standard input': Inappropriate ioctl for device
test_ttk_guionly skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_zipfile64 skipped -- test requires loads of disk-space bytes and a long time to run
test_winconsoleio skipped -- test only relevant on win32
test_ossaudiodev skipped -- [Errno 2] No such file or directory: '/dev/dsp'
test_ioctl skipped -- Unable to open /dev/tty
test_kqueue skipped -- test works only on BSD
test_devpoll skipped -- test works only on Solaris OS family
test_tk skipped -- Tk unavailable due to TclError: no display name and no $DISPLAY environment variab [...]
test_msilib skipped -- No module named '_msi'
python: /home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Modules/cjkcodecs/_codecs_iso2022.c:515: iso2022_decode: Assertion `dsg->mark != '\0'' failed.
Fatal Python error: Aborted

Current thread 0x000003ffb5472700 (most recent call first):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_codecs.py", line 72 in check_state_handling_decode
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_codecs.py", line 2135 in test_decoder_state
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 610 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 658 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/runner.py", line 176 in run
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/support/__init__.py", line 1928 in _run_suite
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/support/__init__.py", line 2022 in run_unittest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 134 in runtest
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 301 in rerun_failed_tests
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 610 in _main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 571 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 627 in main
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/__main__.py", line 2 in <module>
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/runpy.py", line 85 in _run_code
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/runpy.py", line 192 in _run_module_as_main
make: *** [buildbottest] Aborted

@libcthorne

libcthorne commented Nov 1, 2018 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

I'm looking into the buildbot failures to see if they're related.

  • failing test is ...Lib/test/test_codecs.py", line 2135 in test_decoder_state
  • only s390x builds seem to be failing (e.g. this one passes https://buildbot.python.org/all/#/builders/141/builds/728)
  • test passes locally (Ubuntu 16.04 x64)
  • builders were stable before this commit was merged
  • failure caused by _codecs_iso2022.c, and this commit changed that file, so it is quite likely related

If I understand correctly, the s390x systems are big endian, so there is probably a problem in the handling of endianness. I will look into emulating a big endian machine locally in a few hours from now to debug this.

Update: I managed to recreate locally and the problem seems to be in _multibytecodec_MultibyteIncrementalDecoder_getstate_impl. The casting to Py_ssize_t isn't correct (on both little and big endian) and isn't consistent with setstate parsing. I'm going to use PyLongObject instead, in a similar way to how encoder getstate/setstate works.

@libcthorne libcthorne mentioned this pull request Nov 2, 2018
Merged
@libcthorne

libcthorne commented Nov 2, 2018

Copy link
Copy Markdown
Contributor Author

I've opened a PR (#10290) to address the issues found by the big-endian build bots.

@mattip mattip mentioned this pull request Aug 25, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pablogsal pablogsal pablogsal left review comments
@methane methane methane approved these changes
@malemburg malemburg Awaiting requested review from malemburg
@zhangyangyu zhangyangyu Awaiting requested review from zhangyangyu
+1 more reviewer
@doerwalter doerwalter doerwalter left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@libcthorne @the-knights-who-say-ni @bedevere-bot @methane @vstinner @doerwalter @pablogsal @miss-islington