Skip to content

Compiler warnings in ubsan builds #85170

Closed

Description

@tiran
BPO 40998
Nosy @vstinner, @ericvsmith, @tiran, @ericsnowcurrently, @miss-islington
PRs
  • bpo-40998: Address compiler warnings found by ubsan #20929
  • bpo-40998: Fix a refleak in create_filter() #23365
  • [3.9] bpo-40998: Fix a refleak in create_filter() (GH-23365) #23369
  • [3.9] bpo-40998: Address compiler warnings found by ubsan (GH-20929) #23370
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2020-06-17.07:00:21.597>
    labels = ['build', '3.9', '3.10']
    title = 'Compiler warnings in ubsan builds'
    updated_at = <Date 2020-11-18.17:45:56.990>
    user = 'https://github.com/tiran'

    bugs.python.org fields:

    activity = <Date 2020-11-18.17:45:56.990>
    actor = 'miss-islington'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Build']
    creation = <Date 2020-06-17.07:00:21.597>
    creator = 'christian.heimes'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 40998
    keywords = ['patch']
    message_count = 11.0
    messages = ['371712', '371713', '381332', '381334', '381335', '381336', '381338', '381349', '381353', '381354', '381362']
    nosy_count = 5.0
    nosy_names = ['vstinner', 'eric.smith', 'christian.heimes', 'eric.snow', 'miss-islington']
    pr_nums = ['20929', '23365', '23369', '23370']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'compile error'
    url = 'https://bugs.python.org/issue40998'
    versions = ['Python 3.9', 'Python 3.10']

    Activity

    1. tiran commented on Jun 17, 2020

      @tiran
      MemberAuthor

      I'm seeing several compiler warnings in ubsan builds:

      $ ./configure --with-address-sanitizer --with-undefined-behavior-sanitizer
      $ make clean
      $ make
      Parser/string_parser.c: In function ‘decode_unicode_with_escapes’:
      Parser/string_parser.c:98:17: warning: null destination pointer [-Wformat-overflow=]
         98 |                 sprintf(p, "\\U%08x", chr);
            |                 ^~~~~~~~~~~~~~~~~~~~~~~~~~
      Parser/string_parser.c:98:17: warning: null destination pointer [-Wformat-overflow=]
      Parser/string_parser.c:98:17: warning: null destination pointer [-Wformat-overflow=]
      In function ‘assemble_lnotab’,
          inlined from ‘assemble_emit’ at Python/compile.c:5697:25,
          inlined from ‘assemble’ at Python/compile.c:6036:18:
      Python/compile.c:5651:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
       5651 |         *lnotab++ = k;
            |         ~~~~~~~~~~^~~
      Objects/unicodeobject.c: In function ‘xmlcharrefreplace’:
      Objects/unicodeobject.c:849:16: warning: null destination pointer [-Wformat-overflow=]
        849 |         str += sprintf(str, "&#%d;", PyUnicode_READ(kind, data, i));
            |                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      Objects/unicodeobject.c:849:16: warning: null destination pointer [-Wformat-overflow=]
      Objects/unicodeobject.c:849:16: warning: null destination pointer [-Wformat-overflow=]
      Python/pylifecycle.c: In function ‘Py_FinalizeEx’:
      Python/pylifecycle.c:1339:25: warning: unused variable ‘interp’ [-Wunused-variable]
       1339 |     PyInterpreterState *interp = tstate->interp;
            |
    2. tiran commented on Jun 17, 2020

      @tiran
      MemberAuthor

      PR 20929 addresses three out of four warnings found by GCC 10's ubsan on Fedora 32.

    3. tiran commented on Nov 18, 2020

      @tiran
      MemberAuthor

      I see more warnings and a new leak detection on latest master:

      gcc -pthread -c -fsanitize=undefined -fsanitize=address -fno-omit-frame-pointer -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I. -I./Include -DPy_BUILD_CORE -o Objects/codeobject.o Objects/codeobject.c
      In function ‘emit_pair’,
      inlined from ‘emit_delta’ at Objects/codeobject.c:423:14,
      inlined from ‘code_getlnotab’ at Objects/codeobject.c:462:18:
      Objects/codeobject.c:414:15: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
      414 | *lnotab++ = b;
      | ~~~~~~~~~~^~~
      In function ‘emit_pair’,
      inlined from ‘emit_delta’ at Objects/codeobject.c:436:14,
      inlined from ‘code_getlnotab’ at Objects/codeobject.c:462:18:
      Objects/codeobject.c:414:15: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
      414 | *lnotab++ = b;
      | ~~~~~~~~~~^~~
      In function ‘emit_pair’,
      inlined from ‘emit_delta’ at Objects/codeobject.c:429:14,
      inlined from ‘code_getlnotab’ at Objects/codeobject.c:462:18:
      Objects/codeobject.c:414:15: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
      414 | *lnotab++ = b;
      | ~~~~~~~~~~^~~
      gcc -pthread -c -fsanitize=undefined -fsanitize=address -fno-omit-frame-pointer -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I. -I./Include -DPy_BUILD_CORE -o Python/compile.o Python/compile.c
      In function ‘assemble_emit_linetable_pair’,
      inlined from ‘assemble_line_range’ at Python/compile.c:5614:18:
      Python/compile.c:5586:15: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
      5586 | *lnotab++ = ldelta;
      | ~~~~~~~~~~^~~~~~~~
      In function ‘assemble_emit_linetable_pair’,
      inlined from ‘assemble_line_range’ at Python/compile.c:5608:18:
      Python/compile.c:5586:15: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
      5586 | *lnotab++ = ldelta;
      | ~~~~~~~~~~^~~~~~~~
      In function ‘assemble_emit_linetable_pair’,
      inlined from ‘assemble’ at Python/compile.c:6011:10:
      Python/compile.c:5586:15: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
      5586 | *lnotab++ = ldelta;
      | ~~~~~~~~~~^~~~~~~~

      ==669952==ERROR: LeakSanitizer: detected memory leaks

      Direct leak of 57 byte(s) in 1 object(s) allocated from:
      #0 0x7fa6397cc667 in __interceptor_malloc (/lib64/libasan.so.6+0xb0667)
      #1 0x777579 in PyUnicode_New Objects/unicodeobject.c:1459
      #2 0x86aa4e in unicode_decode_utf8 Objects/unicodeobject.c:5129
      #3 0x8b6050 in PyUnicode_DecodeUTF8Stateful Objects/unicodeobject.c:5259
      #4 0x8b6050 in PyUnicode_FromString Objects/unicodeobject.c:2311
      #5 0x8b6050 in PyUnicode_InternFromString Objects/unicodeobject.c:15788
      #6 0x8f1e0b in create_filter Python/_warnings.c:67
      #7 0x8f1e0b in init_filters Python/_warnings.c:95
      #8 0x8f1e0b in _PyWarnings_InitState Python/_warnings.c:123
      #9 0xa52178 in pycore_init_types Python/pylifecycle.c:704
      #10 0xa52178 in pycore_interp_init Python/pylifecycle.c:760
      #11 0xa5eab1 in pyinit_config Python/pylifecycle.c:807
      #12 0xa5eab1 in pyinit_core Python/pylifecycle.c:970
      #13 0xa60037 in Py_InitializeFromConfig Python/pylifecycle.c:1155
      #14 0x47a842 in pymain_init Modules/main.c:66
      #15 0x4802a2 in pymain_main Modules/main.c:698
      #16 0x4802a2 in Py_BytesMain Modules/main.c:731
      #17 0x7fa638a5b041 in __libc_start_main (/lib64/libc.so.6+0x27041)

    4. tiran commented on Nov 18, 2020

      @tiran
      MemberAuthor

      The reference leak was introduced in 86ea581 / #57368 / bpo-36737. PR #65128 fixes it, too.

    5. tiran commented on Nov 18, 2020

      @tiran
      MemberAuthor

      Sorry, I meant to add Eric S. :)

    6. ericvsmith commented on Nov 18, 2020

      @ericvsmith
      Member

      Too many Eric S's!

    7. vstinner commented on Nov 18, 2020

      @vstinner
      Member

      The reference leak was introduced in 86ea581 / #57368 / bpo-36737. PR #65128 fixes it, too.

      Usually, it's not that the subinterpreter work introduces new leak, but makes old leak suddenly visible.

      The create_filter() leak was introduced way earlier:

      commit 9b99747
      Author: Nick Coghlan <ncoghlan@gmail.com>
      Date: Mon Jan 8 12:45:02 2018 +1000

      bpo-31975 (PEP-565): Show DeprecationWarning in __main__ (GH-4458)
      

      It's a minor leak since the create_filter() function is called exactly 5 times at startup. It's a leak of 5 strong references, it's not a big deal :-)

    8. miss-islington commented on Nov 18, 2020

      @miss-islington
      Contributor

      New changeset d1e38d4 by Victor Stinner in branch 'master':
      bpo-40998: Fix a refleak in create_filter() (GH-23365)
      d1e38d4

    9. miss-islington commented on Nov 18, 2020

      @miss-islington
      Contributor

      New changeset 07f2ade by Christian Heimes in branch 'master':
      bpo-40998: Address compiler warnings found by ubsan (GH-20929)
      07f2ade

    10. miss-islington commented on Nov 18, 2020

      @miss-islington
      Contributor

      New changeset 994c68f by Miss Islington (bot) in branch '3.9':
      bpo-40998: Address compiler warnings found by ubsan (GH-20929)
      994c68f

    11. miss-islington commented on Nov 18, 2020

      @miss-islington
      Contributor

      New changeset 35bf8ea by Christian Heimes in branch '3.9':
      [3.9] bpo-40998: Fix a refleak in create_filter() (GH-23365) (GH-23369)
      35bf8ea

    12. transferred this issue fromon Apr 10, 2022
    13. iritkatriel commented on May 20, 2022

      @iritkatriel
      Member

      Is there anything left to do here?

    14. added
      pendingThe issue will be closed if no feedback is provided
      on May 20, 2022
    15. tiran commented on May 20, 2022

      @tiran
      MemberAuthor

      I don't think so.

    16. removed
      pendingThe issue will be closed if no feedback is provided
      on May 20, 2022
    17. vstinner commented on May 23, 2022

      @vstinner
      Member

      xmlcharrefreplace():

              size = sprintf(str, "&#%d;", PyUnicode_READ(kind, data, i));
              if (size < 0) {
                  return NULL;
              }
              str += size;
      

      If sprintf() can return a negative value, a Python exception must be raised when returning NULL. Otherwise, you get a SystemError (return NULL without raising an exception).

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.10 (EOL)end of life3.9 (EOL)end of lifebuildThe build process and cross-build

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions