Skip to content

scan_eol() Buffer Over-read #69177

Closed
Closed

Description

@JohnLeitch
mannequin
BPO 24989
Nosy @vstinner, @larryhastings, @vadmium, @serhiy-storchaka
Files
  • scan_eol_Buffer_Over-read.patch: patch
  • scan_eol_Buffer_Over-read.py: repro
  • scan_eol_Buffer_Over-read_2.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
    closed_at = <Date 2015-09-04.05:13:49.956>
    created_at = <Date 2015-09-02.23:40:26.639>
    labels = ['type-security', 'extension-modules', 'expert-IO', 'release-blocker']
    title = 'scan_eol() Buffer Over-read'
    updated_at = <Date 2015-09-08.18:34:24.218>
    user = 'https://bugs.python.org/JohnLeitch'

    bugs.python.org fields:

    activity = <Date 2015-09-08.18:34:24.218>
    actor = 'Arfrever'
    assignee = 'serhiy.storchaka'
    closed = True
    closed_date = <Date 2015-09-04.05:13:49.956>
    closer = 'larry'
    components = ['Extension Modules', 'IO']
    creation = <Date 2015-09-02.23:40:26.639>
    creator = 'JohnLeitch'
    dependencies = []
    files = ['40326', '40327', '40330']
    hgrepos = []
    issue_num = 24989
    keywords = ['patch']
    message_count = 11.0
    messages = ['249584', '249600', '249602', '249606', '249610', '249617', '249699', '249711', '249717', '249718', '249738']
    nosy_count = 8.0
    nosy_names = ['vstinner', 'larry', 'Arfrever', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'JohnLeitch', 'brycedarling']
    pr_nums = []
    priority = 'release blocker'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'security'
    url = 'https://bugs.python.org/issue24989'
    versions = ['Python 3.5', 'Python 3.6']

    Activity

    1. JohnLeitch commented on Sep 2, 2015

      JohnLeitchmannequin
      MannequinAuthor

      Python 3.5 suffers from a vulnerability caused by the behavior of the scan_eol() function. When called, the function gets a line from the buffer of a BytesIO object by searching for a newline character starting at the position in the buffer.

      However, if the position is set to a value that is larger than the buffer, this logic will result in a call to memchr that reads off the end of the buffer:

      /* Move to the end of the line, up to the end of the string, s. */
      start = PyBytes_AS_STRING(self->buf) + self->pos;
      maxlen = self->string_size - self->pos;
      if (len < 0 || len > maxlen)
          len = maxlen;
      
          if (len) {
              n = memchr(start, '\n', len);

      In some applications, it may be possible to exploit this behavior to disclose the contents of adjacent memory. The buffer over-read can be observed by running the following script:

      import tempfile
      a = tempfile.SpooledTemporaryFile()
      a.seek(0b1)
      a.readlines()

      Which, depending on the arrangement of memory, may produce an exception such as this:

      0:000> g
      (698.188): Access violation - code c0000005 (first chance)
      First chance exceptions are reported before any exception handling.
      This exception may be expected and handled.
      eax=fff8a14c ebx=0a0a0a0a ecx=00000000 edx=05bb1000 esi=061211b0 edi=89090909
      eip=61c6caf2 esp=010af8dc ebp=010af914 iopl=0 nv up ei ng nz ac po nc
      cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010292
      python35!memchr+0x62:
      61c6caf2 8b0a mov ecx,dword ptr [edx] ds:002b:05bb1000=????????
      0:000> k1
      ChildEBP RetAddr
      010af8e 61b640f1 python35!memchr+0x62 [f:\dd\vctools\crt_bld\SELF_X86\crt\src\INTEL\memchr.asm @ 125]

      To fix this issue, it is recommended that scan_eol() be updated to check that the position is not greater than or equal to the size of the buffer. A proposed patch is attached.

    2. vadmium commented on Sep 3, 2015

      @vadmium
      Member

      Simpler test case, which might find a place somewhere like /Lib/test/test_memoryio.py:

      >>> from io import BytesIO
      >>> b = BytesIO()
      >>> b.seek(1)
      1
      >>> b.readlines()  # Should return an empty list
      Segmentation fault (core dumped)
      [Exit 139]

      The patch looks like the right approach. Just watch out for the indentation (tabs vs spaces).

      However I don’t see why we need the GET_BYTES_SIZE() macro or the (size_t) casting. Would it be okay comparing PyBytes_GET_SIZE() directly to self->pos? They are both meant to be Py_ssize_t.

      The bug looks like it was introduced by Serhiy’s changes in bpo-15381.

    3. JohnLeitch commented on Sep 3, 2015

      JohnLeitchmannequin
      MannequinAuthor

      We based our fix on the check in write_bytes:

          if (endpos > (size_t)PyBytes_GET_SIZE(self->buf)) {
              if (resize_buffer(self, endpos) < 0)
                  return -1;
          }

      I see now that our casting was extraneous. As for the macro, it was suspected that similar issues may be present and we wanted to write reusable code, but this also seems unnecessary now that it's known the cast is unneeded.

      Early tomorrow I'll take some time to create a revised patch.

    4. serhiy-storchaka commented on Sep 3, 2015

      @serhiy-storchaka
      Member

      It should be self->string_size, not PyBytes_GET_SIZE(self->buf).

      Thank you for your report and your patch John. Here is revised patch with tests based on Martin's test.

      Larry, perhaps this bug is grave enough to be fixed in RC3.

    5. vstinner commented on Sep 3, 2015

      @vstinner
      Member

      scan_eol_Buffer_Over-read_2.patch looks good to me.

      Larry, perhaps this bug is grave enough to be fixed in RC3.

      Since it looks like a regression in Python 3.5, yes, it's a severe bug. Please send a pull request to Larry for it.

    6. larryhastings commented on Sep 3, 2015

      @larryhastings
      Contributor

      Yes, please create a pull request for this patch. Thanks!

      And just to confirm: I just applied patch 2 to CPython, then undid the change to bytesio.c. The new test fails, and sometimes Python will segmentation fault. If I then apply the patch to bytesio.c it passes. Nice work!

    7. python-dev commented on Sep 3, 2015

      python-devmannequin
      Mannequin

      New changeset a5858c30db7c by Serhiy Storchaka in branch '3.5':
      Issue bpo-24989: Fixed buffer overread in BytesIO.readline() if a position is
      https://hg.python.org/cpython/rev/a5858c30db7c

      New changeset 215800fb955d by Serhiy Storchaka in branch 'default':
      Issue bpo-24989: Fixed buffer overread in BytesIO.readline() if a position is
      https://hg.python.org/cpython/rev/215800fb955d

    8. larryhastings commented on Sep 4, 2015

      @larryhastings
      Contributor

      Pull request accepted. Please forward-merge. Thanks!

    9. python-dev commented on Sep 4, 2015

      python-devmannequin
      Mannequin

      New changeset 2b6ce7e9595c by Serhiy Storchaka in branch '3.5':
      Issue bpo-24989: Fixed buffer overread in BytesIO.readline() if a position is
      https://hg.python.org/cpython/rev/2b6ce7e9595c

    10. python-dev commented on Sep 4, 2015

      python-devmannequin
      Mannequin

      New changeset 07e04c34bab5 by Larry Hastings in branch '3.5':
      Merged in storchaka/cpython350 (pull request #13)
      https://hg.python.org/cpython/rev/07e04c34bab5

    11. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions