Skip to content

textwrap output may change if you wrap a paragraph twice #76578

Open

Description

@larryhastings
BPO 32397
Nosy @larryhastings, @serhiy-storchaka, @JulienPalard, @csabella, @iritkatriel, @akulakov
PRs
  • gh-76578: Fix textwrap.wrap() so it's stable if run twice. #5615
  • gh-76578: new option for TextWrapper to allow stable wrapping #27587
  • Files
  • textwrap.isnt.stable.py
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2017-12-21.14:10:43.127>
    labels = ['type-feature', 'library', '3.11']
    title = 'textwrap output may change if you wrap a paragraph twice'
    updated_at = <Date 2021-08-23.19:11:29.979>
    user = 'https://github.com/larryhastings'

    bugs.python.org fields:

    activity = <Date 2021-08-23.19:11:29.979>
    actor = 'terry.reedy'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)']
    creation = <Date 2017-12-21.14:10:43.127>
    creator = 'larry'
    dependencies = []
    files = ['47344']
    hgrepos = []
    issue_num = 32397
    keywords = ['patch']
    message_count = 10.0
    messages = ['308872', '308873', '342825', '378248', '398726', '398836', '398839', '398846', '398847', '398858']
    nosy_count = 6.0
    nosy_names = ['larry', 'serhiy.storchaka', 'mdk', 'cheryl.sabella', 'iritkatriel', 'andrei.avk']
    pr_nums = ['5615', '27587']
    priority = 'low'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue32397'
    versions = ['Python 3.11']

    Activity

    1. larryhastings commented on Dec 21, 2017

      @larryhastings
      ContributorAuthor

      If you word-wrap a paragraph twice with textwrap, you may get different results. Specifically, you *will* get different results when:

      • the original text has a line that is too long by one character,
      • the last word on the line is the first word in a new sentence, and
      • there are two spaces after the period.

      The first textwrap will replace the two spaces after the period with a newline; the second textwrap will replace the newline with a single space.

      Attached is a test case demonstrating the problem.

      It's not a big problem, but it did cause an assertion failure in blurb. The workaround was to word-wrap all paragraphs twice, which works but is kind of dumb.

    2. added
      stdlibStandard Library Python modules in the Lib/ directory
      on Dec 21, 2017
    3. larryhastings commented on Dec 21, 2017

      @larryhastings
      ContributorAuthor

      FWIW, the test program produces this output:

      ------------------------------

       original: 'xxxx xxxx xxxx xxxx xxxx.  xxxx'
        wrapped: 'xxxx xxxx xxxx xxxx xxxx.\\nxxxx'
      

      wrapped twice: 'xxxx xxxx xxxx xxxx xxxx. xxxx'

      Traceback (most recent call last):
        File "textwrap.isnt.stable.py", line 24, in <module>
          assert wrapped == wrapped2
      AssertionError
    4. csabella commented on May 18, 2019

      @csabella
      Contributor

      @larry, it looks like this was close to being merged pending some review comments by Serhiy. Although this is considered a bug and not a new feature, it might be nice to try to get this in for 3.8. Thanks!

    5. JulienPalard commented on Oct 8, 2020

      @JulienPalard
      Member

      Could be "related" to https://bugs.python.org/issue41975.

    6. akulakov commented on Aug 2, 2021

      @akulakov
      Contributor

      This issue is due to *drop_whitespace* arg being True by default.

      Documentation states that *drop_whitespace* applies after wrapping, so the fix in the PR would break that promise because, as far as I understand, it applies the wrapping after *drop_whitespace*. (Irit's comment on the PR refers to this).

      I feel like the behaviour in the PR is more logical and probably what most users would prefer and need in most cases. But it's a backwards compatibility change of behaviour that's not buggy.

      'foo bar'
      # wrap for width=10
      ['foo','bar'] # current behaviour
      ['foo bar'] # more intuitive wrapping to width=10

      As I was looking at this, I thought that wrapping with drop_whitespace=False would surely be stable. Not so!

      It looks like the logic assumes that \n needs to be replaced by a space for cases like 'foo\nbar', which makes sense because otherwise two words would be joined together. But with drop_whitespace=False, repeated fill() calls will keep replacing \n with a space and then adding a new \n to split lines again:

        original: 'xxxx xxxx xxxx xxxx xxxx.  xxxx'
         wrapped: 'xxxx xxxx xxxx xxxx xxxx.  \\nxxxx'
      

      wrapped twice: 'xxxx xxxx xxxx xxxx xxxx. \nxxxx'
      wrapped thrice: 'xxxx xxxx xxxx xxxx xxxx. \nxxxx'

      Further, a newline with a nearby space within the requested width will be converted to two spaces:
      'a\n b' => 'a b'

      I don't know if the original issue is worth fixing or not, but I think the issue shown above would be good to fix.

    7. akulakov commented on Aug 3, 2021

      @akulakov
      Contributor

      I think fix to make drop_whitespace=False stable, can be as simple as adding two lines in _munge_whitespace():

      + text = re.sub(r' \n', ' ', text)
      + text = re.sub(r'\n ', ' ', text)
      text = text.translate(self.unicode_whitespace_trans)

      The perf impact is not small though, 12% :

      2892 (~/opensource/cpython) % ./python.exe -mtimeit 'import textwrap' 'textwrap.wrap("abc foo\nbar baz", 5)' --INS--
      5000 loops, best of 5: 60.2 usec per loop

      2893 (~/opensource/cpython) % r --INS--
      ./python.exe -mtimeit 'import textwrap' 'textwrap.wrap("abc foo\nbar baz", 5)'
      5000 loops, best of 5: 52.9 usec per loop

      I don't know if it's worth doing, but if yes, the options are:

      • just add this change for drop_whitespace=False, which is not the default, so perf regression will not affect default usage of wrap.

      • add a new arg that will only have effect when drop_whitespace=False, and will run these 2 lines. Name could be something like collapse_space_newline. It's hard to think of a good name.

      If '\r\n' is handled, it needs one additional sub() line, and the perf. difference is 22%.

    8. iritkatriel commented on Aug 3, 2021

      @iritkatriel
      Member

      You should be able to do them in one re, something like

      text = re.sub(r' ?\n', ' ', text)
    9. akulakov commented on Aug 3, 2021

      @akulakov
      Contributor

      Irit: I assume you mean r' \r?\n', that's a great idea, it's much faster than adding a separate replacement step.

      Latest version I came up with is this:

                      if re.search(r' \r?\n', text):
                          text = re.sub(r' \r?\n', ' ', text)
                      if re.search(r'\r?\n ', text):
                          text = re.sub(r'\r?\n ', ' ', text)

      This optimizes the case when there's no newlines, which is likely the most common case for small fragments of text, but it may be the less common case for larger fragments where performance is more important; so I'm not sure if it's worth it.

      Timings:
      # sub() has to run
      2904 (~/opensource/cpython) % ./python.exe -mtimeit 'import textwrap' 'textwrap.wrap("abc foo\n bar baz", 5)' ----VICMD----
      5000 loops, best of 5: 67.6 usec per loop

      # search() runs; but sub() does NOT because there's no adjacent space
      2906 (~/opensource/cpython) % ./python.exe -mtimeit 'import textwrap' 'textwrap.wrap("abc foo\nbar baz", 5)' ----VICMD----
      5000 loops, best of 5: 60.3 usec per loop

    10. akulakov commented on Aug 3, 2021

      @akulakov
      Contributor

      Note that I'm not handling a single '\r' because that was before Mac OS X; but it is handled by the following line (i.e. by the old logic):

                  text = text.translate(self.unicode_whitespace_trans)
    11. akulakov commented on Aug 4, 2021

      @akulakov
      Contributor

      I've added an initial draft PR: https://github.com/python/cpython/pull/27587/files

      I will add docs and news if this looks good in general.

    12. added
      3.11only security fixes
      type-featureA feature request or enhancement
      and removed
      type-bugAn unexpected behavior, bug, or error
      on Aug 23, 2021
    13. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.11only security fixesstdlibStandard Library Python modules in the Lib/ directorytype-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions