You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
textwrap output may change if you wrap a paragraph twice #76578
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee=Noneclosed_at=Nonecreated_at=<Date2017-12-21.14:10:43.127>labels= ['type-feature', 'library', '3.11']
title='textwrap output may change if you wrap a paragraph twice'updated_at=<Date2021-08-23.19:11:29.979>user='https://github.com/larryhastings'
If you word-wrap a paragraph twice with textwrap, you may get different results. Specifically, you *will* get different results when:
the original text has a line that is too long by one character,
the last word on the line is the first word in a new sentence, and
there are two spaces after the period.
The first textwrap will replace the two spaces after the period with a newline; the second textwrap will replace the newline with a single space.
Attached is a test case demonstrating the problem.
It's not a big problem, but it did cause an assertion failure in blurb. The workaround was to word-wrap all paragraphs twice, which works but is kind of dumb.
@larry, it looks like this was close to being merged pending some review comments by Serhiy. Although this is considered a bug and not a new feature, it might be nice to try to get this in for 3.8. Thanks!
This issue is due to *drop_whitespace* arg being True by default.
Documentation states that *drop_whitespace* applies after wrapping, so the fix in the PR would break that promise because, as far as I understand, it applies the wrapping after *drop_whitespace*. (Irit's comment on the PR refers to this).
I feel like the behaviour in the PR is more logical and probably what most users would prefer and need in most cases. But it's a backwards compatibility change of behaviour that's not buggy.
'foo bar'
# wrap for width=10
['foo','bar'] # current behaviour
['foo bar'] # more intuitive wrapping to width=10
As I was looking at this, I thought that wrapping with drop_whitespace=False would surely be stable. Not so!
It looks like the logic assumes that \n needs to be replaced by a space for cases like 'foo\nbar', which makes sense because otherwise two words would be joined together. But with drop_whitespace=False, repeated fill() calls will keep replacing \n with a space and then adding a new \n to split lines again:
I think fix to make drop_whitespace=False stable, can be as simple as adding two lines in _munge_whitespace():
+ text = re.sub(r' \n', ' ', text)
+ text = re.sub(r'\n ', ' ', text)
text = text.translate(self.unicode_whitespace_trans)
The perf impact is not small though, 12% :
2892 (~/opensource/cpython) % ./python.exe -mtimeit 'import textwrap' 'textwrap.wrap("abc foo\nbar baz", 5)' --INS--
5000 loops, best of 5: 60.2 usec per loop
2893 (~/opensource/cpython) % r --INS--
./python.exe -mtimeit 'import textwrap' 'textwrap.wrap("abc foo\nbar baz", 5)'
5000 loops, best of 5: 52.9 usec per loop
I don't know if it's worth doing, but if yes, the options are:
just add this change for drop_whitespace=False, which is not the default, so perf regression will not affect default usage of wrap.
add a new arg that will only have effect when drop_whitespace=False, and will run these 2 lines. Name could be something like collapse_space_newline. It's hard to think of a good name.
If '\r\n' is handled, it needs one additional sub() line, and the perf. difference is 22%.
This optimizes the case when there's no newlines, which is likely the most common case for small fragments of text, but it may be the less common case for larger fragments where performance is more important; so I'm not sure if it's worth it.
Timings:
# sub() has to run
2904 (~/opensource/cpython) % ./python.exe -mtimeit 'import textwrap' 'textwrap.wrap("abc foo\n bar baz", 5)' ----VICMD----
5000 loops, best of 5: 67.6 usec per loop
# search() runs; but sub() does NOT because there's no adjacent space
2906 (~/opensource/cpython) % ./python.exe -mtimeit 'import textwrap' 'textwrap.wrap("abc foo\nbar baz", 5)' ----VICMD----
5000 loops, best of 5: 60.3 usec per loop
Activity
larryhastings commentedon Dec 21, 2017
If you word-wrap a paragraph twice with textwrap, you may get different results. Specifically, you *will* get different results when:
The first textwrap will replace the two spaces after the period with a newline; the second textwrap will replace the newline with a single space.
Attached is a test case demonstrating the problem.
It's not a big problem, but it did cause an assertion failure in blurb. The workaround was to word-wrap all paragraphs twice, which works but is kind of dumb.
larryhastings commentedon Dec 21, 2017
FWIW, the test program produces this output:
------------------------------
wrapped twice: 'xxxx xxxx xxxx xxxx xxxx. xxxx'
csabella commentedon May 18, 2019
@larry, it looks like this was close to being merged pending some review comments by Serhiy. Although this is considered a bug and not a new feature, it might be nice to try to get this in for 3.8. Thanks!
JulienPalard commentedon Oct 8, 2020
Could be "related" to https://bugs.python.org/issue41975.
akulakov commentedon Aug 2, 2021
This issue is due to *drop_whitespace* arg being True by default.
Documentation states that *drop_whitespace* applies after wrapping, so the fix in the PR would break that promise because, as far as I understand, it applies the wrapping after *drop_whitespace*. (Irit's comment on the PR refers to this).
I feel like the behaviour in the PR is more logical and probably what most users would prefer and need in most cases. But it's a backwards compatibility change of behaviour that's not buggy.
'foo bar'
# wrap for width=10
['foo','bar'] # current behaviour
['foo bar'] # more intuitive wrapping to width=10
As I was looking at this, I thought that wrapping with drop_whitespace=False would surely be stable. Not so!
It looks like the logic assumes that \n needs to be replaced by a space for cases like 'foo\nbar', which makes sense because otherwise two words would be joined together. But with drop_whitespace=False, repeated fill() calls will keep replacing \n with a space and then adding a new \n to split lines again:
wrapped twice: 'xxxx xxxx xxxx xxxx xxxx. \nxxxx'
wrapped thrice: 'xxxx xxxx xxxx xxxx xxxx. \nxxxx'
Further, a newline with a nearby space within the requested width will be converted to two spaces:
'a\n b' => 'a b'
I don't know if the original issue is worth fixing or not, but I think the issue shown above would be good to fix.
akulakov commentedon Aug 3, 2021
I think fix to make
drop_whitespace=Falsestable, can be as simple as adding two lines in_munge_whitespace():+ text = re.sub(r' \n', ' ', text)
+ text = re.sub(r'\n ', ' ', text)
text = text.translate(self.unicode_whitespace_trans)
The perf impact is not small though, 12% :
2892 (~/opensource/cpython) % ./python.exe -mtimeit 'import textwrap' 'textwrap.wrap("abc foo\nbar baz", 5)' --INS--
5000 loops, best of 5: 60.2 usec per loop
2893 (~/opensource/cpython) % r --INS--
./python.exe -mtimeit 'import textwrap' 'textwrap.wrap("abc foo\nbar baz", 5)'
5000 loops, best of 5: 52.9 usec per loop
I don't know if it's worth doing, but if yes, the options are:
just add this change for drop_whitespace=False, which is not the default, so perf regression will not affect default usage of wrap.
add a new arg that will only have effect when drop_whitespace=False, and will run these 2 lines. Name could be something like
collapse_space_newline. It's hard to think of a good name.If '\r\n' is handled, it needs one additional
sub()line, and the perf. difference is 22%.iritkatriel commentedon Aug 3, 2021
You should be able to do them in one re, something like
akulakov commentedon Aug 3, 2021
Irit: I assume you mean r' \r?\n', that's a great idea, it's much faster than adding a separate replacement step.
Latest version I came up with is this:
This optimizes the case when there's no newlines, which is likely the most common case for small fragments of text, but it may be the less common case for larger fragments where performance is more important; so I'm not sure if it's worth it.
Timings:
# sub() has to run
2904 (~/opensource/cpython) % ./python.exe -mtimeit 'import textwrap' 'textwrap.wrap("abc foo\n bar baz", 5)' ----VICMD----
5000 loops, best of 5: 67.6 usec per loop
# search() runs; but sub() does NOT because there's no adjacent space
2906 (~/opensource/cpython) % ./python.exe -mtimeit 'import textwrap' 'textwrap.wrap("abc foo\nbar baz", 5)' ----VICMD----
5000 loops, best of 5: 60.3 usec per loop
akulakov commentedon Aug 3, 2021
Note that I'm not handling a single '\r' because that was before Mac OS X; but it is handled by the following line (i.e. by the old logic):
akulakov commentedon Aug 4, 2021
I've added an initial draft PR: https://github.com/python/cpython/pull/27587/files
I will add docs and news if this looks good in general.