Skip to content

Default to universal newline conversion on Windows#9067

Open
headius wants to merge 1 commit intojruby:masterfrom
headius:windows_newline_conversion
Open

Default to universal newline conversion on Windows#9067
headius wants to merge 1 commit intojruby:masterfrom
headius:windows_newline_conversion

Conversation

@headius
Copy link
Member

@headius headius commented Nov 8, 2025

Universal newline conversion is a transcoding mode used when the conversion of newlines from one encoding to another is desired during the IO process. It also handles Windows newlines ("\r\n"), and was originally the mechanism by which CRuby handled newline normalization on Windows.

See ruby/ruby@8761467#diff-686754e19b3c08fbc0880fade77986fed2c09fdd27dcc163fc68e0a7e22b7913R319

Later, CRuby was modified to leverage the O_TEXT mode on Windows, which does OS-level normalization of newlines, and to indicate this new watered-down default they switched it to a non-universal CRLF_NEWLINE mode. Logic later in the transcoding creation and read conversion processes would check whether NEED_READCONV and then use O_TEXT mode instead of a transcoder.

See ruby/ruby@f9a6a1d#diff-686754e19b3c08fbc0880fade77986fed2c09fdd27dcc163fc68e0a7e22b7913R318-R321

JRuby's ported logic largely matches this, except we have no way to specify O_TEXT when opening files with the JDK, so that mode ends up getting ignored and we don't actually do the newline conversions.

A short-term fix, which is really what we should have done years ago, is to switch the default windows transcoding flag from the watered down CRLF_NEWLINE mode back to the UNIVERSAL_NEWLINE mode, forcing the use of the transcoder.

Future improvements to this code could restore the optimized logic if O_TEXT becomes available to us.

This should fix a number of newline conversion issues, but may have other unexpected consequences.

Universal newline conversion is a transcoding mode used when the
conversion of newlines from one encoding to another is desired
during the IO process. It also handles Windows newlines ("\r\n"),
and was originally the mechanism by which CRuby handled newline
normalization on Windows.

See ruby/ruby@8761467#diff-686754e19b3c08fbc0880fade77986fed2c09fdd27dcc163fc68e0a7e22b7913R319

Later, CRuby was modified to leverage the O_TEXT mode on Windows,
which does OS-level normalization of newlines, and to indicate
this new watered-down default they switched it to a non-universal
CRLF_NEWLINE mode. Logic later in the transcoding creation and
read conversion processes would check whether NEED_READCONV and
then use O_TEXT mode instead of a transcoder.

See ruby/ruby@f9a6a1d#diff-686754e19b3c08fbc0880fade77986fed2c09fdd27dcc163fc68e0a7e22b7913R318-R321

JRuby's ported logic largely matches this, except we have no way
to specify O_TEXT when opening files with the JDK, so that mode
ends up getting ignored and we don't actually do the newline
conversions.

A short-term fix, which is really what we should have done years
ago, is to switch the default windows transcoding flag from the
watered down CRLF_NEWLINE mode back to the UNIVERSAL_NEWLINE mode,
forcing the use of the transcoder.

Future improvements to this code could restore the optimized logic
if O_TEXT becomes available to us.

This should fix a number of newline conversion issues, but may have
other unexpected consequences.
@headius headius force-pushed the windows_newline_conversion branch from 1c611f9 to 2058b6d Compare November 10, 2025 21:59
@headius
Copy link
Member Author

headius commented Jan 28, 2026

This is obviously still not right. Punting to 10.0.4.0 for more investigation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant