Ripper fails to lexical analyse around escaped newline on JRuby #4787
Description
The code below is valid on JRuby 9.1.13.0, CRuby 2.4.1 and more old days:
1 == 2 ？ \
puts(true) : \
puts(false)But Ripper on JRuby fails to lexical analyse source code around escaped newline. It returns strange tokens.
I'm in need when using Ripper for RDoc.
Environment
$ uname -a
Linux x250 4.11.6-3-ARCH #1 SMP PREEMPT Thu Jun 22 12:21:46 CEST 2017 x86_64 GNU/Linux
$ ruby -v
jruby 9.1.13.0 (2.3.3) 2017-09-06 8e1c115 OpenJDK 64-Bit Server VM 25.131-b11 on 1.8.0_131-b11 +jit [linux-x86_64]Source Code
require 'ripper'
require 'pp'
pp Ripper.lex("1 == 2 ? \\\n puts(true) : \\\n puts(false)")Expected Behavior
This is on CRuby 2.4.1.
[[[1, 0], :on_int, "1"],
[[1, 1], :on_sp, " "],
[[1, 2], :on_op, "=="],
[[1, 4], :on_sp, " "],
[[1, 5], :on_int, "2"],
[[1, 6], :on_sp, " "],
[[1, 7], :on_op, "?"],
[[1, 8], :on_sp, " "],
[[1, 9], :on_sp, "\\\n"],
[[2, 0], :on_sp, " "],
[[2, 2], :on_ident, "puts"],
[[2, 6], :on_lparen, "("],
[[2, 7], :on_kw, "true"],
[[2, 11], :on_rparen, ")"],
[[2, 12], :on_sp, " "],
[[2, 13], :on_op, ":"],
[[2, 14], :on_sp, " "],
[[2, 15], :on_sp, "\\\n"],
[[3, 0], :on_sp, " "],
[[3, 2], :on_ident, "puts"],
[[3, 6], :on_lparen, "("],
[[3, 7], :on_kw, "false"],
[[3, 12], :on_rparen, ")"]]Actual Behavior
This is on JRuby 9.1.3.0 as described previously.
[[[1, 0], :on_int, "1"],
[[1, 1], :on_sp, " "],
[[1, 2], :on_op, "=="],
[[1, 4], :on_sp, " "],
[[1, 5], :on_int, "2"],
[[1, 6], :on_sp, " "],
[[1, 7], :on_op, "?"],
[[1, 8], :on_sp, " "],
[[1, 9], :on_ident, "\\\n"],
[[2, 0], :on_sp, " "],
[[2, 2], :on_lparen, "puts("],
[[2, 7], :on_kw, "true"],
[[2, 11], :on_rparen, ")"],
[[2, 12], :on_sp, " "],
[[2, 13], :on_op, ":"],
[[2, 14], :on_ident, "\\\n"],
[[3, 0], :on_sp, " "],
[[3, 2], :on_lparen, "puts("],
[[3, 7], :on_kw, "false"],
[[3, 12], :on_rparen, ")"]]