Moving from JRuby 1.7.23 to JRuby 9.1.2.0 causes gsub(RegExp, Hash) to fail with a Java::JavaLang::StackOverflowError. The code works fine for several minutes and only fails after significant concurrent load.

Environment

• jruby 9.1.2.0 (2.3.0) 2016-05-26 7357c8f Java HotSpot(TM) 64-Bit Server VM 25.66-b17 on 1.8.0_66-b17 +jit [linux-x86_64]
• Linux tparelapp1.clarity.net 2.6.32-504.3.3.el6.x86_64 break script engine #1 SMP Wed Dec 17 01:55:02 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

JAVA_OPTS_GENERAL="-server -Dfile.encoding=UTF-8"
JAVA_OPTS_JMX="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=7100 -Dcom.sun.management.jmxremote.ssl=false -Djmx.remote.x.server.connection.timeout=30000"
JAVA_OPTS_MEM="-Xms8192m -Xmx8192m -Xss2048k -XX:G1HeapRegionSize=8m -XX:ReservedCodeCacheSize=512m"
JAVA_OPTS_GC="-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+UseCodeCacheFlushing -XX:+ParallelRefProcEnabled"
JAVA_OPTS_GC_LOG="-XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintAdaptiveSizePolicy  -Xloggc:log/jvm_gc_${INSTANCE}.log"
JAVA_OPTS_JRUBY="-Djruby.thread.pool.enabled=false -Djruby.compile.invokedynamic=false -Djruby.global.require.lock=false"
export JAVA_OPTS="${JAVA_OPTS_GENERAL} ${JAVA_OPTS_JMX} ${JAVA_OPTS_MEM} ${JAVA_OPTS_GC} ${JAVA_OPTS_GC_LOG} ${JAVA_OPTS_JRUBY}"

Applicable gems:

• Haml 4.0.7
• Rails 4.1.15

Expected Behavior

# Extract of the code that fails in HAML on the gsub line
# haml-4.0.7/lib/helpers.rb:539
HTML_ESCAPE = {'&' => '&amp;', '<' => '&lt;', '>' => '&gt;', '"' => '&quot;', "'" => '&#039;', }
HTML_ESCAPE_REGEX = /[\"><&]/

def html_escape(text)
  text = text.to_s
  text.gsub(HTML_ESCAPE_REGEX, HTML_ESCAPE)
end

# Extract of code that fails in Rails on the gsub line
# activesupport-4.1.15/lib/active_support/core_ext/string/output_safety.rb:25
HTML_ESCAPE = { '&' => '&amp;',  '>' => '&gt;',   '<' => '&lt;', '"' => '&quot;', "'" => '&#39;' }
HTML_ESCAPE_REGEXP = /[&"'><]/
def html_escape(s)
  s = s.to_s
  if s.html_safe?
    s
  else
    s.gsub(HTML_ESCAPE_REGEXP, HTML_ESCAPE).html_safe
  end
end

Under MRI and prior to JRuby optimization of the code, the html_escape completes successfully in both cases.

Works without issue currently in production on JRuby 1.7.23, under high concurrent load.

Actual Behavior

• After several minutes of automated testing the application repeatedly throws Java::JavaLang::StackOverflowError when rendering HTML pages. The top line of each stack trace is one of the 2 above sub lines.
• In moving from JRuby 1.7.23 to JRuby 9.1.2.0 no application code changes were made. Only Jruby was upgraded.
• In both cases the gsub is receiving a Hash as the second parameter.
• The complete stack trace: https://gist.github.com/reidmorrison/b8ce8d5916690e6011435f79d413d930

Workaround

• By replacing the gsub(RegExp, Hash) with a gsub(RegExp, &block) we can work around the problem.

When patching the above code with an alternative call to gsub it works without issue:

def html_escape(text)
  text = text.to_s
  text.gsub(HTML_ESCAPE_REGEX) do |match|
    case match
    when '&'
      '&'
    when '<'
      '&lt;'
    when '>'
      '&gt;'
    when '"'
      '&quot;'
    when "'"
      '&#039;'
    end
  end
end

So far we have not been able to get a standalone reproducible example of this failure.