jruby-openssl spins endlessly trying to write to a socket which is in state CLOSE_WAIT #1156
Description
On
$ jruby --version
jruby 1.7.5 (1.9.3p392) 2013-10-07 74e9291 on OpenJDK 64-Bit Server VM 1.7.0_40-b31 [linux-amd64]
with the jruby-openssl-0.9.4 gem installed, when running an application which accessess a HTTPS site as a HTTPS client, I repeatably observe following patterns after some running time:
- The CPU usage of the application is slightly above 100% times n, where n is a positive integer.
- The number of sockets which are actually HTTPS sockets and in CLOSE_WAIT state (as of "lsof") is equal to n.
- The number of threads calling org.jruby.ext.openssl.SSLSocket.syswrite(SSLSocket.java:664) in at a particular time (as evidenced by "jstack" or by "kill -3") is almost always n.
- The thread ids for these n affected threads being the same over time.
I conclude that there seems to be a bug in jruby-openssl that is characterized by a SSL client socket getting into the CLOSE_WAIT state (e.g. the remote party or the network terminating the connection unexpectedly) and jruby-openssl apparently not noticing this state, and, rather than that, attempting to write to the socket in an endless loop.
Some Java stack traces which were captured:
java.lang.Thread.State: RUNNABLE
at java.lang.Object.hashCode(Native Method)
at java.util.HashMap.hash(HashMap.java:366)
at java.util.HashMap.put(HashMap.java:496)
at java.util.HashSet.add(HashSet.java:217)
at sun.nio.ch.EPollSelectorImpl.implRegister(EPollSelectorImpl.java:165)
at sun.nio.ch.SelectorImpl.register(SelectorImpl.java:133)
- locked <0x00000000e3ce1d60> (a java.util.Collections$UnmodifiableSet)
at java.nio.channels.spi.AbstractSelectableChannel.register(AbstractSelectableChannel.java:209)
- locked <0x00000000e42aa638> (a java.lang.Object)
- locked <0x00000000e42aa648> (a java.lang.Object)
at java.nio.channels.SelectableChannel.register(SelectableChannel.java:277)
at org.jruby.ext.openssl.SSLSocket.waitSelect(SSLSocket.java:288)
at org.jruby.ext.openssl.SSLSocket.do_syswrite(SSLSocket.java:644)
at org.jruby.ext.openssl.SSLSocket.syswrite(SSLSocket.java:664)
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollCtl(Native Method)
at sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:287)
- locked <0x00000000e3cdf488> (a java.lang.Object)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:256)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000e3cdf410> (a sun.nio.ch.Util$2)
- locked <0x00000000e3cdf400> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000e3cdf1c8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
at org.jruby.ext.openssl.SSLSocket$2.run(SSLSocket.java:316)
at org.jruby.RubyThread.executeBlockingTask(RubyThread.java:1063)
at org.jruby.ext.openssl.SSLSocket.waitSelect(SSLSocket.java:295)
at org.jruby.ext.openssl.SSLSocket.do_syswrite(SSLSocket.java:644)
at org.jruby.ext.openssl.SSLSocket.syswrite(SSLSocket.java:664)
java.lang.Thread.State: RUNNABLE
at org.jruby.util.io.ChannelStream.fflush(ChannelStream.java:661)
- locked <0x00000000e4430e08> (a org.jruby.util.io.ChannelStream)
at org.jruby.RubyIO.flush(RubyIO.java:2168)
at org.jruby.ext.openssl.SSLSocket.do_syswrite(SSLSocket.java:654)
at org.jruby.ext.openssl.SSLSocket.syswrite(SSLSocket.java:664)
The bug seems to be similar to https://jira.codehaus.org/browse/JRUBY-6973 .