Updating Netty JARs leads to errors in test_httplib and others

@jeff5

I tried updating Netty to 4.1.112 (see #342), but it gives me errors in:

test_httplib
test_robotparser
test_smtpnet
test_ssl_jy
test_urllibnet

The error is always:

SSLError: [Errno 1] SslHandler removed before handshake completed

This suggests a race with the Netty asynchronous processing. Tear-down of Netty is known to be a fragile aspect of the tests.

The latest version known to work at this time is 4.1.73

Originally posted by @jeff5 in #342 (comment)

@jeff5

Thank you, @jeff5 for reporting this. If I find time I will pin the exact problematic netty version via bisection (if someone wants to help here, please feel free to take on that step) and try to derive a minimalistic example from the test code. This should not be a release stopper, should it? There are a couple of Netty-related CVEs though. Hard to tell whether that is critical for a Jython release.

Working out why with a simple reproducer would be good.

The CVEs are a motivation to update, and if the existing ones are not enough, there'll be another one along in a minute. These days Sonatype scores that kind of thing against your project.

4.1.73 passes, .80 fails, latest is .112. Almost certainly we will want to take a much later version than .79. There may be an interesting clue in the exact version at which it stops working, but almost certainly it is something we're doing wrong. Something to do with tearing down the connection or Netty event loop without waiting for actions to complete.

There may be an interesting clue in the exact version at which it stops working

That's exactly my thinking.

BTW I already googled the error and searched stack overflow without much useful results. (Perhaps worth a try to ask AI before we spend our own brainwork on this ^^)

From our perspective, Netty is fragile.
https://github.com/jython/jython/actions/runs/10140118092/job/28034672782

I don't think it is Netty's fault at all, but I can't tell whether deadlocks like that are because of the way the tests tear down connections without waiting, or that our implementation of _socket is defective in a way that matters to users.

What I can say, from much tedious debugging on other issues, is that using netty involves a lot of requesting and waiting, to turn a high-performance asynchronous framework into the synchronous one the Python API describes, and it looks easy to get wrong.

I remember this has blocked releases before and Jim Baker spent a lot of time on getting sockets right. Apparently we'll have to re-investigate the issue, perhaps including Netty alternatives. I somewhat had the impression that Netty might be an overkill for our needs. It is a huge framework and we are using little of it. (Maybe I am mistaken.) Do you have an idea how to solve sockets for Jython 3? ATM I do not really understand the problem. IIRC it was something like Python supports switching a socket between sync and async mode while in Java it is either the one or the other. To provide Python's behavior one therefore has to emulate one of sync/async on top of the other. That's only a very rough picture and may be incomplete or wrong. I would suggest to collect information about the problem in a new issue and then also poll for expertise on the mailing list and evaluate whether there exist alternative approaches for the particular issue currently solved (attempted to, that is) via Netty. I think that will also be crucial for finding a sufficient solution for this issue in Jython 3.

Releasing 2.7.4 with outdated Netty is not ideal but likely preferable over delaying the release, given that the difficulty of the issue is hard to assess.

It occurred to me that according to https://maven.apache.org/pom.html#dependency-version-requirement-specification netty versions are currently softly specified in the pom file (of Jython-Slim), which means that also a newer version may be used. Since that is not the case (for now), we should change them from

<version>4.1.73.Final</version>

to

<version>(,4.1.73.Final]</version>

However, for that we should ideally determine the actual upper bound.

I've never really mastered the art of giving our users the necessary wiggle room on versions while not promising things we haven't tested. When I had the chance to ask it seemed what we were doing was not an obstacle. But if you have a better way ...

We would have to express this in Gradle terms, as Gradle generates the POM, but it is happy to use Maven syntax, although, being Gradle, it also offers a really fussy way.

netty versions are currently softly specified in the pom file

I was surprised by the semantics. Gradle describes this as "exact", which I would have taken to be "strictly" that version. Maybe this is why it's not a problem, and a good choice already.

It's not that it doesn't work at all with later versions. Only it doesn't pass the tests. Better to fix that than delineate the edge in our POM. By the time we next release, netty 4.1.73 will be very old.

Netty 4.1.118 was released yesterday. I replaced Netty 4.1.73 with 4.1.118 in build.xml then ran the following commands:

$ ant clean
$ ant
$ dist/bin/jython -m test.regrtest -e -m regrtest_memo.txt

The httplib test did not fail, but produced the following warning.

test_htmllib
test_htmlparser
test_httplib
io.netty.channel.ChannelInitializer WARNING Failed to initialize a channel. Closing: [id: 0x0d5304d9, L:/127.0.0.1:39049 - R:/127.0.0.1:42223]
test_imaplib
test_imp

Using the following versions of Ant and Java:

Apache Ant(TM) version 1.10.14 compiled on August 16 2023

openjdk version "1.8.0_442"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_442-b06)
OpenJDK 64-Bit Server VM (Temurin)(build 25.442-b06, mixed mode)

The warning message is not always displayed. Running the httplib test by itself, the test sometimes complete without warnings.

dist/bin/jython Lib/test/test_httplib.py 
test_auto_headers (__main__.HeaderTests) ... ok
test_content_length_0 (__main__.HeaderTests) ... ok
test_invalid_headers (__main__.HeaderTests) ... ok
test_ipv6host_header (__main__.HeaderTests) ... ok
test_malformed_headers_coped_with (__main__.HeaderTests) ... ok
test_putheader (__main__.HeaderTests) ... ok
test_responses (__main__.OfflineTest) ... ok
test_bad_status_repr (__main__.BasicTest) ... ok
test_chunked (__main__.BasicTest) ... ok
test_chunked_head (__main__.BasicTest) ... ok
test_early_eof (__main__.BasicTest) ... ok
test_epipe (__main__.BasicTest) ... ok
test_error_leak (__main__.BasicTest) ... ok
test_filenoattr (__main__.BasicTest) ... ok
test_host_port (__main__.BasicTest) ... ok
test_incomplete_read (__main__.BasicTest) ... ok
test_negative_content_length (__main__.BasicTest) ... ok
test_overflowing_chunked_line (__main__.BasicTest) ... ok
test_overflowing_header_line (__main__.BasicTest) ... ok
test_overflowing_status_line (__main__.BasicTest) ... skipped 'disabled for HTTP 0.9 support'
test_partial_reads (__main__.BasicTest) ... ok
test_partial_reads_incomplete_body (__main__.BasicTest) ... ok
test_partial_reads_no_content_length (__main__.BasicTest) ... ok
test_proxy_tunnel_without_status_line (__main__.BasicTest) ... ok
test_read_head (__main__.BasicTest) ... ok
test_response_headers (__main__.BasicTest) ... ok
test_send (__main__.BasicTest) ... ok
test_send_file (__main__.BasicTest) ... ok
test_send_tempfile (__main__.BasicTest) ... ok
test_status_lines (__main__.BasicTest) ... ok
test_too_many_headers (__main__.BasicTest) ... ok
testTimeoutAttribute (__main__.TimeoutTest)
This will prove that the timeout gets through ... ok
testHTTPConnection (__main__.HTTPTest) ... ok
testHTTPWithConnectHostPort (__main__.HTTPTest) ... ok
test_attributes (__main__.HTTPSTest) ... ok
test_host_port (__main__.HTTPSTest) ... ok
test_local_bad_hostname (__main__.HTTPSTest) ... skipped 'FIXME: Jython does not raise proper SSL error'
test_local_good_hostname (__main__.HTTPSTest) ... skipped 'FIXME: Failing test on Jython (causes other exceptions if not skipped)'
test_local_unknown_cert (__main__.HTTPSTest) ... stopping HTTPS server
joining HTTPS thread
ok
test_networked (__main__.HTTPSTest) ... ok
test_networked_bad_cert (__main__.HTTPSTest) ... ok
test_networked_good_cert (__main__.HTTPSTest) ... ok
test_networked_noverification (__main__.HTTPSTest) ... ok
test_networked_trusted_by_default_cert (__main__.HTTPSTest) ... ok
testHTTPConnectionSourceAddress (__main__.SourceAddressTest) ... ok
testHTTPSConnectionSourceAddress (__main__.SourceAddressTest) ... ok
test_connect (__main__.TunnelTests) ... ok

Should Jython move to Netty 4.1.118?

...
skipped = [
    'test_codecmaps_hk',
    'test_curses',
    'test_smtpnet',
    'test_subprocess',
    'test_urllib2net',
    'test_urllibnet',
 ]
6 tests skipped:
    test_codecmaps_hk test_curses test_smtpnet test_subprocess
    test_urllib2net test_urllibnet
0 tests failed:

Should Jython move to Netty 4.1.118?

If your results can be confirmed, I'd say so. Best-case for us would be that this issue was just a netty regression that has been fixed.

Best-case for us would be that this issue was just a netty regression that has been fixed.

Bugs are very creative! It might even be due to a weird interaction between earlier version of Java 8 and later versions of Netty. I think it is important to not only build and test a Jython release again known versions of the Maven dependencies, but also to use a known (fixed) version of Java 8 and Ant. For that reason, I created issue #368 and submitted a pull request with support for installing the latest known stable versions of Java 8 and Ant using SDKMAN.

@jeff5

I tried updating Netty to 4.1.112 (see #342), but it gives me errors in ...

Reran the tests with Netty 4.1.112 and Java 8.0.442 to see if the errors still persists. This time no errors were encountered. So it seems highly likely that the errors @jeff5 encountered were due to a buggy Java VM (unknown version).

Results from re-testing against Netty 4.1.112 and Java 8.0.442:

6 tests skipped:
    test_codecmaps_hk test_curses test_smtpnet test_subprocess
    test_urllib2net test_urllibnet
0 tests failed:

Platform: 
    'Java-1.8.0_442-OpenJDK_64-Bit_Server_VM,_25.442-b06,_Temurin-on-Linux-6.11.0-14-generic-amd64'

Output of git diff build.xml

.sdkmanrc file used:

# Enable auto-env through the sdkman_auto_env config
# Add key=value pairs of SDKs to use below

# Ant
ant=1.10.14

# Java
java=8.0.442-tem

My build and test script (test.sh):

ant clean
ant
dist/bin/jython -m test.regrtest -e -m regrtest_memo.txt
cat regrtest_memo.txt
git diff

I'm afraid I still get test failures on 4.1.119 and 4.1.121 (and not on 4.1.73).

One has to test this with the official ant regrtest, which has the network resource enabled.
Alternatively, and to focus on just on the discovered test failures, make a file:

# netty.tests
# Failures on netty 4.1.112 (or generally beyond 4.1.73)
# Test with network resource enabled:  -m test.regrtest -u network -f netty.tests

test_httplib 
test_robotparser
test_smtpnet
test_ssl_jy
test_urllibnet

then use it to select the tests as dist\bin\jython -m test.regrtest -u network -f netty.tests, explicitly enabling the network resource (-u network).

@jeff5

I'm afraid I still get test failures on 4.1.119 and 4.1.121 (and not on 4.1.73).

Thanks @jeff5. I'll run the tests with -u network enabled.

OK, I now see the errors as well.

 $ dist/bin/jython -m test.regrtest -u network -f netty.tests
== 2.7.5a1-DEV (heads/dev/netty-4.1.118:0c25b2e42, May 4 2025, 13:44:14)
== [OpenJDK 64-Bit Server VM (Temurin)]
== platform: java1.8.0_452
== encodings: stdin=UTF-8, stdout=UTF-8, FS=utf-8
== locale: default=('en_US', 'UTF-8'), actual=(None, None)
test_httplib
io.netty.channel.ChannelInitializer WARNING Failed to initialize a channel. Closing: [id: 0x3a8d9b62, L:/127.0.0.1:43187 - R:/127.0.0.1:58213]
io.netty.channel.ChannelInitializer WARNING Failed to initialize a channel. Closing: [id: 0xb956d7a5, L:/127.0.0.1:44919 - R:/127.0.0.1:34017]
test test_httplib failed -- multiple errors occurred; run in verbose mode for details
test_robotparser
test test_robotparser failed -- Traceback (most recent call last):
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/test/test_robotparser.py", line 306, in testPythonOrg
    parser.read()
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/robotparser.py", line 58, in read
    f = opener.open(self.url)
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/urllib.py", line 219, in open
    return getattr(self, name)(url)
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/urllib.py", line 219, in open
    return getattr(self, name)(url)
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/urllib.py", line 370, in open_http
    return self.http_error(url, fp, errcode, errmsg, headers)
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/urllib.py", line 383, in http_error
    result = method(url, fp, errcode, errmsg, headers)
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/urllib.py", line 679, in http_error_301
    return self.http_error_302(url, fp, errcode, errmsg, headers, data)
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/urllib.py", line 647, in http_error_302
    result = self.redirect_internal(url, fp, errcode, errmsg,
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/urllib.py", line 675, in redirect_internal
    return self.open(newurl)
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/urllib.py", line 219, in open
    return getattr(self, name)(url)
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/urllib.py", line 449, in open_https
    h.endheaders(data)
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/httplib.py", line 1078, in endheaders
    self._send_output(message_body)
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/httplib.py", line 894, in _send_output
    self.send(msg)
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/httplib.py", line 856, in send
    self.connect()
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/httplib.py", line 1302, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/ssl.py", line 1094, in wrap_socket
    return SSLSocket(sock=sock, server_side=server_side,
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/ssl.py", line 596, in __init__
    self.do_handshake()
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/ssl.py", line 712, in do_handshake
    self._sock._handle_channel_future(self._handshake_future, "SSL handshake")
  File "/home/wfouche/IdeaProjects/jython/dist/Lib/_socket.py", line 385, in handle_exception
    raise _map_exception(jlx)
IOError: [Errno socket error] [Errno 1] SslHandler removed before handshake completed

test_smtpnet
test test_smtpnet failed -- multiple errors occurred; run in verbose mode for details
test_ssl_jy
test test_ssl_jy failed -- multiple errors occurred; run in verbose mode for details
test_urllibnet
test test_urllibnet failed -- multiple errors occurred; run in verbose mode for details
5 tests failed:
    test_httplib test_robotparser test_smtpnet test_ssl_jy
    test_urllibnet
5 fails unexpected:
    test_httplib test_robotparser test_smtpnet test_ssl_jy
    test_urllibnet

Just one note: I found netty/netty#12027 which sounds oddly related to the error we are getting "SslHandler removed before handshake completed". And, strange coincidence, it was "fixed" right after Netty 4.1.73. So, in that sense we might have been bug-compatible to Netty until 4.1.74.

Okay, let me bring to attention this method from Lib/_socket.py:

def _map_exception(java_exception):
    if isinstance(java_exception, NettyChannelException):
        java_exception = java_exception.cause  # unwrap
    if isinstance(java_exception, SSLException) or isinstance(java_exception, CertificateException):
        cause = java_exception.cause
        if cause:
            # netty is freaking backwards here. The original exception may be CertificateException
            # but netty wraps it in SSLHandshakeException, we need to unwrap to present the right message
            if isinstance(cause, SSLHandshakeException):
                if isinstance(cause.cause, CertificateException):
                    java_exception = cause.cause

            msg = "%s (%s)" % (java_exception.message, cause)
        else:
            msg = java_exception.message
        py_exception = SSLError(SSL_ERROR_SSL, msg)
    else:
        # Netty defines its own sub-types of java... exceptions that are not
        # public, so we look up the public version.
        if isinstance(java_exception, ConnectException):
            mapped_exception = _exception_map.get(ConnectException)
        elif isinstance(java_exception, NoRouteToHostException):
            mapped_exception = _exception_map.get(NoRouteToHostException)
        elif isinstance(java_exception, ClosedChannelException):
            mapped_exception = _exception_map.get(ClosedChannelException)
        else:
            # Look-up the exact Python class
            mapped_exception = _exception_map.get(java_exception.__class__)
        if mapped_exception:
            py_exception = mapped_exception(java_exception)
        else:
            py_exception = error(-1, 'Unmapped exception: %s' % java_exception)
    py_exception.java_exception = java_exception
    return _add_exception_attrs(py_exception)

It has special treatment for SSLHandshakeException, BUT the netty change set for curing netty/netty/#12027 look like this:

-            SSLHandshakeException cause = null;
+            SSLException cause = null;

            // If the handshake is not done yet we should fail the handshake promise and notify the rest of the
            // If the handshake or SSLEngine closure is not done yet we should fail corresponding promise and
            // notify the rest of the
            // pipeline.
            if (!handshakePromise.isDone()) {
                cause = new SSLHandshakeException("SslHandler removed before handshake completed");
@@ -692,7 +693,7 @@ public void handlerRemoved0(ChannelHandlerContext ctx) throws Exception {
            }
            if (!sslClosePromise.isDone()) {
                if (cause == null) {
-                    cause = new SSLHandshakeException("SslHandler removed before handshake completed");
+                    cause = new SSLException("SslHandler removed before SSLEngine was closed");
                }
                notifyClosePromise(cause);
            }

i.e. they throw an SSLException where used to be an SSLHandshakeException. Maybe this breaks the special handling in _socket.py and all it needs is to adjust that handling to Netty's new behavior.

Okay, I think I solved this one. We will be able to upgrade to newest netty 4.1.x. PR and explanation will follow. However, the issue turned out to have nothing to do with my previous two posts. That said, I notice something odd. I managed to get all network tests pass with latest netty 4.1.134, except for test_urllibnet. However, that tests fails for me already with the currently pinned netty 4.1.73:

 dist/bin/jython -m test.regrtest -v -u network test_urllibnet
== 2.7.5a1-DEV (heads/..., Mai 26 2026, 11:34:07)
== [OpenJDK 64-Bit Server VM (Private Build)]
== platform: java1.8.0_292
== encodings: stdin=UTF-8, stdout=UTF-8, FS=utf-8
== locale: default=('de_DE', 'UTF-8'), actual=(None, None)
test_urllibnet
testURLread (test.test_urllibnet.URLTimeoutTest) ... ok
test_bad_address (test.test_urllibnet.urlopenNetworkTests) ... ok
test_basic (test.test_urllibnet.urlopenNetworkTests) ... ok
test_fileno (test.test_urllibnet.urlopenNetworkTests) ... skipped 'Sockets cannot be used as file descriptors'
test_getcode (test.test_urllibnet.urlopenNetworkTests) ... ok
test_geturl (test.test_urllibnet.urlopenNetworkTests) ... ok
test_info (test.test_urllibnet.urlopenNetworkTests) ... ok
test_readlines (test.test_urllibnet.urlopenNetworkTests) ... ok
test_basic (test.test_urllibnet.urlretrieveNetworkTests) ... ok
test_data_header (test.test_urllibnet.urlretrieveNetworkTests) ... ERROR
test_header (test.test_urllibnet.urlretrieveNetworkTests) ... ok
test_specified_path (test.test_urllibnet.urlretrieveNetworkTests) ... ok

======================================================================
ERROR: test_data_header (test.test_urllibnet.urlretrieveNetworkTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "...jython/dist/Lib/test/test_urllibnet.py", line 206, in test_data_header
    self.fail('Date value not in %r format', dateformat)
TypeError: fail() takes at most 2 arguments (3 given)

----------------------------------------------------------------------
Ran 12 tests in 7.230s

FAILED (errors=1, skipped=1)
test test_urllibnet failed -- Traceback (most recent call last):
  File "...jython/dist/Lib/test/test_urllibnet.py", line 206, in test_data_header
    self.fail('Date value not in %r format', dateformat)
TypeError: fail() takes at most 2 arguments (3 given)

1 test failed:
    test_urllibnet
1 fail unexpected:
    test_urllibnet

You may notice, this is running on Java 8, so maybe it's a Java 8 thing. Ideas? Is it really working for you all? Should I open a separate issue for that one?

Just a quick check on my brand new M5 Mac passes:

./dist/bin/jython -m test.regrtest -v -u network test_urllibnet
== 2.7.5a1-DEV (heads/master:8493cb3aa, May 31 2026, 13:12:31)
== [OpenJDK 64-Bit Server VM (Azul Systems, Inc.)]
== platform: java1.8.0_492
== encodings: stdin=UTF-8, stdout=UTF-8, FS=utf-8
== locale: default=('en_US', 'UTF-8'), actual=(None, None)
test_urllibnet
testURLread (test.test_urllibnet.URLTimeoutTest) ... ok
test_bad_address (test.test_urllibnet.urlopenNetworkTests) ... ok
test_basic (test.test_urllibnet.urlopenNetworkTests) ... ok
test_fileno (test.test_urllibnet.urlopenNetworkTests) ... skipped 'Sockets cannot be used as file descriptors'
test_getcode (test.test_urllibnet.urlopenNetworkTests) ... ok
test_geturl (test.test_urllibnet.urlopenNetworkTests) ... ok
test_info (test.test_urllibnet.urlopenNetworkTests) ... ok
test_readlines (test.test_urllibnet.urlopenNetworkTests) ... ok
test_basic (test.test_urllibnet.urlretrieveNetworkTests) ... ok
test_data_header (test.test_urllibnet.urlretrieveNetworkTests) ... ok
test_header (test.test_urllibnet.urlretrieveNetworkTests) ... ok
test_specified_path (test.test_urllibnet.urlretrieveNetworkTests) ... ok

----------------------------------------------------------------------
Ran 12 tests in 3.116s

OK (skipped=1)
1 test OK.

@ohumbel

Okay, thx @ohumbel . Then I assume it is something with my specific system or network. Maybe I'll look into it another time. For now I'll just move on with the netty upgrade...

Let me know if I can help

It would be very pleasing if there were a way to un-pin this dependency. The networking in Jython based on netty is a considerable achievement (not mine), but fragile.

test test_urllibnet failed -- Traceback (most recent call last):
  File "...jython/dist/Lib/test/test_urllibnet.py", line 206, in test_data_header
    self.fail('Date value not in %r format', dateformat)
TypeError: fail() takes at most 2 arguments (3 given)

Looks like you have to fix the error reporting before you can find out what the error really was. Calling self.fail(ve.message) having caught the exception as ve looks promising.

Navigation Menu

Repository navigation

Updating Netty JARs leads to errors in `test_httplib` and others #349

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Updating Netty JARs leads to errors in test_httplib and others #349

Description

Activity

Stewori commented on Jul 23, 2024

jeff5 commented on Jul 23, 2024

Stewori commented on Jul 23, 2024

jeff5 commented on Jul 29, 2024

Stewori commented on Jul 29, 2024

Stewori commented on Jul 29, 2024

Stewori commented on Aug 20, 2024

jeff5 commented on Aug 21, 2024

wfouche commented on Feb 11, 2025

Stewori commented on Feb 11, 2025

wfouche commented on Feb 11, 2025

wfouche commented on Feb 11, 2025

jeff5 commented on May 3, 2025

wfouche commented on May 4, 2025

wfouche commented on May 4, 2025

Stewori commented on May 28, 2026

Stewori commented on May 28, 2026

Stewori commented on May 31, 2026

ohumbel commented on May 31, 2026

Stewori commented on May 31, 2026

ohumbel commented on May 31, 2026

jeff5 commented on Jun 3, 2026

jeff5 commented on Jul 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Updating Netty JARs leads to errors in `test_httplib` and others #349