Netty 'IllegalStateException: executor not accepting a task' #632
Description
Hi,
I'm experiencing what looks like netty/netty#5313 when changing Testcontainers to use a single shared netty-based DockerClient (as #536 suggests should work). (This change is to avoid testcontainers/testcontainers-java#170, where we're getting apparent file descriptor exhaustion from opening too many netty-based docker clients.)
After switching to a single docker client instance, I'm getting this stack trace reliably on the second container creation:
org.testcontainers.containers.ContainerLaunchException: Container startup failed
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:163)
at org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:486)
at org.testcontainers.containers.FailureDetectingExternalResource$1.evaluate(FailureDetectingExternalResource.java:28)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:253)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:220)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:160)
... 9 more
Caused by: java.lang.IllegalStateException: executor not accepting a task
at io.netty.resolver.AddressResolverGroup.getResolver(AddressResolverGroup.java:58)
at io.netty.bootstrap.Bootstrap.doResolveAndConnect(Bootstrap.java:167)
at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:142)
at io.netty.bootstrap.Bootstrap.connect(Bootstrap.java:123)
at com.github.dockerjava.netty.DockerCmdExecFactoryImpl$InetSocketInitializer.connect(DockerCmdExecFactoryImpl.java:271)
at com.github.dockerjava.netty.DockerCmdExecFactoryImpl.connect(DockerCmdExecFactoryImpl.java:209)
at com.github.dockerjava.netty.DockerCmdExecFactoryImpl.connect(DockerCmdExecFactoryImpl.java:202)
at com.github.dockerjava.netty.DockerCmdExecFactoryImpl.access$000(DockerCmdExecFactoryImpl.java:150)
at com.github.dockerjava.netty.DockerCmdExecFactoryImpl$1.getChannel(DockerCmdExecFactoryImpl.java:176)
at com.github.dockerjava.netty.InvocationBuilder.getChannel(InvocationBuilder.java:155)
at com.github.dockerjava.netty.InvocationBuilder.post(InvocationBuilder.java:287)
at com.github.dockerjava.netty.InvocationBuilder.post(InvocationBuilder.java:278)
at com.github.dockerjava.netty.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:31)
at com.github.dockerjava.netty.exec.CreateContainerCmdExec.execute(CreateContainerCmdExec.java:13)
at com.github.dockerjava.netty.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
at com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
at com.github.dockerjava.core.command.CreateContainerCmdImpl.exec(CreateContainerCmdImpl.java:177)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:179)
... 10 more
I thought this might be resolvable by upgrading docker-java to use the 4.1.1.Final (or later) version of netty, but it seems to still occur.
I'll try and follow up with a proper isolated piece of code to reproduce, but thought I'd raise now in case it looks familiar to anybody else...
Any ideas about what could be going wrong?
Thanks
Richard