Metaspace leak in 9.4 #8842
Description
Environment Information
$ ruby --version
jruby 9.4.12.0 (3.1.4) 2025-02-11 f4ab75096a OpenJDK 64-Bit Server VM 11.0.15+0-adhoc..source on 11.0.15+0-adhoc..source +jit [x86_64-linux]
$ java -version
openjdk version "11.0.15" 2022-04-19
OpenJDK Runtime Environment (build 11.0.15+0-adhoc..source)
OpenJDK 64-Bit Server VM (build 11.0.15+0-adhoc..source, mixed mode)
Behavior
Since upgrading from JRuby 9.3 to 9.4, we are seeing what appears to be unbounded Metaspace usage. This eventually leads to an OOM: Metaspace exception and process exit.
As an example, we can see this behavior in our production monitoring. When running 9.4, Total Classes and Loaded Classes closely track each other and tend to rise over time. The rate of increase tracks the daily usage of the instance. When running 9.3, Total Classes also rises, but Loaded Classes plateaus. This corresponds to a plateau in the total Metaspace usage, as classes appear to be GCed.
Running locally with class loader logging enabled (-Xlog:class+load=debug:loadedClasses-%t-%p.txt -Xlog:class+path=info:loadedClasses-%t-%p.txt -Xlog:class+unload=trace:loadedClasses-%t-%p.txt), I can see in 9.3 that there are both class,load and class,unload log lines.
[282.268s][info ][class,load ] org.apache.calcite.linq4j.tree.Expressions$FluentList source: file:/u
sr/local/google/home/kalenp/src/helltool/ruby-gems/ht_java/lib/jar/helltool-java-all.jar
[282.268s][debug][class,load ] klass: 0x00000008427f6728 super: 0x0000000800004460 interfaces: 0x00
000008002ec6d8 loader: [loader data: 0x00007fa79c96db40 for instance a 'org/jruby/util/JRubyClassLoad
er'{0x00000000a1c2c8b8}] bytes: 1188 checksum: fa4eb98b
[282.268s][info ][class,load ] org.apache.calcite.linq4j.tree.TernaryExpression source: file:/usr/lo
cal/google/home/kalenp/src/helltool/ruby-gems/ht_java/lib/jar/helltool-java-all.jar
[282.268s][debug][class,load ] klass: 0x00000008427f6928 super: 0x00000008425afa28 loader: [loader
data: 0x00007fa79c96db40 for instance a 'org/jruby/util/JRubyClassLoader'{0x00000000a1c2c8b8}] bytes:
3861 checksum: 750abf23
[282.269s][info ][class,load ] usr.local.google.home.kalenp.src.helltool.lib.helltool.models.encrypt
ed.db_connection source: __JVM_DefineClass__
[282.269s][debug][class,load ] klass: 0x00000008427da040 super: 0x0000000800004460 loader: [loader
data: 0x00007fa6bd586500 for instance a 'org/jruby/util/OneShotClassLoader'{0x0000000153874500}] byte
s: 4017 checksum: abf1c398
...
[282.274s][info ][class,unload] unloading class java.lang.invoke.LambdaForm$DMH/0x0000000841d97c40 0x
0000000841d97c40
[282.274s][info ][class,unload] unloading class java.lang.invoke.LambdaForm$MH/0x0000000841d95840 0x0
000000841d95840
[282.274s][info ][class,unload] unloading class java.lang.invoke.LambdaForm$MH/0x0000000841d35c40 0x0
000000841d35c40
[282.274s][info ][class,unload] unloading class java.lang.invoke.LambdaForm$MH/0x0000000841d32c40 0x0
000000841d32c40
[282.274s][info ][class,unload] unloading class java.lang.invoke.LambdaForm$DMH/0x0000000841cca840 0x
0000000841cca840
[282.274s][info ][class,unload] unloading class java.lang.invoke.LambdaForm$MH/0x0000000841ca9440 0x0
000000841ca9440
[282.274s][info ][class,unload] unloading class java.lang.invoke.LambdaForm$MH/0x0000000841ca7440 0x0
000000841ca7440
In 9.4, I see a similar pattern of class,load lines. However, for class,unload, I only see a few at startup and then a large number at application shutdown time. I don't see the same pattern of unloading at shutdown time in 9.3.