Repository navigation
Description
Fixes, by the roots listed below:
- jffi's
Cleanerthread (1)- Cleanup actions no longer reference the objects they clean up, so they run at all: Don't let cleanup actions reference the objects they clean up jnr/jffi#200
- Run cleanup actions without a cleaner thread, so no thread that outlives the class loader holds actions whose classes come from it: Run cleanup actions without a cleaner thread jnr/jffi#201
- JRuby uses a jffi release with both
- Thread-locals (2)
-
LibrarySearcher'skeyWrapperandrangeWrapperare plain fields: Don't cache LibrarySearcher's index wrappers in thread-locals #9894 -
Ruby.DEDUP_WRAPPER_CACHEholds its wrapper through a weak reference: Hold the fstring dedup wrapper weakly in its thread-local #9927
-
-
MethodGatherer'sClassValues only leave JDK types on JDK classes (3): Store MethodGatherer's filtered methods as JDK types #9895 -
StableClassValue.clear()also removes the values from their classes, soJavaSupport.tearDown()releases the Java proxy cache (4): Remove StableClassValue values from their classes when cleared #9922 -
JRubyClassLoader's temp dir no longer registers a shutdown hook whose class is in the runtime's class loader (5): Clean up the nested jar temp dir without a shutdown hook #9928, with tests in Test the nested jar temp dir cleanup #9945
Not planned: the RubyEncoding coder thread-locals hold soft references (JRUBY-6522), which the JVM clears under memory pressure.
Environment Information
- JRuby 10.1.2.0, 9.4.15.0 and 9.4.14.0 (
jruby-completefrom Maven Central), embedded withRuby.newInstance(), no flags orJRUBY_OPTS - OpenJDK 21.0.10 (BellSoft Liberica), Linux 7.0.0 x86_64
- Found through AsciidoctorJ 3.0.1 in the Asciidoctor Gradle plugin, which converts documents in Gradle workers with class loader isolation. Each worker gets a new class loader with JRuby on it, inside the long-lived Gradle daemon.
Expected Behavior
When a runtime is created in its own class loader, used, torn down with tearDown(), and the class loader is dropped, the class loader and the runtime can be garbage collected.
This program does that four times, and checks after each round how many of the class loaders are still alive:
import java.lang.management.ManagementFactory;
import java.lang.management.MemoryPoolMXBean;
import java.lang.ref.WeakReference;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
public class RuntimeLeak {
public static void main(String[] args) throws Exception {
URL[] jruby = {Path.of(args[0]).toUri().toURL()};
List<WeakReference<ClassLoader>> loaders = new ArrayList<>();
for (int round = 1; round <= 4; round++) {
loaders.add(new WeakReference<>(runOnce(jruby)));
for (int i = 0; i < 5; i++) {
System.gc();
Thread.sleep(200);
}
long alive = loaders.stream().filter(r -> r.get() != null).count();
System.out.printf("round %d: class loaders still alive: %d of %d, class space used: %d MB%n",
round, alive, round, classSpaceUsed() / (1024 * 1024));
}
}
static ClassLoader runOnce(URL[] jruby) throws Exception {
URLClassLoader loader = new URLClassLoader(jruby, ClassLoader.getPlatformClassLoader());
Class<?> rubyClass = loader.loadClass("org.jruby.Ruby");
Object runtime = rubyClass.getMethod("newInstance").invoke(null);
rubyClass.getMethod("evalScriptlet", String.class)
.invoke(runtime, "java.lang.StringBuilder.new('jruby').reverse.to_s.upcase");
rubyClass.getMethod("tearDown").invoke(runtime);
loader.close();
return loader;
}
static long classSpaceUsed() {
for (MemoryPoolMXBean pool : ManagementFactory.getMemoryPoolMXBeans()) {
if (pool.getName().equals("Compressed Class Space")) {
return pool.getUsage().getUsed();
}
}
return -1;
}
}Run it with java RuntimeLeak.java jruby-complete-10.1.2.0.jar.
Actual Behavior
None of the class loaders are collected, and each runtime keeps about 5 MB of class space:
round 1: class loaders still alive: 1 of 1, class space used: 6 MB
round 2: class loaders still alive: 2 of 2, class space used: 11 MB
round 3: class loaders still alive: 3 of 3, class space used: 16 MB
round 4: class loaders still alive: 4 of 4, class space used: 21 MB
9.4.15.0 and 9.4.14.0 give the same result. In a long-lived JVM that keeps creating runtimes this way, the class space and heap grow until the JVM runs out of them. In the Gradle daemon, a documentation build with 20 source directories, so 20 runtimes, added about 100 MB of class space per run. After a few runs the daemon stopped itself because the garbage collector was thrashing.
I looked at heap dumps with Eclipse MAT, following paths to GC roots without weak and soft references, and at the reference chains of Java Flight Recorder's old object samples. Heap dumps don't contain Class.classValueMap, so for ClassValues I also read the class value maps of all loaded classes through Instrumentation. These references keep the class loader alive:
-
jffi's
Cleanerthread.com.kenai.jffi.internal.Cleanercreates ajava.lang.ref.Cleanerper class loader, and its thread keeps the registered cleanup actions, such ascom.kenai.jffi.CallContext$1, whose class is in the runtime's class loader:java.net.URLClassLoader <- <classloader> class com.kenai.jffi.CallContext$1 <- <class> com.kenai.jffi.CallContext$1 <- action jdk.internal.ref.CleanerImpl$PhantomCleanableRef <- next jdk.internal.ref.CleanerImpl$PhantomCleanableRef <- phantomCleanableList jdk.internal.ref.CleanerImpl <- <Java Local> jdk.internal.misc.InnocuousThread "Cleaner-0"This is the shortest path with 10.1.2.0 and default settings.
-Djruby.native.enabled=falseremoves it. -
Values left in thread locals of the threads that used the runtime.
Ruby.DEDUP_WRAPPER_CACHE, andLibrarySearcher'skeyWrapperandrangeWrapper, are per-runtimeThreadLocals created withwithInitial:jruby/core/src/main/java/org/jruby/Ruby.java
Line 5120 in 09cff75
private final ThreadLocal<FStringEqual> DEDUP_WRAPPER_CACHE = ThreadLocal.withInitial(FStringEqual::new); jruby/core/src/main/java/org/jruby/runtime/load/LibrarySearcher.java
Lines 533 to 534 in 09cff75
private final ThreadLocal<StringWrapper> keyWrapper = ThreadLocal.withInitial(() -> new StringWrapper(null, 0, 0)); private final ThreadLocal<StringWrapper> rangeWrapper = ThreadLocal.withInitial(() -> new StringWrapper(null, 0, 0)); Nothing removes their values from those threads when the runtime is torn down.
freezeAndDedupStringonly removes theFStringEqualwrapper when it caches a new string, so after a lookup that finds an existing one, the wrapper stays on the thread:jruby/core/src/main/java/org/jruby/Ruby.java
Lines 4961 to 4986 in 09cff75
public RubyString freezeAndDedupString(RubyString string) { var context = getCurrentContext(); if (!string.isBare(context)) { // never cache a non-natural String string.setFrozen(true); return string; } // Populate thread-local wrapper try (FStringEqual wrapper = FStringEqual.cached(this, string)) { WeakReference<RubyString.FString> dedupedRef = dedupMap.get(wrapper); RubyString.FString deduped; if (dedupedRef == null || (deduped = dedupedRef.get()) == null) { // We will insert wrapper one way or another so clear from threadlocal DEDUP_WRAPPER_CACHE.remove(); // Ensure we have an FString deduped = string.dupAsFString(this); return cacheFString(deduped); } return deduped; } } The map entries reference their keys only weakly, but the values are instances of
FStringEqualandStringWrapper, whose classes are in the runtime's class loader. AThreadLocalMaponly drops such an entry when a later access to the same thread's map runs into it after the key has been collected. Until then, the value keeps the class loader alive. With 10.1.2.0, MAT shows this path for the last runtime of the program:java.net.URLClassLoader <- <classloader> class org.jruby.runtime.load.LibrarySearcher$StringWrapper <- <class> org.jruby.runtime.load.LibrarySearcher$StringWrapper <- value java.lang.ThreadLocal$ThreadLocalMap$Entry <- table java.lang.ThreadLocal$ThreadLocalMap <- threadLocals java.lang.Thread "main"With 9.4.14.0 and AsciidoctorJ, it was a
org.jruby.Ruby$FStringEqualon the same kind of path. Thread pools, like Gradle's worker threads, keep these threads and their maps for a long time. -
MethodGatherer's staticClassValues.FILTERED_DECLARED_METHODSandFILTERED_METHODSarejava.lang.ClassValues whose values areMethodGatherer$PartitionedMethods, a JRuby class:jruby/core/src/main/java/org/jruby/javasupport/binding/MethodGatherer.java
Lines 307 to 328 in 09cff75
private static final ClassValue<PartitionedMethods> FILTERED_DECLARED_METHODS = new ClassValue<PartitionedMethods>() { @Override public PartitionedMethods computeValue(Class cls) { return new PartitionedMethods(DECLARED_METHODS.get(cls)); } }; private static final ClassValue<Method[]> METHODS = new ClassValue<Method[]>() { @Override public Method[] computeValue(Class cls) { try { return cls.getMethods(); } catch (SecurityException se) { return EMPTY_METHODS; } } }; private static final ClassValue<PartitionedMethods> FILTERED_METHODS = new ClassValue<PartitionedMethods>() { @Override public PartitionedMethods computeValue(Class cls) { return new PartitionedMethods(METHODS.get(cls)); } }; A
ClassValuestores its value in the class it's computed for. Creating a runtime already computes them for JDK classes, so the JDK classes, which are never unloaded, keep thePartitionedMethodsvalues and through them the runtime's class loader. WithRuby.newInstance()andtearDown()alone, these JDK classes end up holding values for every runtime:java.lang.Object,java.lang.Class,java.lang.Enum,java.lang.Process,java.lang.ProcessBuilder,java.lang.ProcessBuilder$Redirect,java.lang.ProcessBuilder$Redirect$Type,java.lang.reflect.Method,java.lang.reflect.Field,java.lang.reflect.Constructorandjava.util.Map, among others. No option avoids this one. It doesn't show up in MAT because heap dumps don't containClass.classValueMap. -
The Java proxy cache. With the default
jruby.ji.class.values, the proxies are cached in aStableClassValueon the Java classes. With AsciidoctorJ on 9.4.14.0, Flight Recorder showed this path from a JDK class:org.jruby.RubyClass (runtime) <- result org.jruby.util.collections.StableClassValue$StableValue <- value java.lang.ClassValue$Entry <- cacheArray java.lang.ClassValue$ClassValueMap <- classValueMap java.lang.Class java.lang.ProcessBuilder$Redirect$Type-Djruby.ji.class.values=HARD_MAPremoves it. With this program on 10.1.2.0, the class loaders are only collected with that option, see below. -
The shutdown hook of
JRubyClassLoader.getTempDir(), which it registers when it creates the temporary directory for jars inside jars. It's an instance ofJRubyClassLoader$1, sojava.lang.ApplicationShutdownHookskeeps that class loader until the JVM exits. Since the hook is only registered when the directory is created, this pins the first class loader that adds such a jar, not every one. This program doesn't do that; AsciidoctorJ does.jruby/core/src/main/java/org/jruby/util/JRubyClassLoader.java
Lines 117 to 140 in 09cff75
public static synchronized File getTempDir() { if (tempDir != null) return tempDir; tempDir = new File(tempDir(), tempDirName()); if (tempDir.mkdirs()) { Runtime.getRuntime().addShutdownHook(new Thread() { public void run() { for (File f : tempDir.listFiles()) { try { f.delete(); } catch (Exception ex) { LOG.debug(ex); } } try { tempDir.delete(); } catch (Exception ex) { LOG.info("failed to delete temp dir " + tempDir + " : " + ex); } } }); } return tempDir; }
With roots 1 to 4 out of the way, the class loaders are collected. On 10.1.2.0 I ran the program with -Djruby.native.enabled=false and -Djruby.ji.class.values=HARD_MAP, ran each round in a new thread that ends afterwards, and then cleared, through reflection, the class value maps of the JDK classes that held MethodGatherer values. Afterwards all four class loaders were collected, and class space went from 20 MB back to 1 MB:
round 4: class loaders still alive: 4 of 4, class space used: 20 MB
after removing those values: class loaders still alive: 0 of 4, class space used: 1 MB
Without -Djruby.ji.class.values=HARD_MAP, the class loaders stayed alive after the same clearing.
#8842 is related: it's about classes leaking inside a long-lived runtime through the ClassValue caches, while this is about releasing a whole runtime together with its class loader.
Activity
headius commentedon Oct 7, 2026
Thank you for the report! Yeah, these thread local and class value roots are tricky to manage. I believe some of these are supposed to be cleared out by calling Ruby.tearDown, so it would be good to confirm that asciidoctorj is doing that.
The others will need attention.
matrei commentedon Oct 8, 2026
Thanks! The program in the description already calls
tearDown()on each runtime before it drops the class loader, so all the roots above are still there aftertearDown().AsciidoctorJ tears the runtime down in
Asciidoctor.shutdown()andclose(), which both callRuby.tearDown(). The Gradle plugin doesn't call either of them yet. I tried it withshutdown()added after each conversion, and the Gradle daemon kept the runtimes all the same: its class space grew by the same amount per build as without it.I'm happy to test a snapshot or a patch against both the program and the Gradle build.
matrei commentedon Oct 8, 2026
A correction to my last comment: the Gradle plugin does call it now. Since asciidoctor/asciidoctor-gradle-plugin#811, which was merged yesterday but isn't released yet, it closes the
Asciidoctorinstance at the end of each conversion, and that callsRuby.tearDown(). That doesn't change the results above, because the program in the description callstearDown()as well.matrei commentedon Oct 8, 2026
I tried fixing the roots, and with changes in both JRuby and jffi, the class loaders are collected.
JRuby (four commits on master):
LibrarySearcher's key and range wrappers are no longer cached in thread-locals. The dedup lookup creates a new wrapper, and theLibrarySearcherwrappers become plain fields, since all their accessors aresynchronizedanyway.MethodGatherer'sFILTERED_DECLARED_METHODSandFILTERED_METHODSstoreMethod[][]instead ofPartitionedMethods, so the values they leave on JDK classes are JDK types only.StableClassValue.clear()also callsremove(cls)on the underlyingjava.lang.ClassValuefor every class it computed a value for. Until now it only emptied theStableValueholders and left them in the classes, and the holder class is enough to keep the class loader alive. With this,JavaSupport.tearDown()releases the proxy cache, andjruby.ji.class.values=HARD_MAPis no longer needed.JRubyClassLoader.getTempDir()registers the directory and the copied jars withdeleteOnExit()instead of adding a shutdown hook.jffi (one commit):
com.kenai.jffi.internal.Cleanerno longer uses ajava.lang.ref.Cleaner. It keeps its own phantom references and reference queue, and runs the actions of enqueued references whenever the next object is registered. As long as a cleaner thread holds actions whose classes come from jffi's class loader, that class loader can't be collected, so I don't see a way to keep the thread. The trade-off is that a native resource is freed at the next registration instead of right away.While I was in there, I noticed that all the cleanup actions in jffi are anonymous classes that use
Outer.this(CallContext,Library,Aggregate,ClosureMagazine,ClosurePool.Magazine). That makes each registered object strongly reachable from its own action, so as far as I can tell the cleaner never runs them. I haven't measured that, and it's separate from this issue.Results. These numbers are from the program in the description, with all four runtimes on the main thread. To tell soft references apart from strong ones, I ran it with
-XX:SoftRefLRUPolicyMSPerMB=0, which clears soft references at every GC:Without the jffi change, the
Cleaner-0thread still holds every class loader. With both changes, without the flag, the only references left are the soft ones from theRubyEncodingcoder thread-locals (made soft for JRUBY-6522), which the JVM clears under memory pressure.In the Gradle daemon with AsciidoctorJ 3.0.1 (asciidoctor/asciidoctor-gradle-plugin#810), I used a build with 20 source directories, a 1.5 GB heap and
-XX:MaxMetaspaceSize=320m:-XX:SoftRefLRUPolicyMSPerMB=0, class space stayed at 19 MB from the second build on.In the daemon, the soft reference that keeps the older class loaders is mostly the JDK's
java.beansintrospection cache on the plugin's own classes, so that one isn't JRuby's.rake spec:jipasses with the JRuby changes (1008 examples, 0 failures, the same 9 pending as master).Would you like these as PRs, one to JRuby and one to jffi? The JRuby commits are one per root, so they can also go in separately.
headius commentedon Oct 19, 2026
Doing some preemptive review of your changes here. Separate PRs would be good, as these each are very tricky to get right.
JRuby changes:
The change to
LibrarySearcheris fine as long as the accesses are serialized (and slightly better because of less indirection). An alternative would be to have these be weakly held by the threadlocal, but that introduces an extra indirection and more dependence on GC to do the cleanup. We could also just add cleanup totearDown.The fstring wrapper change will introduce a lot of overhead allocating new instances every time. The better fix would be to add this ThreadLocal to
tearDownso it is actively cleared.Another possibility would be to implement our own ThreadLocal that doesn't store in a global location. It would formulate a key from a Thread but not reference the Thread directly. That could be managed locally in the objects consuming them or as a
Ruby-level cache that also includes eager cleanup. Perhaps a separate exercise to explore that.This change is fine, if unfortunate. I prefer to have the named PartitionedMethods abstraction, but the stickiness of ClassValue forces the CL to stay alive.
The Java 17-related deletions here should probably go into a separate cleanup commit/PR as they are not related.
ClassValue continues to be a pain in the butt. At this point we're basically implementing our own version of it, so I'm starting to feel like it's just a broken feature.
This might work for your test but a WeakReference means that the values (usually expensive-to-calculate items like a proxy class or a list of partitioned methods) will get collected very early and have to be recalculated repeatedly. At worst this should be a SoftReference, but ideally we'd find a way to clean these up at teardown.
The Method[][] for example would almost immediately be collected, forcing them to be recalculated continuously.
Proxy classes might live longer, especially if a Ruby object that references them stays alive, but for transient objects that enter Ruby and are never hard-referenced, we'll end up recalculating the proxy repeatedly.
This code used to use
deleteOnExitbut it was modified in #4578 because more and more entries were added to the internal list of deletes (leaking each time a new JRuby instance loads those same paths.However, my reading of the current
deleteOnExitcode is that it should only add the same path once (viaHashSet) so as long as additional runtimes used the same paths, it only represents a "soft" leak.Both versions will do no cleanup until exit, so that much stays the same. Clearly we want to eliminate the classloader reference here, but perhaps it would be better to register a shutdown hook that references only the paths instead of switching back to
deleteOnExit?JFFI change:
Fixes the issue but introduces a delayed cleaning. I get that the fix here is to avoid using global Cleaner stuff to avoid rooting the classloader until Cleaner runs, but we're trading one problem for another. Need to do some research on how to safely use Cleaner without rooting things lke classloaders.
--
We can discuss each of these in separate PRs. A few can probably merge as is, but others need more research.
headius commentedon Oct 22, 2026
This probably should be fixed in the 10.0 line since that's our current LTS branch and this represents an unexpected leak that can't be remedied by calling
tearDownalone.matrei commentedon Oct 26, 2026
Thanks for the review! I'll open separate PRs. Should they target
jruby-10.0, so they get merged forward to master, or master with backports?A few notes on the points, since some of them depend on details that aren't obvious from the summary.
Thread-locals. Clearing them in
tearDown()only helps the thread that callstearDown().ThreadLocal.remove()only removes the current thread's value, and every other thread that ever calledfreezeAndDedupStringkeeps itsFStringEqual, so it keeps the class loader alive. In Gradle those are the pooled worker threads, and none of them is guaranteed to be the one that tears the runtime down. Addingremove()totearDown()is cheap and worth doing, but it doesn't fix the leak by itself.To keep the reuse, the thread-local could hold a
WeakReference<FStringEqual>. Then the value in each thread's map is a JDK type, and the wrapper is only weakly reachable from the thread. When the wrapper is collected, the next lookup makes a new one, so it costs one allocation per thread per GC instead of one per call. Your idea of a per-runtime cache keyed by thread would also work. I can benchmark the dedup hit path with the current code, a new wrapper per call, and the weak holder, before we pick one.For
LibrarySearcherI'll keep the plain fields.MethodGatherer. The Java 17 logic isn't removed.
filterAccessiblewas a static method nested inPartitionedMethods, and it moved up one level unchanged, so the diff shows it as removed and re-added because of the indentation. I'll put the move in its own commit, or keepfilterAccessiblewhere it was and leave a small holder class, whichever you prefer.We can't clean these up at teardown, because they're static
ClassValues shared by every runtime in the class loader. One runtime'stearDown()can't remove them without affecting the others, and they only matter once the whole class loader goes away. With JDK-only values, what's left on JDK classes no longer references JRuby's class loader.StableClassValue. The computed values aren't held through a
WeakReference. They stay strongly referenced from theStableValuein each class's class value map, exactly as before, so nothing is recalculated more often than it is now. (TheMethod[][]is inMethodGatherer, which doesn't useStableClassValue.)The only
WeakReferenceis to theClass. It's stored as the value in thestableValuesside map, which used to benull, so thatclear()knows which class to callproxy.remove(cls)on. It has to be weak because a strong reference to the class would keep its key, theStableValue, reachable through the class's own class value map, and theWeakHashMapwould never drop it.clear()is already called fromJavaSupport.tearDown()throughproxyClassCache.clear(), so this is the teardown cleanup: it now also removes the entries from the classes instead of only emptying them.JRubyClassLoader temp dir. You're right that this brings back what #4578 fixed.
File.createTempFilegives each copied jar a new random name, so the paths aren't the same across runtimes, and thedeleteOnExitset grows with every runtime that adds a nested jar. That's one path string per copied jar, which is much less than a class loader, but it's still a leak.A shutdown hook that references only the paths still needs code to delete them, and any class JRuby defines for that, a lambda included, is loaded by JRuby's class loader. That's what keeps it alive today (
JRubyClassLoader$1). Some options:createTempFile. Runtimes that load the same nested jar then reuse the same file, anddeleteOnExitstores each path only once. This is the case you described.JRubyClassLoaderis closed. Sinceclose()doesn't always close the underlyingURLClassLoader(Errors running multiple independent ScriptingContainers in parallel #6218), the files can still be open, which is a problem on Windows.deleteOnExitas it is and accept the small growth.I'd go with 1, unless you see a problem with reusing the copies.
jffi Cleaner. First, a correction to my earlier comment, plus a separate problem. I wrote that five of the cleanup actions use
Outer.this. In the released 1.4.3 jar it's all seven: the two that only use constructor parameters (ClosurePool.MagazineHolderandNativeMethods.ResourceHolder) also get athis$0field from the Java 8 compiler. So each registered object is strongly reachable from its own action and never becomes phantom reachable. The cleanups never run, withjava.lang.ref.Cleaneror with my change.To check, I opened libc with
Library.openLibrary5,000 times in one runtime, dropped the results, ran GC, and counted the pending actions:ForeignSo today the choice isn't between prompt and delayed cleaning: native resources are never cleaned, and the class loader is pinned on top of that. I've opened the capture fix as its own PR, jnr/jffi#200, since it's independent of the class loader.
Second, I hoped that fixing the captures would also let us keep
java.lang.ref.Cleaner, because the thread would drop each action once it ran. It doesn't. With the captures fixed, the reproducer still keeps 4 of 4 class loaders, even with-XX:SoftRefLRUPolicyMSPerMB=0. After two rounds, each runtime'sCleanerthread still holds 36 actions (32CallContext, 3Libraryand 1NativeMethodsholder), and all of their objects are still alive. They're only reachable through the runtime's class loader: with the thread-less cleaner, the same objects are registered and collected together with the class loader. The cycle is:The object can't become phantom reachable while its action is pending, and the action stays pending until the object becomes phantom reachable. Any object that lives as long as the class loader does keeps it alive, and a runtime always creates some of them.
As far as I can tell, that holds for any thread that outlives the class loader and holds actions whose classes come from it. The JDK's shared cleaner has the same problem. That leaves running the actions on threads jffi doesn't own, as my change does. With the capture fix and the thread-less cleaner together, the reproducer gets 0 of 4 class loaders alive with
-XX:SoftRefLRUPolicyMSPerMB=0. The remaining references are the soft ones from theRubyEncodingthread-locals, as before.With either change alone or both together, jffi's tests (122) and JRuby's
spec/ffi(4,990 examples, 2 pending) pass on Linux x86_64. They also pass with a small heap and serial GC, so collections and cleanups happen during the run. The results are the same as with jffi 1.4.3.On the delay: the queue is drained on every registration, which means whenever a new
CallContext,Library, etc. is created. To make it less dependent on that, the queue could also be drained in other frequently called places, or jffi could add a publicCleaner.clean()that JRuby calls fromtearDown(). I'll wait for your take on the delay before I open the jffi PR for the thread-less cleaner, so I can include whichever of these you prefer.matrei commentedon Oct 26, 2026
Here's the benchmark of the fstring dedup hit path. It's a JMH benchmark calling
Ruby.freezeAndDedupStringwith a bare, unfrozen string whose fstring is already in the dedup table, asHash#[]=does with a String key it has seen before. All three variants are based on current master:ThreadLocal<FStringEqual>FStringEqualper lookup, as in my branchThreadLocal<WeakReference<FStringEqual>>, so a thread's map only holds a JDK type after the runtime is torn down. When the wrapper is collected, the next lookup makes a new one.Average time per call in ns, JMH 1.37 with 5 forks of 10 one-second iterations, on JDK 21 on a 2-socket Xeon Gold 6240:
The new wrapper is about 8% faster, because a thread-local lookup costs more than a 16-byte allocation. But it allocates on every hit, and escape analysis doesn't remove that: there were 35–123 GCs while measuring, against 0–2 for the others. The weak holder costs about 4% and allocates nothing.
* With 8 threads on one key, the thread-local variant sometimes stays several times slower for a whole fork after a GC moves its wrapper: 2 of 10 forks averaged 336 and 441 ns, against about 95 ns for the rest. Across further runs it showed up in 0 to 8 of 15 forks, depending on details of the benchmark harness. Padding the wrapper before and after its field prevented it, so it looks like false sharing, but I haven't found with which object. Neither of the other two variants showed it.
I also checked the class loader with the reproducer: with all the other fixes in place, the thread-local version still keeps a class loader alive, while both the new wrapper and the weak holder release all four.
So I'd go with the weak holder for the fstring wrapper: it keeps the reuse, doesn't allocate, and fixes the leak. If you'd rather avoid the extra indirection, the per-runtime cache keyed by thread that you suggested would also work.
headius commentedon Oct 27, 2026
The jffi fix in jnr/jffi#200 (fixing jnr/jffi#199) has been merged and I'll update all JNR artifacts for JRuby 10.0.8.0 and 10.1.3.0 releases.
headius commentedon Oct 27, 2026
The ClassValue fix has been merged from #9895.
headius commentedon Oct 27, 2026
There's another possible fix: synchronized logic that reuses a single
FStringEqualwhile it is used for the lookup. The wrapper is only used for a brief lookup and is reset to null afterwards. The cost of that short synchronization may beat all of the other options and probably won't impact typical code.headius commentedon Oct 27, 2026
The LibrarySearcher wrapper fix is merged: #9894
headius commentedon Oct 27, 2026
Delayed cleanup is obviously better than the old behavior of not cleaning up at all. I am not particularly concerned about lazy cleanup having some visible impact, since anyone concerned about lazy cleanup should be actively freeing their pointers. Lazy because we defer cleanup is no different in my eyes than lazy because GC hasn't run yet, and nobody should expect guarantees in either case (you walk away from allocated memory, you're implicitly letting us decide when it frees).
headius commentedon Oct 27, 2026
@matrei Update this issue with a checklist of specific fixes, so I know where we stand on progress. Thank you for breaking up the fixes into logical chunks... I'll continue to review and merge as they come in.
I am hoping to release JRuby 10.0.8.0 and 10.1.3.0 later this week, so if we can get these changes wrapped up ASAP that would be great!
matrei commentedon Oct 28, 2026
Thanks for merging these! I've added a checklist to the description.
One thing for the JNR update: jnr/jffi#200 alone doesn't release the class loader. With it, the reproducer still keeps 4 of 4, because the
Cleanerthread holds the actions of objects that live as long as the class loader. That takes the thread-less cleaner as well, so it would be good to have both in the jffi release for 10.0.8.0. Now that you're fine with the delay, I've opened it as jnr/jffi#201. It also runs the actions one at a time, as a cleaner thread would, becauseClosurePool'sRecyclerregisters an object while it holds the pool's lock, and two threads running actions at the same time could deadlock.For the fstring wrapper, I'll benchmark a single synchronized wrapper against the weak holder, with one thread and with several, since
Hash#[]=goes through it for every unfrozen String key, on every thread.The
StableClassValuechange will get its own PR. It's the one where only theClassis held weakly, not the values, as in my comment above.For the temp dir, I'll open a PR with option 1 from my comment above, naming the copies after the source URL, unless you'd prefer another option.
headius commentedon Oct 28, 2026
No problem, I've merged jnr/jffi#201 and it will be in the upcoming releases.
headius commentedon Oct 28, 2026
Another alternative trick: a single cached fstring wrapper "stolen" with an atomic swap. Any other thread that comes in needing a wrapper allocates their own. They each atomically try to swap theirs back in but only one wins. Doesn't matter because it's cleared by then anyway.
This design makes single-thread calls as fast as an atomic swap of a reference (pretty fast), and multi-thread calls slow down only if they happen to see a null fstring wrapper. It's basically a single-object pool that spills into normal allocation if that instance is in use.
headius commentedon Oct 28, 2026
Ok revisiting your reply here...
This only works if we can (semi-)atomically write these files out and verify/load them. Multiple runtimes starting up in multiple threads might all try to write that file at the same time, and then we have a sort of TOC-to-TOU issue where one thread might complete its write but then try to load the partial file from another thread. It makes these jars a cross-thread shared resource and that's obviously sticky.
Security-wise, this also makes the filename predictable, which could be exploited (TOC-to-TOU again).
Yeah also problematic, both because of the unreliable closing of jar resources (I've chased that tail before and it's a mess) and because Windows won't delete files that are open (Windows! 😡). At this point we're talking about limitations of URLClassLoader and its underlying jar loading logic, though.
Undesirable, but perhaps we can mitigate this somewhat. It also doesn't solve the Windows problem, but that's been there for a long time without anyone coming up with a resolution.
I propose a combination of these that mitigates most of the issues:
deleteOnExit. This will be all of the files on Windows, most likely, but on UNIX the deletes should all succeed and nothing leaks.The eager deletion + registration of undeletable files could be shunted to a Cleaner (to speed up teardown) since we can basically just feed it a ready-to-clean file-scrubbing function and walk away. This would, however, keep the ClassLoader alive a bit longer until the Cleaner can run our scrubber.
Windows remains a problem but Windows is always a problem (and the current logic doesn't work on Windows either).
I think we all agree we can't safely use an exit hook anymore since it will always keep a hard reference to the classloader.
headius commentedon Oct 28, 2026
Updates:
I believe we're just down to the JRubyClassLoader jar file cleanup and the fstring wrapper (other than getting jffi and friends updated in JRuby).
matrei commentedon Oct 28, 2026
I benchmarked both of your ideas against the weak holder, on
jruby-10.0: the single synchronized wrapper and the single wrapper taken with an atomic swap. Both fix the leak, but in both, every lookup in the runtime goes through the same object, and that costs more than it saves as soon as a second thread does lookups.These are
freezeAndDedupStringhits, cycling through 1,024 unfrozen String keys that are already in the table, in ns per lookup:That loop does nothing but lookups, which is the worst case for a shared object. With about 490 ns of other work before each lookup, the lookup itself took:
For the atomic swap, the slowdown doesn't come from threads finding the wrapper gone. With 2 threads, only about 1 in 8 lookups had to allocate one, and a lookup still took more than twice as long. Every lookup on every thread does an atomic swap on the same reference, so its cache line keeps moving between cores. Putting the wrapper back with a plain release store instead of
compareAndSetdidn't change much: 244 ns at 2 threads and 3,287 at 32. On one thread, the two atomic operations are also slower than theThreadLocal.get()they replace.On one thread, none of the four allocates. The atomic swap allocates when threads collide, on average 8 bytes per lookup at 8 and 32 threads, so about every second lookup. With the other fixes and
-XX:SoftRefLRUPolicyMSPerMB=0, the reproducer releases all four class loaders with the weak holder, the atomic swap and the synchronized wrapper, and keeps 1 of 4 with the current thread-local.So I'd still go with the weak holder. It's the only option where threads don't share anything, it costs 2–7% on the lookup alone and about 1% with other work in between, and it doesn't allocate. I'll open a PR with it, unless you'd prefer something else.
The numbers are JMH averages over 3 forks, on JDK 21 and a 72-core machine. The synchronized wrapper and the atomic swap ran an hour apart. The current thread-local and the weak holder ran both times and matched within 2%, and the tables show the second run.
headius commentedon Oct 28, 2026
The numbers are not too surprising. I knew the synchronized and atomic swap versions would have more contention under concurrent load. It's an open question how likely this sort of load would be during normal Ruby execution.
Given the consistency of its numbers and simplicity of the change, the weak version does appear to be the safest choice for now.
It would be interesting to explore one final option, though it's a much larger implementation: a custom map designed for FString that does what the wrapper does by default. It would need to be concurrency-safe but it would eliminate this entire issue. A similar case for LoadSearcher would also make sense.
For now let's get the simple fix in place so we don't hold up getting some releases out in the next week.
matrei commentedon Oct 29, 2026
I found another root while looking at jnr/jnr-posix#203: signal handlers that Ruby code installs with
Signal.trap. It doesn't show up in the reproducer, because nothing there traps a signal, but it does whenever the embedded Ruby code does, as servers commonly do for TERM and INT.tearDown()only resets theEXITtrap. A handler installed for a real signal stays in the JDK's static signal table after the runtime is torn down, and keeps the runtime and its class loader alive. It also keeps every earlier runtime alive that trapped the same signal.RubySignal.SIGNAL_FACADEis static, so there's oneSunSignalFacadeper JRuby class loader, and itsoriginalmap records the handler that was installed before that class loader's first trap. For the second runtime, that's the first runtime's handler:jdk.internal.misc.Signal.handlers→sun.misc.Signal$InternalMiscHandler→ runtime 4'sJRubySignalHandler→ its class loader →RubySignal.SIGNAL_FACADE→original→ runtime 3'sJRubySignalHandler→ its class loader → … → runtime 1's class loaderWith the reproducer from the description, plus
Signal.trap('TERM') { puts 'runtime 1 trapped SIGTERM' }(with each runtime's own number) in each runtime's script, all the fixes from the checklist including jffi 1.4.4, and-XX:SoftRefLRUPolicyMSPerMB=0, these class loaders were still alive after four rounds:Signal.trap('TERM', 'DEFAULT')beforetearDown()It's not only memory, though:
'DEFAULT'restores the handler fromoriginal, so with more than one JRuby class loader, it can restore another runtime's handler instead of the JVM's. When only runtime 4 restored'DEFAULT', SIGTERM ran runtime 3's block, and 3 of 4 class loaders were still alive.masterhas the same code.I'd fix this by tracking the traps per runtime instead of per class loader, and in
tearDown(), putting back the handler that was installed before the runtime's first trap of each signal, but only if the runtime's own handler is still the installed one. Runtimes torn down in a different order than they trapped would still need care: the handler to put back could belong to a runtime that's already been torn down. Would you like a PR along those lines, and should I add it to the checklist as (6)?matrei commentedon Oct 29, 2026
With everything on the checklist merged,
jruby-10.0(5a089a3) built with jffi 1.4.4 releases all four class loaders in the reproducer: 0 of 4 with-XX:SoftRefLRUPolicyMSPerMB=0, with and without a nested jar. With jffi 1.4.3 it's 4 of 4. So for the reproducer, only the jffi update is left.The reproducer only reverses a string, though. So I changed it to run a given piece of Ruby code in each of the four runtimes, and ran it for a list of common operations, each in its own JVM, with the same build and flag:
system, backticks,IO.popen,Process.spawnFile.utime, soFileUtils.touchtooFFI::MemoryPointer.new,io/consoleqsortwith a block)IO.pipe, write,IO.select,read_nonblockrequire 'openssl'and a digestThese were 0 of 4 on both:
File.stat, reading and writing files,File.chmod, symlinks,flock,File.mkfifo,File.umask,Dir.globandDir.chdir,ENV,Process.pid,Process.times,Process.getrlimit,Process.kill(0, …),Etc, TCP and UNIX sockets,Thread.new,Signal.trapfollowed by'DEFAULT',SecureRandom,jsonanddate, and an FFI call without a callback.The roots, from heap dumps:
A. jnr-ffi's reference queue thread, on JDK 21. The first
FinalizableReferenceQueuein a class loader starts ajnr.ffi.util.ref.internal.Finalizerthread. Both jnr-ffi and JRuby's own FFI (org.jruby.ext.ffi.jffi.NativeFinalizer) create one. A new thread keeps the access control context of the code that creates it:Finalizerthread →inheritedAccessControlContext→ProtectionDomain→ JRuby's class loaderCreating the thread in
doPrivilegedisn't enough; I tried it. jnr-ffi loadsFinalizerin its ownURLClassLoader, so that the thread doesn't hold the application's class loader. But on JDK 21 aURLClassLoaderalso keeps its creator's context, inacc, so the path becomesFinalizer→ its class loader →acc→ProtectionDomain→ JRuby's class loader. JDK 25 has neither field, and there the thread stops once the class loader is collected. Until then it also keeps running cleanups after the runtime is torn down. Once the class loader is closed, they fail withSEVERE: Error cleaning up after referenceand aNoClassDefFoundErrorforCachingNativeMemoryAllocator$Bucket$CacheElement.B. jnr-ffi's
TransientNativeMemory.currentMagazine, on any JDK. It's a staticThreadLocal<Magazine>. The thread that made the call keeps the magazine, and through its class, the class loader. jnr-ffi allocates from it for small direct memory, for example:posix_spawnp(CharSequenceArrayParameterConverter)Struct.arrayOffor the twoTimespecs ofutimensatThe thread-local is only a cache, because
referenceSetkeeps each magazine until it's freed. So it could hold aWeakReference, the way JRuby's ownCachingNativeMemoryAllocatordoes.C. JRuby's
PosixShim.errno, on any JDK. EachPosixShim, and so eachOpenFile, has had its ownThreadLocal<Errno>since a141c69, for #5706. When the IO is collected, its entry in the thread's map goes stale. The entry's value, ajnr.constants.platform.Errno, stays until the map clears stale entries, and holds the class loader until then. That's presumably why one of the four was released. Storing the errno number instead, or aWeakReference<Errno>, would keep JRuby's classes out of the thread's map.D. BouncyCastle 1.86, on any JDK. The static initializer of
CryptoServicesRegistrarsets its default parameters as global properties.localSetGlobalPropertyalso puts them in the staticthreadPropertiesthread-local, for the current thread:thread's map →
HashMap→DSAParameters[]→ BouncyCastle's class loader (the runtime'sJRubyClassLoader) → its parent, JRuby's class loaderThat's in BouncyCastle. It has
clearThreadProperty, so jruby-openssl might be able to clear the properties after initializing, but I haven't tried that.E. jffi closures, on any JDK. A JNI global reference to
com.kenai.jffi.ClosurePool$Proxyholds the class loader. It's the same kind of root as the jnr-ffi closures in jnr/jnr-posix#203. I haven't looked for a fix yet.To check A and B, I patched two jnr-ffi 2.3.4 classes into the jar:
currentMagazineholds aWeakReference<Magazine>.FinalizableReferenceQueuenever starts the thread. It already falls back to that when it can't start the thread: every new reference then drains the queue.With both,
system,File.utime,FFI::MemoryPointerandio/consolewere 0 of 4 on JDK 21 and 25, and theSEVERElogs were gone. A real change would need the single drainer from jnr/jffi#201, though. JRuby'sBucket.recyclecreates aFinalizableWeakReferencewhile it holds the bucket's lock, so draining from the reference's constructor would run other cleanups under that lock, and recursively.jnr-ffi still needs a release for the dependency update. Would you like a jnr-ffi PR for A and B in time for that release, and a JRuby PR for C against
jruby-10.0? D and E could be follow-ups. And should I add these to the checklist, along with thetraproot?headius commentedon Oct 29, 2026
I'm not a fan of scope creep in a single issue. Would you say that your original concerns have been address sufficiently, or are the additional roots you've recently discovered still necessary?
I ask because we're probably chasing a very long tail of classloader roots along the weird edges of JRuby and its component libraries. Some, like the issues you found in Bouncy Castle, which may be difficult for us to fix.
I'd like to say this issue has been addressed, and perhaps open a discussion with the remaining items for us to resolve individually.
Thoughts?
matrei commentedon Oct 29, 2026
Yes, the original case is covered. I ran the reproducer from asciidoctor/asciidoctor-gradle-plugin#810 in one Gradle daemon, with plugin 5.1.0, which closes AsciidoctorJ after converting, and soft references cleared:
jruby-10.0+ jffi 1.4.4jruby-10.0+ jffi 1.4.3With default settings, the old class loaders are only softly reachable. With Gradle's default 384 MB metaspace cap, metaspace dropped back when it filled up, and 15 builds passed. With 9.4.14.0, the daemon died at build 11.
So the jffi update is all that's left here, and I agree with closing this once it's in. I'm happy to open a discussion with the remaining items, apart from
trap, since you're handling that. The jnr-ffi PR for A and B is ready whenever you want it.headius commentedon Oct 29, 2026
@matrei I'm considering options for trap but please open a discussion and include it as one of the outstanding items. We might as well completely cover all known cases (and keep adding newly-discovered cases) and wiggle our way toward fixing this for all JRuby use cases.
I will mark this as complete once the jffi merge has fully landed (still evaluating unrelated PRs in the JNR stack).
headius commentedon Oct 30, 2026
Fixed by numerous items!