Skip to content

A runtime in a disposable class loader can't be garbage collected after tearDown #9781

Closed

Description

@matrei

Fixes, by the roots listed below:

Not planned: the RubyEncoding coder thread-locals hold soft references (JRUBY-6522), which the JVM clears under memory pressure.


Environment Information

  • JRuby 10.1.2.0, 9.4.15.0 and 9.4.14.0 (jruby-complete from Maven Central), embedded with Ruby.newInstance(), no flags or JRUBY_OPTS
  • OpenJDK 21.0.10 (BellSoft Liberica), Linux 7.0.0 x86_64
  • Found through AsciidoctorJ 3.0.1 in the Asciidoctor Gradle plugin, which converts documents in Gradle workers with class loader isolation. Each worker gets a new class loader with JRuby on it, inside the long-lived Gradle daemon.

Expected Behavior

When a runtime is created in its own class loader, used, torn down with tearDown(), and the class loader is dropped, the class loader and the runtime can be garbage collected.

This program does that four times, and checks after each round how many of the class loaders are still alive:

import java.lang.management.ManagementFactory;
import java.lang.management.MemoryPoolMXBean;
import java.lang.ref.WeakReference;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;

public class RuntimeLeak {

    public static void main(String[] args) throws Exception {
        URL[] jruby = {Path.of(args[0]).toUri().toURL()};
        List<WeakReference<ClassLoader>> loaders = new ArrayList<>();
        for (int round = 1; round <= 4; round++) {
            loaders.add(new WeakReference<>(runOnce(jruby)));
            for (int i = 0; i < 5; i++) {
                System.gc();
                Thread.sleep(200);
            }
            long alive = loaders.stream().filter(r -> r.get() != null).count();
            System.out.printf("round %d: class loaders still alive: %d of %d, class space used: %d MB%n",
                round, alive, round, classSpaceUsed() / (1024 * 1024));
        }
    }

    static ClassLoader runOnce(URL[] jruby) throws Exception {
        URLClassLoader loader = new URLClassLoader(jruby, ClassLoader.getPlatformClassLoader());
        Class<?> rubyClass = loader.loadClass("org.jruby.Ruby");
        Object runtime = rubyClass.getMethod("newInstance").invoke(null);
        rubyClass.getMethod("evalScriptlet", String.class)
            .invoke(runtime, "java.lang.StringBuilder.new('jruby').reverse.to_s.upcase");
        rubyClass.getMethod("tearDown").invoke(runtime);
        loader.close();
        return loader;
    }

    static long classSpaceUsed() {
        for (MemoryPoolMXBean pool : ManagementFactory.getMemoryPoolMXBeans()) {
            if (pool.getName().equals("Compressed Class Space")) {
                return pool.getUsage().getUsed();
            }
        }
        return -1;
    }
}

Run it with java RuntimeLeak.java jruby-complete-10.1.2.0.jar.

Actual Behavior

None of the class loaders are collected, and each runtime keeps about 5 MB of class space:

round 1: class loaders still alive: 1 of 1, class space used: 6 MB
round 2: class loaders still alive: 2 of 2, class space used: 11 MB
round 3: class loaders still alive: 3 of 3, class space used: 16 MB
round 4: class loaders still alive: 4 of 4, class space used: 21 MB

9.4.15.0 and 9.4.14.0 give the same result. In a long-lived JVM that keeps creating runtimes this way, the class space and heap grow until the JVM runs out of them. In the Gradle daemon, a documentation build with 20 source directories, so 20 runtimes, added about 100 MB of class space per run. After a few runs the daemon stopped itself because the garbage collector was thrashing.

I looked at heap dumps with Eclipse MAT, following paths to GC roots without weak and soft references, and at the reference chains of Java Flight Recorder's old object samples. Heap dumps don't contain Class.classValueMap, so for ClassValues I also read the class value maps of all loaded classes through Instrumentation. These references keep the class loader alive:

  1. jffi's Cleaner thread. com.kenai.jffi.internal.Cleaner creates a java.lang.ref.Cleaner per class loader, and its thread keeps the registered cleanup actions, such as com.kenai.jffi.CallContext$1, whose class is in the runtime's class loader:

    java.net.URLClassLoader
    <- <classloader> class com.kenai.jffi.CallContext$1
    <- <class> com.kenai.jffi.CallContext$1
    <- action jdk.internal.ref.CleanerImpl$PhantomCleanableRef
    <- next jdk.internal.ref.CleanerImpl$PhantomCleanableRef
    <- phantomCleanableList jdk.internal.ref.CleanerImpl
    <- <Java Local> jdk.internal.misc.InnocuousThread "Cleaner-0"
    

    This is the shortest path with 10.1.2.0 and default settings. -Djruby.native.enabled=false removes it.

  2. Values left in thread locals of the threads that used the runtime. Ruby.DEDUP_WRAPPER_CACHE, and LibrarySearcher's keyWrapper and rangeWrapper, are per-runtime ThreadLocals created with withInitial:

    private final ThreadLocal<FStringEqual> DEDUP_WRAPPER_CACHE = ThreadLocal.withInitial(FStringEqual::new);

    private final ThreadLocal<StringWrapper> keyWrapper = ThreadLocal.withInitial(() -> new StringWrapper(null, 0, 0));
    private final ThreadLocal<StringWrapper> rangeWrapper = ThreadLocal.withInitial(() -> new StringWrapper(null, 0, 0));

    Nothing removes their values from those threads when the runtime is torn down. freezeAndDedupString only removes the FStringEqual wrapper when it caches a new string, so after a lookup that finds an existing one, the wrapper stays on the thread:

    public RubyString freezeAndDedupString(RubyString string) {
    var context = getCurrentContext();
    if (!string.isBare(context)) {
    // never cache a non-natural String
    string.setFrozen(true);
    return string;
    }
    // Populate thread-local wrapper
    try (FStringEqual wrapper = FStringEqual.cached(this, string)) {
    WeakReference<RubyString.FString> dedupedRef = dedupMap.get(wrapper);
    RubyString.FString deduped;
    if (dedupedRef == null || (deduped = dedupedRef.get()) == null) {
    // We will insert wrapper one way or another so clear from threadlocal
    DEDUP_WRAPPER_CACHE.remove();
    // Ensure we have an FString
    deduped = string.dupAsFString(this);
    return cacheFString(deduped);
    }
    return deduped;
    }
    }

    The map entries reference their keys only weakly, but the values are instances of FStringEqual and StringWrapper, whose classes are in the runtime's class loader. A ThreadLocalMap only drops such an entry when a later access to the same thread's map runs into it after the key has been collected. Until then, the value keeps the class loader alive. With 10.1.2.0, MAT shows this path for the last runtime of the program:

    java.net.URLClassLoader
    <- <classloader> class org.jruby.runtime.load.LibrarySearcher$StringWrapper
    <- <class> org.jruby.runtime.load.LibrarySearcher$StringWrapper
    <- value java.lang.ThreadLocal$ThreadLocalMap$Entry
    <- table java.lang.ThreadLocal$ThreadLocalMap
    <- threadLocals java.lang.Thread "main"
    

    With 9.4.14.0 and AsciidoctorJ, it was a org.jruby.Ruby$FStringEqual on the same kind of path. Thread pools, like Gradle's worker threads, keep these threads and their maps for a long time.

  3. MethodGatherer's static ClassValues. FILTERED_DECLARED_METHODS and FILTERED_METHODS are java.lang.ClassValues whose values are MethodGatherer$PartitionedMethods, a JRuby class:

    private static final ClassValue<PartitionedMethods> FILTERED_DECLARED_METHODS = new ClassValue<PartitionedMethods>() {
    @Override
    public PartitionedMethods computeValue(Class cls) {
    return new PartitionedMethods(DECLARED_METHODS.get(cls));
    }
    };
    private static final ClassValue<Method[]> METHODS = new ClassValue<Method[]>() {
    @Override
    public Method[] computeValue(Class cls) {
    try {
    return cls.getMethods();
    } catch (SecurityException se) {
    return EMPTY_METHODS;
    }
    }
    };
    private static final ClassValue<PartitionedMethods> FILTERED_METHODS = new ClassValue<PartitionedMethods>() {
    @Override
    public PartitionedMethods computeValue(Class cls) {
    return new PartitionedMethods(METHODS.get(cls));
    }
    };

    A ClassValue stores its value in the class it's computed for. Creating a runtime already computes them for JDK classes, so the JDK classes, which are never unloaded, keep the PartitionedMethods values and through them the runtime's class loader. With Ruby.newInstance() and tearDown() alone, these JDK classes end up holding values for every runtime: java.lang.Object, java.lang.Class, java.lang.Enum, java.lang.Process, java.lang.ProcessBuilder, java.lang.ProcessBuilder$Redirect, java.lang.ProcessBuilder$Redirect$Type, java.lang.reflect.Method, java.lang.reflect.Field, java.lang.reflect.Constructor and java.util.Map, among others. No option avoids this one. It doesn't show up in MAT because heap dumps don't contain Class.classValueMap.

  4. The Java proxy cache. With the default jruby.ji.class.values, the proxies are cached in a StableClassValue on the Java classes. With AsciidoctorJ on 9.4.14.0, Flight Recorder showed this path from a JDK class:

    org.jruby.RubyClass (runtime)
    <- result org.jruby.util.collections.StableClassValue$StableValue
    <- value java.lang.ClassValue$Entry
    <- cacheArray java.lang.ClassValue$ClassValueMap
    <- classValueMap java.lang.Class java.lang.ProcessBuilder$Redirect$Type
    

    -Djruby.ji.class.values=HARD_MAP removes it. With this program on 10.1.2.0, the class loaders are only collected with that option, see below.

  5. The shutdown hook of JRubyClassLoader.getTempDir(), which it registers when it creates the temporary directory for jars inside jars. It's an instance of JRubyClassLoader$1, so java.lang.ApplicationShutdownHooks keeps that class loader until the JVM exits. Since the hook is only registered when the directory is created, this pins the first class loader that adds such a jar, not every one. This program doesn't do that; AsciidoctorJ does.

    public static synchronized File getTempDir() {
    if (tempDir != null) return tempDir;
    tempDir = new File(tempDir(), tempDirName());
    if (tempDir.mkdirs()) {
    Runtime.getRuntime().addShutdownHook(new Thread() {
    public void run() {
    for (File f : tempDir.listFiles()) {
    try {
    f.delete();
    } catch (Exception ex) {
    LOG.debug(ex);
    }
    }
    try {
    tempDir.delete();
    } catch (Exception ex) {
    LOG.info("failed to delete temp dir " + tempDir + " : " + ex);
    }
    }
    });
    }
    return tempDir;
    }

With roots 1 to 4 out of the way, the class loaders are collected. On 10.1.2.0 I ran the program with -Djruby.native.enabled=false and -Djruby.ji.class.values=HARD_MAP, ran each round in a new thread that ends afterwards, and then cleared, through reflection, the class value maps of the JDK classes that held MethodGatherer values. Afterwards all four class loaders were collected, and class space went from 20 MB back to 1 MB:

round 4: class loaders still alive: 4 of 4, class space used: 20 MB
after removing those values: class loaders still alive: 0 of 4, class space used: 1 MB

Without -Djruby.ji.class.values=HARD_MAP, the class loaders stayed alive after the same clearing.

#8842 is related: it's about classes leaking inside a long-lived runtime through the ClassValue caches, while this is about releasing a whole runtime together with its class loader.

Activity

  1. headius commented on Oct 7, 2026

    @headius
    Member

    Thank you for the report! Yeah, these thread local and class value roots are tricky to manage. I believe some of these are supposed to be cleared out by calling Ruby.tearDown, so it would be good to confirm that asciidoctorj is doing that.

    The others will need attention.

  2. matrei commented on Oct 8, 2026

    @matrei
    ContributorAuthor

    Thanks! The program in the description already calls tearDown() on each runtime before it drops the class loader, so all the roots above are still there after tearDown().

    AsciidoctorJ tears the runtime down in Asciidoctor.shutdown() and close(), which both call Ruby.tearDown(). The Gradle plugin doesn't call either of them yet. I tried it with shutdown() added after each conversion, and the Gradle daemon kept the runtimes all the same: its class space grew by the same amount per build as without it.

    I'm happy to test a snapshot or a patch against both the program and the Gradle build.

  3. matrei commented on Oct 8, 2026

    @matrei
    ContributorAuthor

    A correction to my last comment: the Gradle plugin does call it now. Since asciidoctor/asciidoctor-gradle-plugin#811, which was merged yesterday but isn't released yet, it closes the Asciidoctor instance at the end of each conversion, and that calls Ruby.tearDown(). That doesn't change the results above, because the program in the description calls tearDown() as well.

  4. matrei commented on Oct 8, 2026

    @matrei
    ContributorAuthor

    I tried fixing the roots, and with changes in both JRuby and jffi, the class loaders are collected.

    JRuby (four commits on master):

    1. The fstring dedup wrapper and LibrarySearcher's key and range wrappers are no longer cached in thread-locals. The dedup lookup creates a new wrapper, and the LibrarySearcher wrappers become plain fields, since all their accessors are synchronized anyway.
    2. MethodGatherer's FILTERED_DECLARED_METHODS and FILTERED_METHODS store Method[][] instead of PartitionedMethods, so the values they leave on JDK classes are JDK types only.
    3. StableClassValue.clear() also calls remove(cls) on the underlying java.lang.ClassValue for every class it computed a value for. Until now it only emptied the StableValue holders and left them in the classes, and the holder class is enough to keep the class loader alive. With this, JavaSupport.tearDown() releases the proxy cache, and jruby.ji.class.values=HARD_MAP is no longer needed.
    4. JRubyClassLoader.getTempDir() registers the directory and the copied jars with deleteOnExit() instead of adding a shutdown hook.

    jffi (one commit): com.kenai.jffi.internal.Cleaner no longer uses a java.lang.ref.Cleaner. It keeps its own phantom references and reference queue, and runs the actions of enqueued references whenever the next object is registered. As long as a cleaner thread holds actions whose classes come from jffi's class loader, that class loader can't be collected, so I don't see a way to keep the thread. The trade-off is that a native resource is freed at the next registration instead of right away.

    While I was in there, I noticed that all the cleanup actions in jffi are anonymous classes that use Outer.this (CallContext, Library, Aggregate, ClosureMagazine, ClosurePool.Magazine). That makes each registered object strongly reachable from its own action, so as far as I can tell the cleaner never runs them. I haven't measured that, and it's separate from this issue.

    Results. These numbers are from the program in the description, with all four runtimes on the main thread. To tell soft references apart from strong ones, I ran it with -XX:SoftRefLRUPolicyMSPerMB=0, which clears soft references at every GC:

    class loaders alive after 4 rounds
    master 4 of 4
    master + JRuby changes 4 of 4
    master + JRuby changes + jffi change 0 of 4

    Without the jffi change, the Cleaner-0 thread still holds every class loader. With both changes, without the flag, the only references left are the soft ones from the RubyEncoding coder thread-locals (made soft for JRUBY-6522), which the JVM clears under memory pressure.

    In the Gradle daemon with AsciidoctorJ 3.0.1 (asciidoctor/asciidoctor-gradle-plugin#810), I used a build with 20 source directories, a 1.5 GB heap and -XX:MaxMetaspaceSize=320m:

    • master: metaspace grew by about 31 MB per build, to 311 MB after 8 builds, and the daemon died during the 9th.
    • Both changes: metaspace also grew until build 8, reaching 310 MB. At build 9 the soft references were cleared, metaspace dropped back to 123 MB, and all 15 builds passed. With -XX:SoftRefLRUPolicyMSPerMB=0, class space stayed at 19 MB from the second build on.

    In the daemon, the soft reference that keeps the older class loaders is mostly the JDK's java.beans introspection cache on the plugin's own classes, so that one isn't JRuby's.

    rake spec:ji passes with the JRuby changes (1008 examples, 0 failures, the same 9 pending as master).

    Would you like these as PRs, one to JRuby and one to jffi? The JRuby commits are one per root, so they can also go in separately.

  5. headius commented on Oct 19, 2026

    @headius
    Member

    Doing some preemptive review of your changes here. Separate PRs would be good, as these each are very tricky to get right.

    JRuby changes:

    The fstring dedup wrapper and LibrarySearcher's key and range wrappers are no longer cached in thread-locals. The dedup lookup creates a new wrapper, and the LibrarySearcher wrappers become plain fields, since all their accessors are synchronized anyway.

    The change to LibrarySearcher is fine as long as the accesses are serialized (and slightly better because of less indirection). An alternative would be to have these be weakly held by the threadlocal, but that introduces an extra indirection and more dependence on GC to do the cleanup. We could also just add cleanup to tearDown.

    The fstring wrapper change will introduce a lot of overhead allocating new instances every time. The better fix would be to add this ThreadLocal to tearDown so it is actively cleared.

    Another possibility would be to implement our own ThreadLocal that doesn't store in a global location. It would formulate a key from a Thread but not reference the Thread directly. That could be managed locally in the objects consuming them or as a Ruby-level cache that also includes eager cleanup. Perhaps a separate exercise to explore that.

    MethodGatherer's FILTERED_DECLARED_METHODS and FILTERED_METHODS store Method[][] instead of PartitionedMethods, so the values they leave on JDK classes are JDK types only.

    This change is fine, if unfortunate. I prefer to have the named PartitionedMethods abstraction, but the stickiness of ClassValue forces the CL to stay alive.

    The Java 17-related deletions here should probably go into a separate cleanup commit/PR as they are not related.

    StableClassValue.clear() also calls remove(cls) on the underlying java.lang.ClassValue for every class it computed a value for. Until now it only emptied the StableValue holders and left them in the classes, and the holder class is enough to keep the class loader alive. With this, JavaSupport.tearDown() releases the proxy cache, and jruby.ji.class.values=HARD_MAP is no longer needed.

    ClassValue continues to be a pain in the butt. At this point we're basically implementing our own version of it, so I'm starting to feel like it's just a broken feature.

    This might work for your test but a WeakReference means that the values (usually expensive-to-calculate items like a proxy class or a list of partitioned methods) will get collected very early and have to be recalculated repeatedly. At worst this should be a SoftReference, but ideally we'd find a way to clean these up at teardown.

    The Method[][] for example would almost immediately be collected, forcing them to be recalculated continuously.

    Proxy classes might live longer, especially if a Ruby object that references them stays alive, but for transient objects that enter Ruby and are never hard-referenced, we'll end up recalculating the proxy repeatedly.

    JRubyClassLoader.getTempDir() registers the directory and the copied jars with deleteOnExit() instead of adding a shutdown hook.

    This code used to use deleteOnExit but it was modified in #4578 because more and more entries were added to the internal list of deletes (leaking each time a new JRuby instance loads those same paths.

    However, my reading of the current deleteOnExit code is that it should only add the same path once (via HashSet) so as long as additional runtimes used the same paths, it only represents a "soft" leak.

    Both versions will do no cleanup until exit, so that much stays the same. Clearly we want to eliminate the classloader reference here, but perhaps it would be better to register a shutdown hook that references only the paths instead of switching back to deleteOnExit?

    JFFI change:

    com.kenai.jffi.internal.Cleaner no longer uses a java.lang.ref.Cleaner. It keeps its own phantom references and reference queue, and runs the actions of enqueued references whenever the next object is registered.

    Fixes the issue but introduces a delayed cleaning. I get that the fix here is to avoid using global Cleaner stuff to avoid rooting the classloader until Cleaner runs, but we're trading one problem for another. Need to do some research on how to safely use Cleaner without rooting things lke classloaders.

    --

    We can discuss each of these in separate PRs. A few can probably merge as is, but others need more research.

  6. headius commented on Oct 22, 2026

    @headius
    Member

    This probably should be fixed in the 10.0 line since that's our current LTS branch and this represents an unexpected leak that can't be remedied by calling tearDown alone.

  7. added this to the JRuby 10.0.8.0 milestone on Oct 22, 2026
  8. matrei commented on Oct 26, 2026

    @matrei
    ContributorAuthor

    Thanks for the review! I'll open separate PRs. Should they target jruby-10.0, so they get merged forward to master, or master with backports?

    A few notes on the points, since some of them depend on details that aren't obvious from the summary.

    Thread-locals. Clearing them in tearDown() only helps the thread that calls tearDown(). ThreadLocal.remove() only removes the current thread's value, and every other thread that ever called freezeAndDedupString keeps its FStringEqual, so it keeps the class loader alive. In Gradle those are the pooled worker threads, and none of them is guaranteed to be the one that tears the runtime down. Adding remove() to tearDown() is cheap and worth doing, but it doesn't fix the leak by itself.

    To keep the reuse, the thread-local could hold a WeakReference<FStringEqual>. Then the value in each thread's map is a JDK type, and the wrapper is only weakly reachable from the thread. When the wrapper is collected, the next lookup makes a new one, so it costs one allocation per thread per GC instead of one per call. Your idea of a per-runtime cache keyed by thread would also work. I can benchmark the dedup hit path with the current code, a new wrapper per call, and the weak holder, before we pick one.

    For LibrarySearcher I'll keep the plain fields.

    MethodGatherer. The Java 17 logic isn't removed. filterAccessible was a static method nested in PartitionedMethods, and it moved up one level unchanged, so the diff shows it as removed and re-added because of the indentation. I'll put the move in its own commit, or keep filterAccessible where it was and leave a small holder class, whichever you prefer.

    We can't clean these up at teardown, because they're static ClassValues shared by every runtime in the class loader. One runtime's tearDown() can't remove them without affecting the others, and they only matter once the whole class loader goes away. With JDK-only values, what's left on JDK classes no longer references JRuby's class loader.

    StableClassValue. The computed values aren't held through a WeakReference. They stay strongly referenced from the StableValue in each class's class value map, exactly as before, so nothing is recalculated more often than it is now. (The Method[][] is in MethodGatherer, which doesn't use StableClassValue.)

    The only WeakReference is to the Class. It's stored as the value in the stableValues side map, which used to be null, so that clear() knows which class to call proxy.remove(cls) on. It has to be weak because a strong reference to the class would keep its key, the StableValue, reachable through the class's own class value map, and the WeakHashMap would never drop it. clear() is already called from JavaSupport.tearDown() through proxyClassCache.clear(), so this is the teardown cleanup: it now also removes the entries from the classes instead of only emptying them.

    JRubyClassLoader temp dir. You're right that this brings back what #4578 fixed. File.createTempFile gives each copied jar a new random name, so the paths aren't the same across runtimes, and the deleteOnExit set grows with every runtime that adds a nested jar. That's one path string per copied jar, which is much less than a class loader, but it's still a leak.

    A shutdown hook that references only the paths still needs code to delete them, and any class JRuby defines for that, a lambda included, is loaded by JRuby's class loader. That's what keeps it alive today (JRubyClassLoader$1). Some options:

    1. Name the copies after the source URL, plus a hash of the content or the modification time, instead of createTempFile. Runtimes that load the same nested jar then reuse the same file, and deleteOnExit stores each path only once. This is the case you described.
    2. Delete a runtime's copies when its JRubyClassLoader is closed. Since close() doesn't always close the underlying URLClassLoader (Errors running multiple independent ScriptingContainers in parallel #6218), the files can still be open, which is a problem on Windows.
    3. Keep deleteOnExit as it is and accept the small growth.

    I'd go with 1, unless you see a problem with reusing the copies.

    jffi Cleaner. First, a correction to my earlier comment, plus a separate problem. I wrote that five of the cleanup actions use Outer.this. In the released 1.4.3 jar it's all seven: the two that only use constructor parameters (ClosurePool.MagazineHolder and NativeMethods.ResourceHolder) also get a this$0 field from the Java 8 compiler. So each registered object is strongly reachable from its own action and never becomes phantom reachable. The cleanups never run, with java.lang.ref.Cleaner or with my change.

    To check, I opened libc with Library.openLibrary 5,000 times in one runtime, dropped the results, ran GC, and counted the pending actions:

    pending after GC
    jffi 1.4.3 5,050
    actions as static nested classes that capture only the address and Foreign 50

    So today the choice isn't between prompt and delayed cleaning: native resources are never cleaned, and the class loader is pinned on top of that. I've opened the capture fix as its own PR, jnr/jffi#200, since it's independent of the class loader.

    Second, I hoped that fixing the captures would also let us keep java.lang.ref.Cleaner, because the thread would drop each action once it ran. It doesn't. With the captures fixed, the reproducer still keeps 4 of 4 class loaders, even with -XX:SoftRefLRUPolicyMSPerMB=0. After two rounds, each runtime's Cleaner thread still holds 36 actions (32 CallContext, 3 Library and 1 NativeMethods holder), and all of their objects are still alive. They're only reachable through the runtime's class loader: with the thread-less cleaner, the same objects are registered and collected together with the class loader. The cycle is:

    Cleaner thread -> action -> action's class -> class loader -> static state -> registered object
    

    The object can't become phantom reachable while its action is pending, and the action stays pending until the object becomes phantom reachable. Any object that lives as long as the class loader does keeps it alive, and a runtime always creates some of them.

    As far as I can tell, that holds for any thread that outlives the class loader and holds actions whose classes come from it. The JDK's shared cleaner has the same problem. That leaves running the actions on threads jffi doesn't own, as my change does. With the capture fix and the thread-less cleaner together, the reproducer gets 0 of 4 class loaders alive with -XX:SoftRefLRUPolicyMSPerMB=0. The remaining references are the soft ones from the RubyEncoding thread-locals, as before.

    With either change alone or both together, jffi's tests (122) and JRuby's spec/ffi (4,990 examples, 2 pending) pass on Linux x86_64. They also pass with a small heap and serial GC, so collections and cleanups happen during the run. The results are the same as with jffi 1.4.3.

    On the delay: the queue is drained on every registration, which means whenever a new CallContext, Library, etc. is created. To make it less dependent on that, the queue could also be drained in other frequently called places, or jffi could add a public Cleaner.clean() that JRuby calls from tearDown(). I'll wait for your take on the delay before I open the jffi PR for the thread-less cleaner, so I can include whichever of these you prefer.

  9. matrei commented on Oct 26, 2026

    @matrei
    ContributorAuthor

    Here's the benchmark of the fstring dedup hit path. It's a JMH benchmark calling Ruby.freezeAndDedupString with a bare, unfrozen string whose fstring is already in the dedup table, as Hash#[]= does with a String key it has seen before. All three variants are based on current master:

    • thread-local: master, a ThreadLocal<FStringEqual>
    • new wrapper: a new FStringEqual per lookup, as in my branch
    • weak holder: a ThreadLocal<WeakReference<FStringEqual>>, so a thread's map only holds a JDK type after the runtime is torn down. When the wrapper is collected, the next lookup makes a new one.

    Average time per call in ns, JMH 1.37 with 5 forks of 10 one-second iterations, on JDK 21 on a 2-socket Xeon Gold 6240:

    1 key, 1 thread 1 key, 8 threads 1,024 keys, 1 thread 1,024 keys, 8 threads allocated per call
    thread-local 93.7 95* 111.9 119.4 0 B
    new wrapper 86.3 83.9 103.0 109.1 16 B
    weak holder 97.6 101.7 116.7 123.3 0 B

    The new wrapper is about 8% faster, because a thread-local lookup costs more than a 16-byte allocation. But it allocates on every hit, and escape analysis doesn't remove that: there were 35–123 GCs while measuring, against 0–2 for the others. The weak holder costs about 4% and allocates nothing.

    * With 8 threads on one key, the thread-local variant sometimes stays several times slower for a whole fork after a GC moves its wrapper: 2 of 10 forks averaged 336 and 441 ns, against about 95 ns for the rest. Across further runs it showed up in 0 to 8 of 15 forks, depending on details of the benchmark harness. Padding the wrapper before and after its field prevented it, so it looks like false sharing, but I haven't found with which object. Neither of the other two variants showed it.

    I also checked the class loader with the reproducer: with all the other fixes in place, the thread-local version still keeps a class loader alive, while both the new wrapper and the weak holder release all four.

    So I'd go with the weak holder for the fstring wrapper: it keeps the reuse, doesn't allocate, and fixes the leak. If you'd rather avoid the extra indirection, the per-runtime cache keyed by thread that you suggested would also work.

  10. headius commented on Oct 27, 2026

    @headius
    Member

    The jffi fix in jnr/jffi#200 (fixing jnr/jffi#199) has been merged and I'll update all JNR artifacts for JRuby 10.0.8.0 and 10.1.3.0 releases.

  11. headius commented on Oct 27, 2026

    @headius
    Member

    The ClassValue fix has been merged from #9895.

  12. headius commented on Oct 27, 2026

    @headius
    Member

    So I'd go with the weak holder for the fstring wrapper: it keeps the reuse, doesn't allocate, and fixes the leak. If you'd rather avoid the extra indirection, the per-runtime cache keyed by thread that you suggested would also work.

    There's another possible fix: synchronized logic that reuses a single FStringEqual while it is used for the lookup. The wrapper is only used for a brief lookup and is reset to null afterwards. The cost of that short synchronization may beat all of the other options and probably won't impact typical code.

  13. headius commented on Oct 27, 2026

    @headius
    Member

    The LibrarySearcher wrapper fix is merged: #9894

  14. headius commented on Oct 27, 2026

    @headius
    Member

    On the delay

    Delayed cleanup is obviously better than the old behavior of not cleaning up at all. I am not particularly concerned about lazy cleanup having some visible impact, since anyone concerned about lazy cleanup should be actively freeing their pointers. Lazy because we defer cleanup is no different in my eyes than lazy because GC hasn't run yet, and nobody should expect guarantees in either case (you walk away from allocated memory, you're implicitly letting us decide when it frees).

  15. headius commented on Oct 27, 2026

    @headius
    Member

    @matrei Update this issue with a checklist of specific fixes, so I know where we stand on progress. Thank you for breaking up the fixes into logical chunks... I'll continue to review and merge as they come in.

    I am hoping to release JRuby 10.0.8.0 and 10.1.3.0 later this week, so if we can get these changes wrapped up ASAP that would be great!

  16. matrei commented on Oct 28, 2026

    @matrei
    ContributorAuthor

    Thanks for merging these! I've added a checklist to the description.

    One thing for the JNR update: jnr/jffi#200 alone doesn't release the class loader. With it, the reproducer still keeps 4 of 4, because the Cleaner thread holds the actions of objects that live as long as the class loader. That takes the thread-less cleaner as well, so it would be good to have both in the jffi release for 10.0.8.0. Now that you're fine with the delay, I've opened it as jnr/jffi#201. It also runs the actions one at a time, as a cleaner thread would, because ClosurePool's Recycler registers an object while it holds the pool's lock, and two threads running actions at the same time could deadlock.

    For the fstring wrapper, I'll benchmark a single synchronized wrapper against the weak holder, with one thread and with several, since Hash#[]= goes through it for every unfrozen String key, on every thread.

    The StableClassValue change will get its own PR. It's the one where only the Class is held weakly, not the values, as in my comment above.

    For the temp dir, I'll open a PR with option 1 from my comment above, naming the copies after the source URL, unless you'd prefer another option.

  17. headius commented on Oct 28, 2026

    @headius
    Member

    One thing for the JNR update

    No problem, I've merged jnr/jffi#201 and it will be in the upcoming releases.

  18. headius commented on Oct 28, 2026

    @headius
    Member

    For the fstring wrapper,

    Another alternative trick: a single cached fstring wrapper "stolen" with an atomic swap. Any other thread that comes in needing a wrapper allocates their own. They each atomically try to swap theirs back in but only one wins. Doesn't matter because it's cleared by then anyway.

    This design makes single-thread calls as fast as an atomic swap of a reference (pretty fast), and multi-thread calls slow down only if they happen to see a null fstring wrapper. It's basically a single-object pool that spills into normal allocation if that instance is in use.

  19. headius commented on Oct 28, 2026

    @headius
    Member

    JRubyClassLoader temp dir.

    Ok revisiting your reply here...

    1. Name the copies after the source URL

    This only works if we can (semi-)atomically write these files out and verify/load them. Multiple runtimes starting up in multiple threads might all try to write that file at the same time, and then we have a sort of TOC-to-TOU issue where one thread might complete its write but then try to load the partial file from another thread. It makes these jars a cross-thread shared resource and that's obviously sticky.

    Security-wise, this also makes the filename predictable, which could be exploited (TOC-to-TOU again).

    1. Delete a runtime's copies when its JRubyClassLoader is closed.

    Yeah also problematic, both because of the unreliable closing of jar resources (I've chased that tail before and it's a mess) and because Windows won't delete files that are open (Windows! 😡). At this point we're talking about limitations of URLClassLoader and its underlying jar loading logic, though.

    1. Keep deleteOnExit as it is and accept the small growth.

    Undesirable, but perhaps we can mitigate this somewhat. It also doesn't solve the Windows problem, but that's been there for a long time without anyone coming up with a resolution.

    I propose a combination of these that mitigates most of the issues:

    1. A combination of (2) and (3): JRubyClassLoader.close makes a best effort to shut itself down and delete leftover files. Any files that can't be deleted are added to deleteOnExit. This will be all of the files on Windows, most likely, but on UNIX the deletes should all succeed and nothing leaks.

    The eager deletion + registration of undeletable files could be shunted to a Cleaner (to speed up teardown) since we can basically just feed it a ready-to-clean file-scrubbing function and walk away. This would, however, keep the ClassLoader alive a bit longer until the Cleaner can run our scrubber.

    Windows remains a problem but Windows is always a problem (and the current logic doesn't work on Windows either).

    I think we all agree we can't safely use an exit hook anymore since it will always keep a hard reference to the classloader.

  20. headius commented on Oct 28, 2026

    @headius
    Member

    Updates:

    • StableValue fix is merged.
    • All jffi cleaner fixes are merged and jffi 1.4.4 is released with those changes. The rest of jnr-* still need to be released to maintain dependency convergence.

    I believe we're just down to the JRubyClassLoader jar file cleanup and the fstring wrapper (other than getting jffi and friends updated in JRuby).

  21. matrei commented on Oct 28, 2026

    @matrei
    ContributorAuthor

    Another alternative trick: a single cached fstring wrapper "stolen" with an atomic swap.

    I benchmarked both of your ideas against the weak holder, on jruby-10.0: the single synchronized wrapper and the single wrapper taken with an atomic swap. Both fix the leak, but in both, every lookup in the runtime goes through the same object, and that costs more than it saves as soon as a second thread does lookups.

    These are freezeAndDedupString hits, cycling through 1,024 unfrozen String keys that are already in the table, in ns per lookup:

    threads thread-local (current) weak holder atomic swap synchronized
    1 109 112 120 104
    2 111 118 266 413
    8 116 122 1,122 3,879
    32 128 131 3,552 13,152

    That loop does nothing but lookups, which is the worst case for a shared object. With about 490 ns of other work before each lookup, the lookup itself took:

    threads thread-local (current) weak holder atomic swap synchronized
    1 103 105 120 101
    2 115 123 291 234
    8 119 126 644 5,064
    32 129 134 3,154 20,910

    For the atomic swap, the slowdown doesn't come from threads finding the wrapper gone. With 2 threads, only about 1 in 8 lookups had to allocate one, and a lookup still took more than twice as long. Every lookup on every thread does an atomic swap on the same reference, so its cache line keeps moving between cores. Putting the wrapper back with a plain release store instead of compareAndSet didn't change much: 244 ns at 2 threads and 3,287 at 32. On one thread, the two atomic operations are also slower than the ThreadLocal.get() they replace.

    On one thread, none of the four allocates. The atomic swap allocates when threads collide, on average 8 bytes per lookup at 8 and 32 threads, so about every second lookup. With the other fixes and -XX:SoftRefLRUPolicyMSPerMB=0, the reproducer releases all four class loaders with the weak holder, the atomic swap and the synchronized wrapper, and keeps 1 of 4 with the current thread-local.

    So I'd still go with the weak holder. It's the only option where threads don't share anything, it costs 2–7% on the lookup alone and about 1% with other work in between, and it doesn't allocate. I'll open a PR with it, unless you'd prefer something else.

    The numbers are JMH averages over 3 forks, on JDK 21 and a 72-core machine. The synchronized wrapper and the atomic swap ran an hour apart. The current thread-local and the weak holder ran both times and matched within 2%, and the tables show the second run.

  22. headius commented on Oct 28, 2026

    @headius
    Member

    The numbers are not too surprising. I knew the synchronized and atomic swap versions would have more contention under concurrent load. It's an open question how likely this sort of load would be during normal Ruby execution.

    Given the consistency of its numbers and simplicity of the change, the weak version does appear to be the safest choice for now.

    It would be interesting to explore one final option, though it's a much larger implementation: a custom map designed for FString that does what the wrapper does by default. It would need to be concurrency-safe but it would eliminate this entire issue. A similar case for LoadSearcher would also make sense.

    For now let's get the simple fix in place so we don't hold up getting some releases out in the next week.

  23. matrei commented on Oct 29, 2026

    @matrei
    ContributorAuthor

    I found another root while looking at jnr/jnr-posix#203: signal handlers that Ruby code installs with Signal.trap. It doesn't show up in the reproducer, because nothing there traps a signal, but it does whenever the embedded Ruby code does, as servers commonly do for TERM and INT.

    tearDown() only resets the EXIT trap. A handler installed for a real signal stays in the JDK's static signal table after the runtime is torn down, and keeps the runtime and its class loader alive. It also keeps every earlier runtime alive that trapped the same signal. RubySignal.SIGNAL_FACADE is static, so there's one SunSignalFacade per JRuby class loader, and its original map records the handler that was installed before that class loader's first trap. For the second runtime, that's the first runtime's handler:

    jdk.internal.misc.Signal.handlers → sun.misc.Signal$InternalMiscHandler → runtime 4's JRubySignalHandler → its class loader → RubySignal.SIGNAL_FACADE → original → runtime 3's JRubySignalHandler → its class loader → … → runtime 1's class loader

    With the reproducer from the description, plus Signal.trap('TERM') { puts 'runtime 1 trapped SIGTERM' } (with each runtime's own number) in each runtime's script, all the fixes from the checklist including jffi 1.4.4, and -XX:SoftRefLRUPolicyMSPerMB=0, these class loaders were still alive after four rounds:

    each runtime alive
    doesn't trap 0 of 4
    traps TERM 4 of 4
    traps TERM, then Signal.trap('TERM', 'DEFAULT') before tearDown() 0 of 4

    It's not only memory, though:

    • After all four runtimes were torn down, sending SIGTERM to the JVM ran runtime 4's block, which printed "runtime 4 trapped SIGTERM", and the JVM kept running. So a JVM in which an embedded runtime trapped TERM no longer stops on SIGTERM after that runtime is torn down.
    • 'DEFAULT' restores the handler from original, so with more than one JRuby class loader, it can restore another runtime's handler instead of the JVM's. When only runtime 4 restored 'DEFAULT', SIGTERM ran runtime 3's block, and 3 of 4 class loaders were still alive.

    master has the same code.

    I'd fix this by tracking the traps per runtime instead of per class loader, and in tearDown(), putting back the handler that was installed before the runtime's first trap of each signal, but only if the runtime's own handler is still the installed one. Runtimes torn down in a different order than they trapped would still need care: the handler to put back could belong to a runtime that's already been torn down. Would you like a PR along those lines, and should I add it to the checklist as (6)?

  24. matrei commented on Oct 29, 2026

    @matrei
    ContributorAuthor

    With everything on the checklist merged, jruby-10.0 (5a089a3) built with jffi 1.4.4 releases all four class loaders in the reproducer: 0 of 4 with -XX:SoftRefLRUPolicyMSPerMB=0, with and without a nested jar. With jffi 1.4.3 it's 4 of 4. So for the reproducer, only the jffi update is left.

    The reproducer only reverses a string, though. So I changed it to run a given piece of Ruby code in each of the four runtimes, and ran it for a list of common operations, each in its own JVM, with the same build and flag:

    Ruby code in each runtime JDK 21 JDK 25 root
    system, backticks, IO.popen, Process.spawn 4 of 4 4 of 4 A, B
    File.utime, so FileUtils.touch too 4 of 4 4 of 4 A, B
    FFI::MemoryPointer.new, io/console 4 of 4 0 of 4 A
    an FFI callback (qsort with a block) 4 of 4 4 of 4 E
    IO.pipe, write, IO.select, read_nonblock 3 of 4 3 of 4 C
    require 'openssl' and a digest 4 of 4 4 of 4 D

    These were 0 of 4 on both: File.stat, reading and writing files, File.chmod, symlinks, flock, File.mkfifo, File.umask, Dir.glob and Dir.chdir, ENV, Process.pid, Process.times, Process.getrlimit, Process.kill(0, …), Etc, TCP and UNIX sockets, Thread.new, Signal.trap followed by 'DEFAULT', SecureRandom, json and date, and an FFI call without a callback.

    The roots, from heap dumps:

    A. jnr-ffi's reference queue thread, on JDK 21. The first FinalizableReferenceQueue in a class loader starts a jnr.ffi.util.ref.internal.Finalizer thread. Both jnr-ffi and JRuby's own FFI (org.jruby.ext.ffi.jffi.NativeFinalizer) create one. A new thread keeps the access control context of the code that creates it:

    Finalizer thread → inheritedAccessControlContext → ProtectionDomain → JRuby's class loader

    Creating the thread in doPrivileged isn't enough; I tried it. jnr-ffi loads Finalizer in its own URLClassLoader, so that the thread doesn't hold the application's class loader. But on JDK 21 a URLClassLoader also keeps its creator's context, in acc, so the path becomes Finalizer → its class loader → acc → ProtectionDomain → JRuby's class loader. JDK 25 has neither field, and there the thread stops once the class loader is collected. Until then it also keeps running cleanups after the runtime is torn down. Once the class loader is closed, they fail with SEVERE: Error cleaning up after reference and a NoClassDefFoundError for CachingNativeMemoryAllocator$Bucket$CacheElement.

    B. jnr-ffi's TransientNativeMemory.currentMagazine, on any JDK. It's a static ThreadLocal<Magazine>. The thread that made the call keeps the magazine, and through its class, the class loader. jnr-ffi allocates from it for small direct memory, for example:

    • the argv and envp arrays for posix_spawnp (CharSequenceArrayParameterConverter)
    • Struct.arrayOf for the two Timespecs of utimensat

    The thread-local is only a cache, because referenceSet keeps each magazine until it's freed. So it could hold a WeakReference, the way JRuby's own CachingNativeMemoryAllocator does.

    C. JRuby's PosixShim.errno, on any JDK. Each PosixShim, and so each OpenFile, has had its own ThreadLocal<Errno> since a141c69, for #5706. When the IO is collected, its entry in the thread's map goes stale. The entry's value, a jnr.constants.platform.Errno, stays until the map clears stale entries, and holds the class loader until then. That's presumably why one of the four was released. Storing the errno number instead, or a WeakReference<Errno>, would keep JRuby's classes out of the thread's map.

    D. BouncyCastle 1.86, on any JDK. The static initializer of CryptoServicesRegistrar sets its default parameters as global properties. localSetGlobalProperty also puts them in the static threadProperties thread-local, for the current thread:

    thread's map → HashMap → DSAParameters[] → BouncyCastle's class loader (the runtime's JRubyClassLoader) → its parent, JRuby's class loader

    That's in BouncyCastle. It has clearThreadProperty, so jruby-openssl might be able to clear the properties after initializing, but I haven't tried that.

    E. jffi closures, on any JDK. A JNI global reference to com.kenai.jffi.ClosurePool$Proxy holds the class loader. It's the same kind of root as the jnr-ffi closures in jnr/jnr-posix#203. I haven't looked for a fix yet.

    To check A and B, I patched two jnr-ffi 2.3.4 classes into the jar:

    • currentMagazine holds a WeakReference<Magazine>.
    • FinalizableReferenceQueue never starts the thread. It already falls back to that when it can't start the thread: every new reference then drains the queue.

    With both, system, File.utime, FFI::MemoryPointer and io/console were 0 of 4 on JDK 21 and 25, and the SEVERE logs were gone. A real change would need the single drainer from jnr/jffi#201, though. JRuby's Bucket.recycle creates a FinalizableWeakReference while it holds the bucket's lock, so draining from the reference's constructor would run other cleanups under that lock, and recursively.

    jnr-ffi still needs a release for the dependency update. Would you like a jnr-ffi PR for A and B in time for that release, and a JRuby PR for C against jruby-10.0? D and E could be follow-ups. And should I add these to the checklist, along with the trap root?

  25. headius commented on Oct 29, 2026

    @headius
    Member

    I'd fix this by tracking the traps per runtime instead of per class loader, ...
    Would you like a PR along those lines, and should I add it to the checklist as (6)?

    I'm not a fan of scope creep in a single issue. Would you say that your original concerns have been address sufficiently, or are the additional roots you've recently discovered still necessary?

    I ask because we're probably chasing a very long tail of classloader roots along the weird edges of JRuby and its component libraries. Some, like the issues you found in Bouncy Castle, which may be difficult for us to fix.

    I'd like to say this issue has been addressed, and perhaps open a discussion with the remaining items for us to resolve individually.

    Thoughts?

  26. matrei commented on Oct 29, 2026

    @matrei
    ContributorAuthor

    Yes, the original case is covered. I ran the reproducer from asciidoctor/asciidoctor-gradle-plugin#810 in one Gradle daemon, with plugin 5.1.0, which closes AsciidoctorJ after converting, and soft references cleared:

    JRuby class space after 8 builds
    jruby-10.0 + jffi 1.4.4 21 MB, flat from build 2
    jruby-10.0 + jffi 1.4.3 51 MB, +5 MB per build
    9.4.14.0, as AsciidoctorJ 3.0.1 ships it 49 MB, +5 MB per build

    With default settings, the old class loaders are only softly reachable. With Gradle's default 384 MB metaspace cap, metaspace dropped back when it filled up, and 15 builds passed. With 9.4.14.0, the daemon died at build 11.

    So the jffi update is all that's left here, and I agree with closing this once it's in. I'm happy to open a discussion with the remaining items, apart from trap, since you're handling that. The jnr-ffi PR for A and B is ready whenever you want it.

  27. headius commented on Oct 29, 2026

    @headius
    Member

    @matrei I'm considering options for trap but please open a discussion and include it as one of the outstanding items. We might as well completely cover all known cases (and keep adding newly-discovered cases) and wiggle our way toward fixing this for all JRuby use cases.

    I will mark this as complete once the jffi merge has fully landed (still evaluating unrelated PRs in the JNR stack).

  28. linked a pull request that will close this issueUpdate Java Native Runtime #9925on Oct 29, 2026
  29. headius commented on Oct 30, 2026

    @headius
    Member

    Fixed by numerous items!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions