Vfs API

The classgraph-vfs library is a read-only virtual filesystem. It reads directories, jarfiles and modules through one interface, however they are named, and lets you read what you find as a stream, a channel, a byte array, a string, a ByteBuffer, or a java.nio.file.Path. It is the storage layer that ClassGraph itself scans through, and it can be used on its own, with no scanning involved.

<dependency>
    <groupId>io.github.classgraph</groupId>
    <artifactId>classgraph-vfs</artifactId>
    <version>LATEST</version>
</dependency>

If you already depend on classgraph, you have this library too: classgraph depends on classgraph-classpath, which depends on classgraph-vfs, which depends on classgraph-base.

Three classes carry the API: Vfs opens things, a VfsRoot is one opened directory, jarfile or module, and a VfsEntry is one file within it. Two more turn up as you use them: CloseableByteBuffer wraps a buffer you have to close, and VfsVisitor is the callback that VfsRoot#walk() takes. That is the entire public API.

import io.github.classgraph.vfs.*;

try (Vfs vfs = new Vfs();
        VfsRoot root = vfs.open("outer.jar!/lib/inner.jar")) {
    for (VfsEntry entry : root) {
        System.out.println(entry.getName() + " (" + entry.getLength() + " bytes)");
    }
}

• One interface over every kind of storage
• Reading through java.nio.file
• Nested jarfiles, package roots and multi-release jarfiles
• Vfs
• VfsRoot
• VfsEntry
• VfsVisitor
• Lifetime and closing
• Thread safety
• What is not reported

The point of the library is that the same three lines of code read all of these, and nothing in them says which kind of storage is being read:

Every one of those returns a VfsRoot, and every VfsRoot lists VfsEntry objects: it is Iterable<VfsEntry>, so a for-each loop over the root visits them all, and getEntries(), getEntries(prefix) and walk() are there for the cases a for-each loop does not cover. root.getKind() tells you which of DIRECTORY, ARCHIVE or MODULE you got, if you care; most code does not need to.

Once you have a VfsEntry, there are seven ways to read it, so that you can hand its content to whatever the next piece of code expects without a copy in between:

For example, this reads one file out of a jarfile nested inside another jarfile, decodes it, and never writes anything to disk:

try (Vfs vfs = new Vfs();
        VfsRoot root = vfs.open("/path/to/outer.jar!/BOOT-INF/lib/inner.jar")) {
    VfsEntry entry = root.getEntry("META-INF/MANIFEST.MF");
    if (entry != null) {
        System.out.println(entry.loadAsString());
    }
}

root.asFileSystem() returns a read-only java.nio.file.FileSystem over any root, so anything that takes a Path can read a directory, a jarfile, a nested jarfile, a package root, a module, or a jarfile that exists only in RAM, without knowing which it has:

try (Vfs vfs = new Vfs();
        VfsRoot root = vfs.open("/path/to/outer.jar!/BOOT-INF/lib/inner.jar");
        FileSystem fs = root.asFileSystem()) {
    byte[] classfile = Files.readAllBytes(fs.getPath("/com/xyz/Widget.class"));
    try (Stream<Path> paths = Files.walk(fs.getPath("/"))) {
        paths.filter(Files::isRegularFile).forEach(System.out::println);
    }
}

The separator is / and the root directory is /, whichever kind of root it is a view of. Files are read with Files.newInputStream, Files.newByteChannel, Files.readAllBytes and Files.readString; directories are listed with Files.newDirectoryStream, Files.list and Files.walk, and are synthesized from the names of the entries below them, since a jarfile need not contain an entry for every directory whose contents it holds. getPathMatcher supports both glob: and regex:. basic is the only attribute view.

Everything that would write -- Files.delete, Files.write, Files.copy, Files.move, Files.createDirectory, Files.newOutputStream, setTimes, and newByteChannel with a write option -- throws ReadOnlyFileSystemException. FileSystem#close() closes the VfsRoot it is a view of, and closing the root closes the filesystem, so either one can go in the try-with-resources; after that isOpen() returns false and every read throws ClosedFileSystemException, as java.nio.file.FileSystem specifies. Neither of them releases any storage: the file handles, memory mappings and temporary files belong to the Vfs, and are released when the Vfs is closed, which also makes every filesystem view it handed out report itself closed.

The JDK ships a zip filesystem provider, and it is the right tool when you have a jarfile you can name as a Path. It cannot open a module, and it cannot open a jarfile that was never written to disk. It also cannot name a nested jarfile: given jar:file:/path/to/outer.jar!/BOOT-INF/lib/inner.jar it keeps only the part before the first !/, and silently opens outer.jar instead, so the entries of inner.jar are not there.

A jarfile inside another jarfile is named by separating the two with !/, to any depth:

try (VfsRoot inner = vfs.open("outer.jar!/lib/inner.jar!/lib/innermost.jar")) {
    // ...
}

A nested jarfile that is stored uncompressed is read in place, with no copy. One that is stored deflated is inflated into RAM, or spilled to a temporary file if it exceeds the maximum buffered jar RAM size (64MB by default); any such file is deleted when the Vfs is closed. Construct the Vfs with nested jars disabled if !/ should only ever mean a package root.

A jar: or jar:file: prefix is accepted, but is not needed and changes nothing: /path/outer.jar!/lib/inner.jar, file:/path/outer.jar!/lib/inner.jar and jar:file:/path/outer.jar!/lib/inner.jar all open the same nested jarfile. Which ! separates the levels is not decided by the prefix, because it cannot be: ! is a legal filename character on every platform, JarURLConnection defines the separator as !/ and offers no way to escape a literal one, so /dir!/x.jar is genuinely ambiguous between a jarfile x.jar in a directory named dir! and an entry x.jar inside a jarfile named dir. It is decided by looking at the storage instead: each ! is tried in turn, and the first one whose preceding path names an existing file is the separator, since the outermost jarfile has to exist to be read at all. A path with a remote URL scheme cannot be checked that way, so for those the first ! is taken to be the separator.

If the last !/ section names a directory rather than a jarfile, it is a package root: only the entries below it are reported, and the root is stripped from their names. This is how a Spring Boot jarfile's classes are read:

try (Vfs vfs = new Vfs();
        VfsRoot classes = vfs.open("/path/to/app.jar!/BOOT-INF/classes")) {
    // An entry stored as "BOOT-INF/classes/com/xyz/Widget.class"
    // is reported as "com/xyz/Widget.class"
    VfsEntry widget = classes.getEntry("com/xyz/Widget.class");
    System.out.println("package root: " + classes.getPackageRoot());
}

VfsRoot#getPath() returns the jarfile's path without the package root; VfsRoot#getPackageRoot() returns the package root, or the empty string if the whole jarfile is the root.

For a multi-release jarfile, only the newest version of each entry that the running JVM can use is reported, and the META-INF/versions/<version>/ prefix is stripped from the name -- so getEntry("com/xyz/Widget.class") returns the right version of the class for this JVM, and META-INF/versions/ never appears in an entry name. Construct the Vfs with multi-release versions enabled to see every version instead.

A Vfs opens things, and owns the file handles, memory mappings and temporary files that back them. It is AutoCloseable, and belongs in a try-with-resources block.

A directory or jarfile is opened once however it is named, so the same VfsRoot comes back for a plain path, for the file: or jar: URL of the same thing, for a path that reaches it through a symlink, and, on Windows, for a path written with backslashes rather than forward slashes, or one that names a directory by its 8.3 short name. A root names itself at the canonical path, so getPath() is not always the path you opened it by.

A Vfs is configured by its constructor and cannot be reconfigured afterwards, so that the threads sharing it never see a setting change under them. new Vfs() uses the default of every option; the four-argument constructor sets them all:

new Vfs(/* enableNestedJars = */ true, /* enableMultiReleaseVersions = */ false,
        /* urlSchemes = */ Set.of("https"), /* maxBufferedJarRAMSize = */ 64 * 1024 * 1024)

Vfs.DEFAULT_ENABLE_NESTED_JARS, Vfs.DEFAULT_ENABLE_MULTI_RELEASE_VERSIONS and Vfs.DEFAULT_MAX_BUFFERED_JAR_RAM_SIZE name the defaults, for setting one option and leaving the rest alone. An invalid URL scheme or a negative maxBufferedJarRAMSize throws IllegalArgumentException.

A Vfs caches every root it opens, so opening the same path twice returns the same VfsRoot, and a jarfile that encloses several nested jarfiles is only read once.

One opened directory, jarfile or module. AutoCloseable.

One file within a root.

VfsEntry has one further public method, getZipEntry(), which returns a type from the library's internal packages. Those packages are exported only to ClassGraph's own modules, so the method cannot be called from a module of your own, and it is not covered by the API compatibility guarantees.

CloseableByteBuffer is AutoCloseable, and getByteBuffer() returns the buffer:

try (Vfs vfs = new Vfs();
        VfsRoot root = vfs.open("/path/to/library.jar")) {
    VfsEntry entry = root.getEntry("com/xyz/Widget.class");
    try (CloseableByteBuffer buf = entry.read()) {
        ByteBuffer byteBuffer = buf.getByteBuffer();
        // ...
    }
}

Iterating a root, or calling getEntries(), builds a list of every file in it. When only some of them are wanted, walk() is cheaper: it offers each directory before the entries in it, so an unwanted one can be skipped, and for a directory tree a skipped directory is never even listed. When the wanted entries all sit under one path, getEntries(prefix) does exactly this for you.

try (Vfs vfs = new Vfs();
        VfsRoot root = vfs.open("/path/to/classes")) {
    root.walk(new VfsVisitor() {
        @Override
        public boolean enterDirectory(String dirName) {
            // Return false to skip this directory
            return !dirName.startsWith("com/xyz/test/");
        }

        @Override
        public boolean visitEntry(VfsEntry entry) {
            System.out.println(entry.getName());
            // Return false to stop the walk early
            return true;
        }
    });
}

How much a skipped directory skips differs by root, and the difference matters: a directory tree skips the whole subtree, because not listing it is the entire saving, whereas a jarfile or a module already has its entry list in hand, so only that directory's own entries are skipped and the directories below it are still offered. That is deliberate -- a caller that strips a package root prefix such as BOOT-INF/classes/ from the names before judging them would otherwise prune BOOT-INF/ and lose everything under it.

A Vfs owns storage: the open file handles, the memory mappings, and the temporary files that a nested jarfile has to be spilled to when it cannot be read in place. Closing the Vfs releases all of them and closes every root it handed out, so a Vfs belongs in a try-with-resources block, and no VfsRoot, VfsEntry, InputStream or ByteBuffer obtained from it should be allowed to escape that block. The one exception is entry.load(), which returns your own copy of the content and stays valid afterwards.

A VfsRoot is AutoCloseable too, but closing one only drops that root: its entries stop working, its FileSystem view starts throwing ClosedFileSystemException, and opening the same path again builds a fresh root. The jarfile behind it stays open, because other roots may be reading the same jarfile. Only the Vfs releases storage.

So a root does not have to be closed, and there is nothing to leak by not closing one -- it is worth closing when a long-lived Vfs opens many roots, since a Vfs holds every root it has opened until it is closed itself. An IDE cannot know that, though: with resource analysis switched on, which is the default in Eclipse, VfsRoot root = vfs.open(path) is reported as "Potential resource leak: 'root' may not be closed", because vfs.open() returns an AutoCloseable. Declaring the root in the try-with-resources silences it and costs nothing, and every example on this page does that. Where a root is opened and used without being named -- vfs.open(path).getEntries() -- the same warning is reported against the unnamed value, and can be ignored.

Reading anything after the Vfs has been closed throws IOException rather than returning wrong bytes.

A Vfs and everything it hands out is safe to use from many threads at once. Two threads that open the same path get the same VfsRoot, and only one of them does the work of reading it. This is what lets ClassGraph scan a jarfile in parallel.

That matters more than it might sound, because java.util.zip.ZipFile serializes on the instance monitor: every public method takes synchronized (this), and so does every read call on the streams it hands out. classgraph-vfs never has a cursor to protect. Every read names the absolute offset it wants: a file is read either through a memory-mapped ByteBuffer, which is indexed absolutely, or through FileChannel#read(ByteBuffer, long), which takes the position as an argument rather than carrying one, and which reaches pread without taking the channel's position lock. The central directory is parsed once into an immutable index. So entry lookup and entry reads take no locks at all. Entry lookup under ZipFile does not merely fail to speed up with more threads, it gets slower; see the measurements in the classgraph-vfs README.

Iterating a root, getEntries() and walk() report files, not directories, and skip anything that cannot be read as a file:

• Directory entries, i.e. zip entries whose name ends in /.
• Encrypted entries.
• Entries stored with a compression method other than stored or deflated, which are the two the JDK's inflater supports.
• Entries whose central directory record is inconsistent -- a compressed or uncompressed size, an offset, or a local header position that falls outside the file.

Open the Vfs with verbose() to see which entries were skipped and why.

One thing a directory root does still report is a file the process has no permission to read. Telling the files from the subdirectories uses the metadata the walk already reads, and a permission check would cost a second syscall per file, so the walk does not make one. Reading such an entry throws an IOException, and getEntry() returns null for its name -- so a name that iterating the root reported can still come back null from getEntry().