Repository navigation
Description
What happens
Main.formatFiles reads every file on the main thread, as new String(Files.readAllBytes(path), UTF_8), keeps the text in a map and submits it to the pool. It then takes the results on the main thread, in the order the files were given, and compares, writes or prints each one there. Every input and every output stays referenced until the loop is over, and a file that is done waits for all the files before it to be written or printed.
google-java-format
The same, and it waits longer. It reads on the main thread too, and an ExecutorCompletionService hands back the results as they finish, but all of them go into a list, which is sorted by path, and only then is anything written or printed.
Where the time goes now
865 files from java.base of JDK 21, 16 MB of source, formatted with --replace on 10 cores with 21.0.2:
| Wall | CPU | Peak RSS | |
|---|---|---|---|
| Native binary | 7.0 s | 15 s, about 2 cores busy | 2.4 GB |
java -cp ... Main, JDK 21 |
6.2 s | 44 s, about 7 cores busy | 4.3 GB |
Reading the 865 files on one thread takes 0.04 s, and their text with the formatted text comes to some tens of megabytes. Neither is what holds the run back. In the native run, -XX:+PrintGC shows 40 collections that paused it for 5.0 of its 6.9 s: 17 full ones for 3.05 s and 23 incremental ones for 1.94 s, with the heap at up to 1.9 GB before a collection. The image is built without --gc, so it has the serial collector, which stops every thread: 20 threads format on about 2 cores.
Proposal
- One task does all of one file and hands back an immutable result. It reads the file on its worker thread, formats it, writes it for
--replace, and returns the path, whether the file changed, the diagnostics or the exception, and the formatted text only where it is printed, to standard output without--replaceor--dry-run. - The main thread passes on paths and prints results. It takes the results in the order the files were given for what it prints, the formatted text, the paths of
--dry-runand the messages, so that the output keeps its order, and lets go of each result once it is printed. With--replaceno file waits for an earlier one to be written. - A bound on what is in flight, a small multiple of the pool size, so that memory does not grow with the number of files.
To measure first
For the native binary the collector is the bigger lever: a larger young generation or another collector where GraalVM offers one for the platform, and fewer than 20 threads under a serial collector. Each change can be measured on the run above.
Activity