Skip to content

Speed up incremental builds#8326

Merged
headius merged 3 commits intojruby:masterfrom
headius:faster_incrementals
Aug 4, 2024
Merged

Speed up incremental builds#8326
headius merged 3 commits intojruby:masterfrom
headius:faster_incrementals

Conversation

@headius
Copy link
Member

@headius headius commented Jul 31, 2024

This is a series of patches to make a fast-as-possible incremental build profile.

Reducing the build

Currently when doing a default build of JRuby, the three modules "core" (jruby-base, the actual JRuby sources), "shaded" (jruby-core, the combined JRuby and dependencies jar), and "lib" (jruby-stdlib) are always build. The "lib" module is expensive to build because it launches JRuby to conditionally fetch and unpack or install gems to populate the standard library. This can become tedious when iterating on source-only changes to JRuby.

Making jar builds faster

The core and shaded jar files are always built with full compression, which is unnecessary and slow for local incremental builds. The shaded jar also builds a shaded sources jar which is useless for local development and produces warnings about duplicate files. We can make these jars faster to build and omit unnecessary bits.

This is a first attempt to make a faster incremental build by
omitting the lib (stdlib) module. Because profiles are applied on
top of the default configuration, I had to trim the default
module set and instead activate the lib modules via a missing
property "core". This allows specifying "-Dcore" to skip building
the lib (stdlib) module.

Because the property will normally be unset, all executions
without "-Dcore" will continue to build stdlib.
@headius headius added this to the JRuby 9.4.9.0 milestone Jul 31, 2024
@headius
Copy link
Member Author

headius commented Jul 31, 2024

The best way to conditionally active the "lib" module I have found is to only add it to the active modules when a property "core" is unset. This allows a developer to pass "-Dcore" to mvn to avoid adding and building the "lib" module.

For me this reduces the zero-change rebuild time from 12-13s down to 6.5-7.5s.

I believe this does not impact any other build configurations since the "core" property will always be unset by default.

This patch attempts to do the following:

* Avoid building the sources jar during shading, since it adds
  time and produces many duplicate file warnings. We explicitly
  re-activate the sources jar in the release profile (and others).
* Avoid compression of the shaded jar except during release and
  other profiles. I'm not sure this is working right, because it
  appears the files are still getting compressed.
@headius
Copy link
Member Author

headius commented Jul 31, 2024

@mkristian If you have any suggestions here I would appreciate the help! I am having some difficulty figuring out how to disable jar compression; nothing I do seems to turn it off.

Also, it seems I can only disable the build of the shaded sources jar at the top-level configuration. I've pushed a patch for you to review. Obviously we don't want to break maven central pushing, so we need the sources jar to be generated and deployed during release.

cc @enebo for any concerns or ideas

@enebo
Copy link
Member

enebo commented Jul 31, 2024

I remember a few years ago mvn -pl core worked then stopped at some point. I think it still compressed the jars though?

@headius
Copy link
Member Author

headius commented Aug 1, 2024

@enebo mvn -pl core still "works", but doesn't fully rebuild lib/jruby.jar. When we moved the shaded jar build to ./shaded we broke the ability to just rebuild the ./core module. The former is now jruby-core and produces the shaded lib/jruby.jar file and the latter is now jruby-base unshaded; therefore in order to get the equivalent of "just rebuild the lib/jruby.jar" you must do mvn -pl core -pl shaded. The change here basically makes that into mvn -Dcore.

@headius headius merged commit 52b51d5 into jruby:master Aug 4, 2024
@headius headius deleted the faster_incrementals branch August 4, 2024 14:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants