Repository navigation
Description
Following up on #9049 / #9052, where generating the docs at release time was called the "longer-term project". I looked into it because the current download keeps showing up in CI failures. It also turns out the docs it ships don't work on 10.1.
The shipped docs don't work on 10.1.x
maven/jruby-dist downloads ruby3.4-doc_3.4.5*.deb from Debian experimental and copies share/ri/3.4.0/system into the dist. On 10.1 that fails in two ways.
-
Wrong directory. JRuby 10.1 has
RbConfig::CONFIG['ruby_version'] == "4.0.0", sorilooks inshare/ri/4.0.0/system. The path3.4.0is hardcoded inassembly/common.xml. On an unmodified 10.1.3.0 dist:$ bin/jruby -S ri -T String#match Nothing known about String -
The data can't be loaded by the rdoc we ship. Pointing
ri -dat the shipped directory crashes:Unhandled Java exception: java.lang.ClassCastException: class org.jruby.RubyNil cannot be cast to class org.jruby.RubyArray __member__ at org/jruby/RubyStruct.java:607 unmarshalFrom at org/jruby/RubyStruct.java:1047The Ruby 3.4 data marshals
RDoc::Markup::Headingas a Struct, but rdoc 7.2 on Ruby 4.x defines it as a class. CRuby 4.0 also refuses this data, but with a cleanTypeError(class RDoc::Markup::Heading not a struct). So renaming the directory wouldn't fix it. The ri data has to match the rdoc version we ship.Separately, JRuby raising a Java
ClassCastExceptionwhere CRuby raisesTypeErrorlooks like a Marshal bug of its own. I can open a separate issue for it.
10.0.x (Ruby 3.4, rdoc 6.14) is not affected: ri String#match works there.
There's also no ruby4.0 package in Debian yet, so the current approach can't just be bumped. It relies on an experimental package whose filename already had to be hardcoded once (f9b1525) when 3.4.9 showed up.
CI cost
unpack-ri-docs does an anonymous passive FTP download from rsync.osuosl.org. It runs 10 times per push to master:
- 2 in
ci.yml(mvn -Pdist) - 1 in build-dist
- 1 in the Windows installer build
- 6 in src verification (3 OSes x 2 JDKs)
It also runs in the nightly snapshot. Nothing in CI or the tests reads share/ri.
How often it fails: I checked every failed job that runs this step, across all branches and PRs, over the last 1,000 Dist Verification and 1,000 JRuby CI runs (Aug 23 to Nov 11).
- 27 jobs in 27 runs failed in
unpack-ri-docs, and all 27 werejava.net.SocketTimeoutException: Connect timed outto rsync.osuosl.org. - 26 of the 27 were
src verification on macos-latest. The other was onemvn -Pdiston Ubuntu. Passive FTP from the macOS runners seems to be the weak spot. - That's 27 of the 99 failed Dist Verification runs in the window (974 runs in total). 4 were on master and 23 on PR or other branches.
- In terms of rate, it's roughly 1 in 75 macOS src-verification jobs (26 of about 1,950).
Proposal: generate the ri data with JRuby's own rdoc
I ran JRuby's rdoc over the CRuby 4.0.9 release tarball (version.ruby in default.build.properties), the same way CRuby's make rdoc does:
rdoc --ri --op OUT --no-force-update --exclude '^lib/rubygems/core_ext/kernel_require\.rb$' .
- It takes about 40 seconds and produces about 57 MB (most of the extra over the Debian data is generated Prism files, which are already excluded).
- An unmodified 10.1.3.0 dist reads it with
ri -d OUT String#match. - Because JRuby's rdoc writes it, the Marshal format always matches the rdoc we ship. That was also the advice in ri docs are not available for core #9049.
- The tarball's SHA256 is published (ruby/www.ruby-lang.org
_data/releases.yml), so the download can be checked.
It doesn't need to run on every CI worker. Releases are built locally (rake maven:prepare_release / deploy_release with -Prelease), and the data is the same on every platform. Some options:
- A release-only step. A rake task or Maven profile downloads and checks the CRuby tarball, runs
bin/jruby -S rdoc, and puts the result inshare/ri/<ruby_version>/system. Normal CI dist builds skip it, which removes the FTP download from every push. - A GitHub Actions workflow (weekly and on demand) that builds JRuby, generates the docs, and checks that
ri String#matchworks, so the release step doesn't break silently. It could also publish the result as an artifact that the release build downloads, but that's one more artifact to version.
My suggestion is 1 + 2: generate at release time, check it on a schedule, and skip it in normal CI.
Open questions
- Bundled gems. The Debian data includes docs for bundled gems (rake, rdoc, reline, logger, benchmark, …). Their sources are in the tarball's
.bundle/gems, but CRuby's.documentdoesn't include them. Do we want them? - Exclude list.
common.xmlexcludes CRuby-specific content. It needs updating for the new top-level entries (e.g.jit,ZJIT,Ruby,security). - rdoc version coupling. The data has to be regenerated when we bump the rdoc gem. Generating at release time handles that automatically.
- JRuby-specific docs (the list in ri docs are not available for core #9049:
jruby/javaextensions, jar-dependencies, …) could come later. The Ruby sources undercore/src/main/ruby/jrubyalready go through rdoc. The Java extensions would need an rdoc parser for@JRubyMethod+ Javadoc, and most of those methods don't have doc comments yet. - Short term: should we drop or skip the current FTP step on master (10.1) until this is in place? The data it ships can't be used there anyway.
@headius what do you think? If it sounds good, I'm happy to start on the release-time generation.
Activity