Skip to content

Proposal: generate ri docs with JRuby's rdoc at release time instead of downloading Debian's package #10039

Open

Description

@zev

Following up on #9049 / #9052, where generating the docs at release time was called the "longer-term project". I looked into it because the current download keeps showing up in CI failures. It also turns out the docs it ships don't work on 10.1.

The shipped docs don't work on 10.1.x

maven/jruby-dist downloads ruby3.4-doc_3.4.5*.deb from Debian experimental and copies share/ri/3.4.0/system into the dist. On 10.1 that fails in two ways.

  1. Wrong directory. JRuby 10.1 has RbConfig::CONFIG['ruby_version'] == "4.0.0", so ri looks in share/ri/4.0.0/system. The path 3.4.0 is hardcoded in assembly/common.xml. On an unmodified 10.1.3.0 dist:

    $ bin/jruby -S ri -T String#match
    Nothing known about String
    
  2. The data can't be loaded by the rdoc we ship. Pointing ri -d at the shipped directory crashes:

    Unhandled Java exception: java.lang.ClassCastException: class org.jruby.RubyNil cannot be cast to class org.jruby.RubyArray
              __member__ at org/jruby/RubyStruct.java:607
           unmarshalFrom at org/jruby/RubyStruct.java:1047
    

    The Ruby 3.4 data marshals RDoc::Markup::Heading as a Struct, but rdoc 7.2 on Ruby 4.x defines it as a class. CRuby 4.0 also refuses this data, but with a clean TypeError (class RDoc::Markup::Heading not a struct). So renaming the directory wouldn't fix it. The ri data has to match the rdoc version we ship.

    Separately, JRuby raising a Java ClassCastException where CRuby raises TypeError looks like a Marshal bug of its own. I can open a separate issue for it.

10.0.x (Ruby 3.4, rdoc 6.14) is not affected: ri String#match works there.

There's also no ruby4.0 package in Debian yet, so the current approach can't just be bumped. It relies on an experimental package whose filename already had to be hardcoded once (f9b1525) when 3.4.9 showed up.

CI cost

unpack-ri-docs does an anonymous passive FTP download from rsync.osuosl.org. It runs 10 times per push to master:

  • 2 in ci.yml (mvn -Pdist)
  • 1 in build-dist
  • 1 in the Windows installer build
  • 6 in src verification (3 OSes x 2 JDKs)

It also runs in the nightly snapshot. Nothing in CI or the tests reads share/ri.

How often it fails: I checked every failed job that runs this step, across all branches and PRs, over the last 1,000 Dist Verification and 1,000 JRuby CI runs (Aug 23 to Nov 11).

  • 27 jobs in 27 runs failed in unpack-ri-docs, and all 27 were java.net.SocketTimeoutException: Connect timed out to rsync.osuosl.org.
  • 26 of the 27 were src verification on macos-latest. The other was one mvn -Pdist on Ubuntu. Passive FTP from the macOS runners seems to be the weak spot.
  • That's 27 of the 99 failed Dist Verification runs in the window (974 runs in total). 4 were on master and 23 on PR or other branches.
  • In terms of rate, it's roughly 1 in 75 macOS src-verification jobs (26 of about 1,950).

Proposal: generate the ri data with JRuby's own rdoc

I ran JRuby's rdoc over the CRuby 4.0.9 release tarball (version.ruby in default.build.properties), the same way CRuby's make rdoc does:

rdoc --ri --op OUT --no-force-update --exclude '^lib/rubygems/core_ext/kernel_require\.rb$' .
  • It takes about 40 seconds and produces about 57 MB (most of the extra over the Debian data is generated Prism files, which are already excluded).
  • An unmodified 10.1.3.0 dist reads it with ri -d OUT String#match.
  • Because JRuby's rdoc writes it, the Marshal format always matches the rdoc we ship. That was also the advice in ri docs are not available for core #9049.
  • The tarball's SHA256 is published (ruby/www.ruby-lang.org _data/releases.yml), so the download can be checked.

It doesn't need to run on every CI worker. Releases are built locally (rake maven:prepare_release / deploy_release with -Prelease), and the data is the same on every platform. Some options:

  1. A release-only step. A rake task or Maven profile downloads and checks the CRuby tarball, runs bin/jruby -S rdoc, and puts the result in share/ri/<ruby_version>/system. Normal CI dist builds skip it, which removes the FTP download from every push.
  2. A GitHub Actions workflow (weekly and on demand) that builds JRuby, generates the docs, and checks that ri String#match works, so the release step doesn't break silently. It could also publish the result as an artifact that the release build downloads, but that's one more artifact to version.

My suggestion is 1 + 2: generate at release time, check it on a schedule, and skip it in normal CI.

Open questions

  • Bundled gems. The Debian data includes docs for bundled gems (rake, rdoc, reline, logger, benchmark, …). Their sources are in the tarball's .bundle/gems, but CRuby's .document doesn't include them. Do we want them?
  • Exclude list. common.xml excludes CRuby-specific content. It needs updating for the new top-level entries (e.g. jit, ZJIT, Ruby, security).
  • rdoc version coupling. The data has to be regenerated when we bump the rdoc gem. Generating at release time handles that automatically.
  • JRuby-specific docs (the list in ri docs are not available for core #9049: jruby/java extensions, jar-dependencies, …) could come later. The Ruby sources under core/src/main/ruby/jruby already go through rdoc. The Java extensions would need an rdoc parser for @JRubyMethod + Javadoc, and most of those methods don't have doc comments yet.
  • Short term: should we drop or skip the current FTP step on master (10.1) until this is in place? The data it ships can't be used there anyway.

@headius what do you think? If it sounds good, I'm happy to start on the release-time generation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions