Environment Information
JRuby 9.2.19.0
JRUBY_OPTS is empty
JAVA_OPTS is empty
Mac OSX 10.15.7 (Darwin beam-splitter.local 19.6.0 Darwin Kernel Version 19.6.0: Thu May 6 00:48:39 PDT 2021; root:xnu-6153.141.33~1/RELEASE_X86_64 x86_64)

Other relevant info:
I installed JRuby using RVM version 1.29.12. I installed the JDK by downloading it from Oracle directly.

Notably, both my JRUBY_OPTS and JAVA_OPTS were empty during the replication.

Replication
Running IRB:

jruby-9.2.19.0 :001 > require "open3"
 => true 
jruby-9.2.19.0 :002 > return_value, captured_err, status = Open3.capture3('git --version')
git version 2.32.0
 => ["", "", #<Process::Status: pid 99327 exit 0>] 

Expected Behavior
This first string should be "git version 2.32.0\n".

Actual Behavior
It's actually "".

This is easy enough to fix - I can add JAVA_OPTS as follows and resolve the issue for myself:
export JAVA_OPTS="--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED"

However, I found this case because I was using Bundler to pull a custom gem (gem '...', git: '...') and Bundler uses Open3 under the covers. Because this failure is silent (returns an empty string) Bundler breaks in a very messy but cryptic way:

NoMethodError: undefined method `[]' for nil:NilClass
  /Users/worldnamer/.rvm/gems/jruby-9.2.19.0/gems/bundler-2.2.20/lib/bundler/source/git/git_proxy.rb:82:in `version'
  /Users/worldnamer/.rvm/gems/jruby-9.2.19.0/gems/bundler-2.2.20/lib/bundler/source/git/git_proxy.rb:256:in `supports_minus_c?'
...

This is particularly difficult because the Open3 interface doesn't give a clean way to indicate that anything actually failed - stdout and stderr might have had values but we don't know them. The process completed gracefully so there's no error code. So I think the only mechanism left would be to throw an exception, but it would probably need to be a custom exception.

In any case, adding the JAVA_OPTS did the trick, and IRB warns you of this when you run IRB without the permissions:

2021-06-17T12:42:45.891-05:00 [main] WARN FilenoUtil : Native subprocess control requires open access to the JDK IO subsystem
Pass '--add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.base/java.io=ALL-UNNAMED' to enable.

This error does get emitted when we run Bundler, as well! It's just very unclear these things are related. After consulting on matrix, it seemed the consensus was that it would be best if Open3 opened the correct Java libraries such that nothing needed to be passed through JAVA_OPTS in the first place. In any case, adding this ticket so that the appropriate resolution can be found.