Well this is a first...we have a bug that only manifests when running in mixed mode.

~/projects/jruby/maven/jruby/target/it/runnable/tmp $ JARS_HOME=. GEM_PATH=. GEM_HOME=$GEM_PATH java -cp . org.jruby.Main -X-C -e 'require "thread_safe"; require "virtus"; require "rspec/mocks";'

~/projects/jruby/maven/jruby/target/it/runnable/tmp $ JARS_HOME=. GEM_PATH=. GEM_HOME=$GEM_PATH java -cp . -Djruby.jit.threshold=0 org.jruby.Main -e 'require "thread_safe"; require "virtus"; require "rspec/mocks";'

~/projects/jruby/maven/jruby/target/it/runnable/tmp $ JARS_HOME=. GEM_PATH=. GEM_HOME=$GEM_PATH java -cp . org.jruby.Main -e 'require "thread_safe"; require "virtus"; require "rspec/mocks";'
LoadError: no such file to load -- rspec/mocks
     require at org/jruby/RubyKernel.java:966
  __script__ at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:58
  __script__ at -e:1

Something inside RubyGems is breaking when running under mixed-mode execution that works in pure-interpreter or pure-jit. I dug into it a bit, and it looks like variables are getting munged, probably due to call protocols not aligning correctly. I have not dug into it any further.

This is the cause of -Pmain failure right now. You can run the short version via:

mvn clean install -U -Pmain -Dinvoker.test=runnable

You'll get the same rspec/mocks error. If you unpack the runnable.jar it creates into a temporary directory, you should be able to run the commands above to reproduce. It will be running JRuby entirely from within that dir, so use those files for investigation.

What I Know

I tracked the problem as far as Gem.try_activate, which calls find_inactive_path and find_by_path. Those eventually get to BasicSpecification#contains_requirable_file?, and then here things started to go screwy. The base and suffixes variables could be seen to change values; base would be correct outside the suffixes.any? loop, but an empty string inside. suffixes changed to base's value at one point as I added and removed puts logging. I believe at one point it started working...I had added enough code that it compiled differently.

That's about all I have.