hmm, this kind of surprises me. i'm going to take some time and try to better understand the issue. but i should have this in soon. thanks!

yea, if this works without the move_jars_to_webinf_lib we should definitely get rid of it. I'll merge this and kill the commented sections. thanks again.

interesting, I wonder if it would still work with those that do not expand the .war archive (rare but still).
might have some related issue as well at jruby/activerecord-jdbc-adapter#536 ... but it ain't confirmed yet

the problem before the merge was that warbler copied the jopenssl.jar to WEB-INF/lib and that needs bouncycastle jars. but BC were in jruby-classloader which has the webapp-classloader as parent, i.e. jopenssl.jar could not see the bouncy-castle jars

your problem might be related - I do understand to little what 'arjdbc/jdbc/java' does - but with that patch applied the whole warble warfile is much closer to jruby -S rails server which could solve the activerecord issue as well.

Thanks for the clarification Christian, that call just triggers a require '.../adapter.jar' which contains the native extension classes compiled ... in this particular case it seems to be not getting loaded up correctly. Which is strange but might have something to do with how the application is warbled.

My concern (even as we ignore the issue mentioned) is if this would work with all the JRuby versions supported esp. if you consider not all application servers explode the .war archive - thus JRuby needs to load the jars (missing from WEB-INF/lib) using jar: references ... but maybe I assumed wrong and this does not effect such cases ?!

yes, I also wonder if a packed warfile can load those bouncy-castle jars
from within WEB-INF/lib/jruby-complete.jar, i.e. it is jar in a jar in a
jar ;)
I agree that sound like problems have to pop up. maybe I find time to run
packed warfile and see (at least openssl).

exactly - see it will probably work in most cases but probably not when say on Tomcat unpackWARs=false
... not an uncommon thing with other servers as well.

With your change in place our warfile lib directory went from:

gems-gems-activerecord-jdbc-adapter-1.3.6-lib-arjdbc-jdbc-adapter_java.jar
gems-gems-bcrypt-ruby-3.1.2-java-lib-bcrypt_ext.jar
gems-gems-jdbc-mysql-5.1.28-lib-mysql-connector-java-5.1.28-bin.jar
gems-gems-jruby-jars-1.7.10-lib-jruby-core-complete-1.7.10.jar
gems-gems-jruby-jars-1.7.10-lib-jruby-stdlib-complete-1.7.10.jar
gems-gems-jruby-rack-1.1.13.3-lib-jruby-rack-1.1.13.3.jar
gems-gems-json-1.8.1-java-lib-json-ext-generator.jar
gems-gems-json-1.8.1-java-lib-json-ext-parser.jar
gems-gems-therubyrhino_jar-1.7.4-jar-rhino-1.7R4.jar
gems-gems-warbler-1.4.0-lib-warbler_jar.jar

to

jruby-core-complete-1.7.10.jar
jruby-rack-1.1.13.3.jar
jruby-stdlib-complete-1.7.10.jar

I don't think this was the intention was it? I don't think this was the intention was it? Unless I'm not understanding? Reason for my interest is extremely rare

uninitialized constant Krypt::ASN1::BOOLEAN

errors upon starting tomcat. So rare that I can go almost half a day of starting and stopping tomcat and only see it once. So I was thinking an issue with warbler in copying out the jar files within the jar files.

@malletto thanks ... would you please maybe setup a minimal rails app where this case be reproduced ...

@malletto I do not get it completely. so before the change you saw never such

uninitialized constant Krypt::ASN1::BOOLEAN

ans now you see it occasionally ? is that about what you are saying ?

@kares I wanted to had one more thing to our discussion. just look at a normal java servlet project they have to pack jruby-complete.jar (or the modulare maven artifact) into WEB-INF/lib and aspect it to work, there you do not have a 'warbler' which starts to move jar file from gems or move from within jruby-complete.jar. so if things are not working it is a loading problem with jruby itself.

@malletto looking at your jar list:

gems-gems-jruby-jars-1.7.10-lib-jruby-core-complete-1.7.10.jar
gems-gems-jruby-jars-1.7.10-lib-jruby-stdlib-complete-1.7.10.jar
gems-gems-jruby-rack-1.1.13.3-lib-jruby-rack-1.1.13.3.jar

are the same jars as

jruby-core-complete-1.7.10.jar
jruby-rack-1.1.13.3.jar
jruby-stdlib-complete-1.7.10.jar

Krypt::ASN1::BOOLEAN is part of ruby-stdlib-complete-1.7.10.jar or in gems-gems-jruby-jars-1.7.10-lib-jruby-stdlib-complete-1.7.10.jar

so it is really hard to see a difference between the two setups regarding Krypt::ASN1::BOOLEAN

would be nice to see the output of jruby when the system property
debug.loadService=true
is set.

or the obvious question from my claim that there is no difference: did you try the same start+stop tomcat debug cycle with the old setup as well ?

@mkristian very true, but historically Warbler did the moving pretty "smart" ... this likely avoided some of the loading issues JRuby had (and might still have) e.g. the edge case with non-exploded .war(s), also some web-servers might pop up having issues. maybe what would probably the best - to move things forward while keeping an option for this "old but still useful" behaviour. so I would like to propose a Warbler configuration option something like config.move_jars_to_webinf_lib (by default false) ... ideas?

@kares agreed

maybe ask the user to report an issue if they NEED to use the old behaviour ;)

@jkutner what do you think about the extra config ?

@kares for jruby-9000.dev the copy thing will have extra side-effects since the classloader there looks first to loaded jar via require before looking into the parent classloader. i.e. you can have a servlet using bouncy-castle-1.50 and jruby can safely use bouncy-castle-1.47 which is bundled right now.

Yea, I like the extra config. I'll create a ticket and I can do that work myself.

Have a look at #243 and see if that's what you had in mind.

+1

@mkristian that BC loading does sounds pretty scary ... I'm not sure why and if I wanted that (no "option" to "override" the BC jar). I actually did some BC internals hacking with JRuby - might open a pull, would love to hear your input as well ... maybe it turns out all stupid but I'd like to avoid leakage with multiple runtimes and it seems quite necessary to "avoid" registering the provider.

@kares my BC loading was just example. any servlet container allows to reverse the classloader from parent container to separate your WEB-INF/lib from whatever is load in the parent classloader. next time I use jline.jar as example or xerces from nokogiri ;)

reagarding the registering the security provider in jriuby-openssl: I saw the code once and had the feeling that is not the job of jruby to register such a provider - just my thoughts. happy to give another pair of eyes on BC loading code of yours.

Yes will do on monday. I'll report back my findings.

On Sat, Mar 1, 2014 at 3:17 AM, Karol Bucek notifications@github.comwrote:

@malletto https://github.com/malletto thanks ... would you please maybe
setup a minimal rails app where this case be reproduced ...

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/241#issuecomment-36419345
.

No sorry for the confusion. We get very rare Krypt::ASN1::BOOLEAN errors
before the change. Since I'm grasping at straws as to the reason I
attempted the change to see what would happen and noticed the differences.
I haven't attempted to start my server but will do after I setup a
barebones rails app to see the before and after differences.

On Sat, Mar 1, 2014 at 4:42 AM, Christian Meier notifications@github.comwrote:

@malletto https://github.com/malletto I do not get it completely. so
before the change you saw never such

uninitialized constant Krypt::ASN1::BOOLEAN

ans now you see it occasionally ? is that about what you are saying ?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/241#issuecomment-36420636
.

@kares I setup a barebones rails app. My gemfile contains:

source 'http://rubygems.org'

# Package dependency manager
gem 'bundler'

# Ruby on Rails
gem 'rails', '3.2.17'

platforms :jruby do
  gem 'activerecord-jdbcsqlite3-adapter', '1.3.3'
  gem 'jruby-rack', '1.1.13.3'
  gem 'jruby-jars', '1.7.10'

  # Not needed in production, but needed on our build server
  # Must be >= 1.4.0 to pick up fixes #144 and #180
  gem 'warbler', '1.4.0'
end

With the modified warbler my lib directory looks like:

ls -l
total 43232
drwxr-xr-x  2         68 Mar  3 09:16 assets
-rwxr-xr-x  1   13243641 Mar  3 09:16 jruby-core-complete-1.7.10.jar
-rwxr-xr-x  1     243757 Mar  3 09:16 jruby-rack-1.1.13.3.jar
-rwxr-xr-x  1    8639499 Mar  3 09:16 jruby-stdlib-complete-1.7.10.jar
drwxr-xr-x  2         68 Mar  3 09:16 tasks

With the original unmodified warbler it looks like:

ls -l
total 93160
drwxr-xr-x  2        68 Mar  3 09:15 assets
-rwxr-xr-x  1    134889 Mar  3 09:15 gems-gems-activerecord-jdbc-adapter-1.3.6-lib-arjdbc-jdbc-adapter_java.jar
-rwxr-xr-x  1   3201128 Mar  3 09:15 gems-gems-jdbc-sqlite3-3.7.2.1-lib-sqlite-jdbc-3.7.2.jar
-rwxr-xr-x  1  13243641 Mar  3 09:15 gems-gems-jruby-jars-1.7.10-lib-jruby-core-complete-1.7.10.jar
-rwxr-xr-x  1   8639499 Mar  3 09:15 gems-gems-jruby-jars-1.7.10-lib-jruby-stdlib-complete-1.7.10.jar
-rwxr-xr-x  1    243757 Mar  3 09:15 gems-gems-jruby-rack-1.1.13.3-lib-jruby-rack-1.1.13.3.jar
-rwxr-xr-x  1     36523 Mar  3 09:15 gems-gems-json-1.8.1-java-lib-json-ext-generator.jar
-rwxr-xr-x  1     29097 Mar  3 09:15 gems-gems-json-1.8.1-java-lib-json-ext-parser.jar
-rwxr-xr-x  1     16846 Mar  3 09:15 gems-gems-warbler-1.4.0-lib-warbler_jar.jar
-rwxr-xr-x  1  13243641 Mar  3 09:15 jruby-core-complete-1.7.10.jar
-rwxr-xr-x  1    243757 Mar  3 09:15 jruby-rack-1.1.13.3.jar
-rwxr-xr-x  1   8639499 Mar  3 09:15 jruby-stdlib-complete-1.7.10.jar
drwxr-xr-x  2        68 Mar  3 09:15 tasks

Created the war with:

RAILS_ENV=production bundle exec warble executable war

Running the 2 war files (since they are executable) with java -jar <war_file> they both start. So maybe this is just a cosmetic thing? Not sure. This doesn't get me any closer to figuring out our extremely random (maybe 4% of the time) Krypt::ASN1::BOOLEAN error problem that has been bothering us for months though, and I have no idea if its related.

@malletto I just downloaded your testapp from jruby/jruby#1119 (comment)

and the first thing I could see were some warning:
WARNING: while creating new bindings for class org.jruby.ext.krypt.asn1.RubyTemplate$RubyAsn1Template,

and you have 8 jruby runtimes. so the first thought is that all the problems you see are concurrency issue with krypt gem ! especially since they are rare and thus depending on execution timings.

is the strong reason for having 8 runtimes and not only 1 ?

there is krypt-0.0.2.rc1 gem now, could you add that to your Gemfile and see if that works better. that new gem is already part of jruby-1.7.12-SNAPSHOT and on jruby-head

@mkristian Yeah we aren't running in threadsafe mode, yet. So we need to have a pool of runtimes. I'll look at the new rc1 of krypt and see if I can add that to my test app.

Not sure why we would have concurrency issues if there are 8 separate runtimes? Aren't they all self contained?

@malletto I have one more thing for you since it is really a startup problem:
with jruby-rack you can set the context-param
jruby.runtime.init.serial
to true
that should work around your problem (not solving the underlying issue with the krypt extension)

@mkristian I'll check that out, thanks. What does it do though? I looked in the documentation:

When using runtime pooling, this flag indicates that the pool should be created serially in the foreground rather than spawning (background) threads, it's by default off (set to false). For environments where creating threads is not permitted.

Any negatives to setting to true?

it does start each jruby runtime one after the other. there is no use of
concurrency during the startup of the runtime which might take a few
seconds longer ;)