Skip to content

[ci] an attempt to run tests on Windows#6340

Merged
kares merged 23 commits intojruby:masterfrom
kares:win-ci
Sep 18, 2020
Merged

[ci] an attempt to run tests on Windows#6340
kares merged 23 commits intojruby:masterfrom
kares:win-ci

Conversation

@kares
Copy link
Member

@kares kares commented Jul 23, 2020

a bit of Windows CI de-ja-vu, this time trying to run test:jruby
chosen JRuby suite as it is very stable, and in out control, compared to say ruby-specs

motivated by: #6322 (can re-target to jruby-9.2 if needed)

@kares
Copy link
Member Author

kares commented Jul 23, 2020

currently at: 98.8722% passed

2020-07-23T14:59:21.1879431Z --------------------------------------------------------------------------------
2020-07-23T14:59:21.1880487Z 799 tests, 4434 assertions, 3 failures, 3 errors, 3 pendings, 1 omissions, 0 notifications

Most of the (relevant) failures are Windows compatibility bugs, but I am fine with pending those for now:

2020-07-23T14:58:06.5244096Z ================================================================================
2020-07-23T14:58:06.5244400Z Failure: test_stat_directory_in_jar_with_trailing_slash(TestDir)
2020-07-23T14:58:06.5244684Z D:/a/jruby/jruby/test/jruby/test_dir.rb:355:in `test_stat_directory_in_jar_with_trailing_slash'
2020-07-23T14:58:06.5244994Z      352:     assert File.exist?(source_file), "test is wrong, #{source_file} doesn't even exist"
2020-07-23T14:58:06.5245338Z      353:     assert_equal false, File.directory?(source_file)
2020-07-23T14:58:06.5245602Z      354:     assert_equal false, File.directory?(source_file + "/")
2020-07-23T14:58:06.5249688Z   => 355:     assert_raise(Errno::ENOENT) do
2020-07-23T14:58:06.5258930Z      356:       File.stat(source_file + "/")
2020-07-23T14:58:06.5259537Z      357:     end
2020-07-23T14:58:06.5267668Z      358:     source_dir = File.dirname(source_file)
2020-07-23T14:58:06.5472581Z org/jruby/RubyKernel.java:1263:in `catch'
2020-07-23T14:58:06.5473543Z org/jruby/RubyKernel.java:1258:in `catch'
2020-07-23T14:58:06.5474043Z org/jruby/RubyKernel.java:1263:in `catch'
2020-07-23T14:58:06.5742289Z org/jruby/RubyKernel.java:1258:in `catch'
2020-07-23T14:58:06.5840571Z 
2020-07-23T14:58:06.5841831Z <Errno::ENOENT> expected but was
2020-07-23T14:58:06.5842681Z <SystemCallError(<Unknown error (SystemCallError) - Unknown Error (123) - jar:file:D:/a/jruby/jruby/test/jruby/jar_with_relative_require1.jar!/test/require_relative1.rb/>)>

2020-07-23T14:58:34.1725684Z ================================================================================
2020-07-23T14:58:34.1728354Z Failure: test_ios_with_incompatible_flags(TestIO): <Errno::EINVAL> exception was expected but none was thrown.
2020-07-23T14:58:34.1732791Z D:/a/jruby/jruby/test/jruby/test_io.rb:97:in `test_ios_with_incompatible_flags'
2020-07-23T14:58:34.1746592Z       94:     # original IO
2020-07-23T14:58:34.1747324Z       95:     f = File.new(@file2, "w")
2020-07-23T14:58:34.1747783Z       96:     @to_close << f
2020-07-23T14:58:34.1771560Z   =>  97:     assert_raises(Errno::EINVAL) { IO.new(f.fileno, "r") }
2020-07-23T14:58:34.1773498Z       98:     f.close
2020-07-23T14:58:34.1775280Z       99: 
2020-07-23T14:58:34.1781057Z      100:     f = File.new(@file, "r")
2020-07-23T14:58:34.1790417Z org/jruby/RubyKernel.java:1263:in `catch'
2020-07-23T14:58:34.1792135Z org/jruby/RubyKernel.java:1258:in `catch'
2020-07-23T14:58:34.1796098Z org/jruby/RubyKernel.java:1263:in `catch'
2020-07-23T14:58:34.1806378Z org/jruby/RubyKernel.java:1258:in `catch'
2020-07-23T14:58:34.1808508Z ================================================================================

2020-07-23T14:58:34.2940083Z ================================================================================
2020-07-23T14:58:34.2940349Z Failure: test_reopen(TestIO)
2020-07-23T14:58:34.2940576Z D:/a/jruby/jruby/test/jruby/test_io.rb:145:in `test_reopen'
2020-07-23T14:58:34.2940810Z      142:     file2_fileno = file2.fileno;
2020-07-23T14:58:34.2941049Z      143:     file2 = file2.reopen(file)
2020-07-23T14:58:34.2941271Z      144:     assert_equal(file.pos, file2.pos)
2020-07-23T14:58:34.2941496Z   => 145:     assert_equal(file2_fileno, file2.fileno);
2020-07-23T14:58:34.2941721Z      146:     assert(file.fileno != file2.fileno);
2020-07-23T14:58:34.2941914Z      147:     file2.close
2020-07-23T14:58:34.2942130Z      148:     file.close
2020-07-23T14:58:34.2942349Z org/jruby/RubyKernel.java:1263:in `catch'
2020-07-23T14:58:34.2942572Z org/jruby/RubyKernel.java:1258:in `catch'
2020-07-23T14:58:34.2942788Z org/jruby/RubyKernel.java:1263:in `catch'
2020-07-23T14:58:34.2942998Z org/jruby/RubyKernel.java:1258:in `catch'
2020-07-23T14:58:34.2943207Z <3> expected but was
2020-07-23T14:58:34.2943415Z <4>
2020-07-23T14:58:34.2943635Z ================================================================================

2020-07-23T14:58:35.5389350Z ================================================================================
2020-07-23T14:58:35.5825573Z Error: test_unget_before_position_change(TestIO): IOError: byte oriented read for character buffered IO
2020-07-23T14:58:35.7342007Z org/jruby/RubyIO.java:3069:in `read'
2020-07-23T14:58:35.9200661Z org/jruby/RubyIO.java:3051:in `read'
2020-07-23T14:58:35.9754138Z D:/a/jruby/jruby/test/jruby/test_io.rb:310:in `block in test_unget_before_position_change'
2020-07-23T14:58:35.9813642Z      307:       f.read(3)
2020-07-23T14:58:35.9814310Z      308:       f.ungetc(100)
2020-07-23T14:58:35.9864250Z      309:       f.pos = 2
2020-07-23T14:58:35.9864858Z   => 310:       assert_equal("LLO", f.read(3))
2020-07-23T14:58:35.9865250Z      311: 
2020-07-23T14:58:35.9865553Z      312:       f.ungetc(100)
2020-07-23T14:58:35.9865951Z      313:       f.seek(2)
2020-07-23T14:58:35.9866233Z org/jruby/RubyIO.java:1160:in `open'
2020-07-23T14:58:35.9866531Z D:/a/jruby/jruby/test/jruby/test_io.rb:306:in `test_unget_before_position_change'
2020-07-23T14:58:35.9866831Z org/jruby/RubyKernel.java:1263:in `catch'
2020-07-23T14:58:35.9900120Z org/jruby/RubyKernel.java:1258:in `catch'
2020-07-23T14:58:35.9900493Z org/jruby/RubyKernel.java:1263:in `catch'
2020-07-23T14:58:35.9900787Z org/jruby/RubyKernel.java:1258:in `catch'
2020-07-23T14:58:35.9901085Z ================================================================================

let me know what you think, should we instead run bigger suites (e.g. the MRI ones) and exclude Windows failures?

@headius
Copy link
Member

headius commented Aug 14, 2020

@kares I am working on Windows a bit lately and would also like to see us running the specs and MRI tests. Specs first, since we know they have been "green enough" in the past. I will play with it a bit today to see how much we need to tag to get specs running to completion.

@headius
Copy link
Member

headius commented Aug 14, 2020

I have opened #6357 for Windows RubySpec fixes.

@kares
Copy link
Member Author

kares commented Aug 17, 2020

have put the Windows failing stuff (as pending) in commit-by-commit, still I got:
798 tests, 4430 assertions, 0 failures, 2 errors, 8 pendings, 1 omissions, 0 notifications
and I am not really seeing those 2 errors in the CI logs ... so this might need me to try running the specs on Windows, for real
😞

@kares
Copy link
Member Author

kares commented Aug 17, 2020

was able to locate the tests failures/errors by changing test-unit to always print summary at the end,
made it a default for all test:jruby runs on CI!

@headius
Copy link
Member

headius commented Aug 18, 2020

I am on Windows and could give those last few items a try here. For now I'd say just mark them pending and we can get to them as soon as we have time.

@kares
Copy link
Member Author

kares commented Aug 20, 2020

tests should be green (all failures are pended) but we still get a 1 exit status.
test-unit is only showing 1 omission and 10 pendings (no failures or errors) for the jruby suite.

@headius
Copy link
Member

headius commented Aug 21, 2020

I'll give your branch a try locally and try to figure out the bad exit.

Possibly related: what's up with all this:

WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens
WARNING: Unknown module: org.jruby.dist specified to --add-opens

@headius
Copy link
Member

headius commented Aug 21, 2020

I notice that -Ptest is still green on Windows, so we could add that too.

@headius
Copy link
Member

headius commented Aug 21, 2020

Here's the output I get at the end of rake test:jruby:

Finished in 117.995 seconds.
--------------------------------------------------------------------------------
798 tests, 4956 assertions, 0 failures, 1 errors, 10 pendings, 0 omissions, 0 notifications
98.6216% passed
--------------------------------------------------------------------------------
6.76 tests/s, 42.00 assertions/s
LoadError: no such file to load -- rbconfig
  require at org/jruby/RubyKernel.java:1043
   <main> at <script>:1
          at :1
          at :1
    catch at org/jruby/RubyKernel.java:1263
    catch at org/jruby/RubyKernel.java:1258
    catch at org/jruby/RubyKernel.java:1263
    catch at org/jruby/RubyKernel.java:1258
rake aborted!
Command failed with status (1)
C:\Users\headius\work\jruby\bin\rake:23:in `<main>'
Tasks: TOP => test:jruby => test:jruby:int
(See full trace by running task with --trace)

The one failure can be ignored; my environment has both Java 8 and 11 installed and they are stepping on each other.

I'll look into the rbconfig error.

@headius
Copy link
Member

headius commented Aug 21, 2020

The rbconfig failure appears to be coming from this test:

https://github.com/jruby/jruby/blob/254a32779c769f77a3898fcf1a998e38c579edbe/test/jruby/test_missing_jruby_home.rb

I'm not sure this should actually be expected to pass, since rbconfig.rb lives in the stdlib. We could consider moving it into the main jar, but as it is written I would not expect this test to pass normally.

I'm looking into why it triggers this late error, and will also consider how to make it pass.

@headius
Copy link
Member

headius commented Aug 21, 2020

Reproduces directly when running the contents of this test outside of the test harness, so it is at least a bad test:

>jruby blah.rb
LoadError: no such file to load -- rbconfig
  require at org/jruby/RubyKernel.java:1043
   <main> at <script>:1
   <main> at blah.rb:6
   <main> at blah.rb:5

@headius
Copy link
Member

headius commented Aug 21, 2020

I am unsure why the error is delayed. Usually this means we're leaving $! set somewhere and the final error output is not shown until the test harness or JRuby itself exits and sees that there's an in-flight error to print out.

In any case, I believe the following patch will resolve this, avoiding the attempt to load rbconfig without a JRUBY_HOME set:

diff --git a/test/jruby/test_missing_jruby_home.rb b/test/jruby/test_missing_jruby_home.rb
index c820a895d1..f99aa9499c 100644
--- a/test/jruby/test_missing_jruby_home.rb
+++ b/test/jruby/test_missing_jruby_home.rb
@@ -9,7 +9,7 @@ class TestMissingJRubyHome < Test::Unit::TestCase
     begin
       runtime = org.jruby.Ruby.new_instance
       assert_nothing_raised do
-        runtime.eval_scriptlet('require "rbconfig"')
+        assert_equal([0, 1, 2], runtime.eval_scriptlet('3.times.to_a'))
       end
     ensure
       java.lang.System.set_property('jruby.home', old) if old

I'll file a bug about rbconfig and some other libraries being stdlib-dependent when they may be safer to keep in the jar.

@kares kares marked this pull request as ready for review September 14, 2020 17:08
@kares
Copy link
Member Author

kares commented Sep 14, 2020

fixed the remaining Windows compatibility issues and added mvn -P test phase to follow rake test:jruby.
the commits that are worth revisiting (in terms of Windows support) are [test] TODO: ... prefixed.

@kares kares added this to the JRuby 9.3.0.0 milestone Sep 14, 2020
@headius
Copy link
Member

headius commented Sep 17, 2020

I believe this can be merged any time and we can investigate the pending items and TODOs from there. I think some of your fixes here may help specs so I want to land this before I tag off specs.

@headius
Copy link
Member

headius commented Sep 17, 2020

I'm going to go ahead ad land #6357 as is since it makes enough fixes to get specs running to completion. I did not add a Windows target since it makes more sense for me to wait until your PR lands. Then we can collab on getting a spec run and making it greener.

@kares kares merged commit e40431f into jruby:master Sep 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants