melting-pot: generate another build script layer

ctrueden · ctrueden · commit 66c777b56fa6 · 2019-06-30T22:19:12.000+02:00
This layer builds all the modules one by one, all pinned to the same versions, but all separately, rather than in one multi-module build. This means that each build is more isolated and standalone. A succinct summary of each component's success or failure is now emitted, so the situation can be perused more quickly, rather than needing to wade through a giant build log on the console for each component. Each build is now logged to its own build.log file. It also avoids a problem we encountered while attempting to melt the Fiji project recently: the imagej-scripting component has failing tests when they run as part of a multi-module build including imglib2, but has passing tests otherwise. Here is a transcript of the first failure: [INFO] Running net.imagej.scripting.ImageJ2ScriptTest log4j:WARN No appenders could be found for logger (org.bushe.swing.event.EventService). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. [ERROR] null Traceback (most recent call last): File "/Users/curtis/code/fiji/fiji/melting-pot/net.imagej/imagej-scripting/target/classes/script_templates/ImageJ2/Rotate_Stack.py", line 17, in <module> from net.imglib2.view import Views java.lang.NoClassDefFoundError: net/imglib2/view (wrong name: net/imglib2/View) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.python.core.Py.loadAndInitClass(Py.java:1107) at org.python.core.Py.findClassInternal(Py.java:1042) at org.python.core.Py.findClassEx(Py.java:1093) at org.python.core.packagecache.SysPackageManager.findClass(SysPackageManager.java:142) at org.python.core.packagecache.PackageManager.findClass(PackageManager.java:33) at org.python.core.packagecache.SysPackageManager.findClass(SysPackageManager.java:130) at org.python.core.PyJavaPackage.__findattr_ex__(PyJavaPackage.java:134) at org.python.core.PyObject.__findattr__(PyObject.java:965) at org.python.core.PyObject.impAttr(PyObject.java:1103) at org.python.core.imp.import_next(imp.java:840) at org.python.core.imp.import_logic(imp.java:905) at org.python.core.imp.import_module_level(imp.java:970) at org.python.core.imp.importName(imp.java:1057) at org.python.core.ImportFunction.__call__(__builtin__.java:1280) at org.python.core.PyObject.__call__(PyObject.java:450) at org.python.core.__builtin__.__import__(__builtin__.java:1232) at org.python.core.imp.importFromAs(imp.java:1149) at org.python.core.imp.importFrom(imp.java:1124) at org.python.pycode._pyx0.f$0(/Users/curtis/code/fiji/fiji/melting-pot/net.imagej/imagej-scripting/target/classes/script_templates/ImageJ2/Rotate_Stack.py:89) at org.python.pycode._pyx0.call_function(/Users/curtis/code/fiji/fiji/melting-pot/net.imagej/imagej-scripting/target/classes/script_templates/ImageJ2/Rotate_Stack.py) at org.python.core.PyTableCode.call(PyTableCode.java:171) at org.python.core.PyCode.call(PyCode.java:18) at org.python.core.Py.runCode(Py.java:1614) at org.python.core.__builtin__.eval(__builtin__.java:497) at org.python.core.__builtin__.eval(__builtin__.java:501) at org.python.util.PythonInterpreter.eval(PythonInterpreter.java:259) at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:57) at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31) at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264) at org.scijava.script.ScriptModule.run(ScriptModule.java:160) at org.scijava.module.ModuleRunner.run(ModuleRunner.java:168) at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127) at org.scijava.module.ModuleRunner.call(ModuleRunner.java:66) at org.scijava.thread.DefaultThreadService$3.call(DefaultThreadService.java:238) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) java.lang.NoClassDefFoundError: java.lang.NoClassDefFoundError: net/imglib2/view (wrong name: net/imglib2/View) at org.python.core.Py.JavaError(Py.java:552) at org.python.core.Py.findClassEx(Py.java:1099) at org.python.core.packagecache.SysPackageManager.findClass(SysPackageManager.java:142) at org.python.core.packagecache.PackageManager.findClass(PackageManager.java:33) at org.python.core.packagecache.SysPackageManager.findClass(SysPackageManager.java:130) at org.python.core.PyJavaPackage.__findattr_ex__(PyJavaPackage.java:134) at org.python.core.PyObject.__findattr__(PyObject.java:965) at org.python.core.PyObject.impAttr(PyObject.java:1103) at org.python.core.imp.import_next(imp.java:840) at org.python.core.imp.import_logic(imp.java:905) at org.python.core.imp.import_module_level(imp.java:970) at org.python.core.imp.importName(imp.java:1057) at org.python.core.ImportFunction.__call__(__builtin__.java:1280) at org.python.core.PyObject.__call__(PyObject.java:450) at org.python.core.__builtin__.__import__(__builtin__.java:1232) at org.python.core.imp.importFromAs(imp.java:1149) at org.python.core.imp.importFrom(imp.java:1124) at org.python.pycode._pyx0.f$0(/Users/curtis/code/fiji/fiji/melting-pot/net.imagej/imagej-scripting/target/classes/script_templates/ImageJ2/Rotate_Stack.py:89) at org.python.pycode._pyx0.call_function(/Users/curtis/code/fiji/fiji/melting-pot/net.imagej/imagej-scripting/target/classes/script_templates/ImageJ2/Rotate_Stack.py) at org.python.core.PyTableCode.call(PyTableCode.java:171) at org.python.core.PyCode.call(PyCode.java:18) at org.python.core.Py.runCode(Py.java:1614) at org.python.core.__builtin__.eval(__builtin__.java:497) at org.python.core.__builtin__.eval(__builtin__.java:501) at org.python.util.PythonInterpreter.eval(PythonInterpreter.java:259) at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:57) at org.python.jsr223.PyScriptEngine.eval(PyScriptEngine.java:31) at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:264) at org.scijava.script.ScriptModule.run(ScriptModule.java:160) at org.scijava.module.ModuleRunner.run(ModuleRunner.java:168) at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127) at org.scijava.module.ModuleRunner.call(ModuleRunner.java:66) at org.scijava.thread.DefaultThreadService$3.call(DefaultThreadService.java:238) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NoClassDefFoundError: net/imglib2/view (wrong name: net/imglib2/View) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:411) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.python.core.Py.loadAndInitClass(Py.java:1107) at org.python.core.Py.findClassInternal(Py.java:1042) at org.python.core.Py.findClassEx(Py.java:1093) ... 35 more The relevant issue in the Jython bug tracker is probably: https://bugs.jython.org/issue2666 The bug may be fixed on the jython 2.7.2 development branch, although I did not test as of this writing. Unfortunately, I was not able to concoct an MCVE; nothing I tried was able to reproduce the problem except for a melting-pot multi-module build that includes imglib2 and imagej-scripting. When running a multi-module build not including imglib2, the apparent difference is that the imagej-scripting project's classpath includes the imglib2 JAR, rather than the linked imglib2 module's target/classes folder. But my naive attempts to create an MCVE with an analogous classpath structure did not yield failure to import classes within Jython.
diff --git a/melting-pot.sh b/melting-pot.sh
@@ -592,16 +592,28 @@ meltDown() {
 	info "Generating aggregator POM"
 	generatePOM
 
-	# Build everything.
+	# Generate build scripts.
 	echo "mvn $args \\\\\n  test \$@" > build.sh
+	echo '#!/bin/sh
+trap "exit" INT
+echo "Melting the pot..."
+dir=$(pwd)
+for f in */*
+do (
+  cd "$f"
+  sh "$dir/build.sh" > build.log 2>&1 &&
+    echo "[SUCCESS] $f" ||
+    echo "[FAILURE] $f"
+) done' > melt.sh
+
+	# Build everything.
 	if [ "$skipBuild" ]
 	then
-		info "Skipping the build; run build.sh to do it."
+		info "Skipping the build; run melt.sh to do it."
 	else
 		info "Building the project!"
 		# NB: All code is fresh; no need to clean.
-		debug "mvn $args test"
-		sh build.sh
+		sh melt.sh
 	fi
 
 	info "$1: complete"
