Skip to content
This repository was archived by the owner on Oct 26, 2026. It is now read-only.

Scijava/scijava ops engine/skip map history - #64

Merged
ctrueden merged 6 commits into
mainfrom
scijava/scijava-ops-engine/skip-map-history
Oct 31, 2022
Merged

ctrueden merged 6 commits into
mainfrom
scijava/scijava-ops-engine/skip-map-history

Conversation

@hinerm

@hinerm hinerm commented Oct 21, 2022

Copy link
Copy Markdown
Member

Closes scijava/scijava#99

See also #100 and #101

@hinerm
hinerm requested review from ctrueden and gselzer October 21, 2022 21:14

@ctrueden ctrueden left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks awesome. I commented on one small thing, but I think we should merge this as-is as soon as possible, to get the nondeterministic failures under control.


if (centerAwareOp instanceof RichOp) {
((RichOp)centerAwareOp).metadata().hints().minus(
BaseOpHints.History.SKIP_RECORDING);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If SKIP_RECORDING hint was already set before getting plussed in above... then should we still be minusing it out again here? Don't we just want to restore the hints back to their original state?

@ctrueden

ctrueden commented Oct 21, 2022 •

Copy link
Copy Markdown
Member

On this branch, I do get one failure in imagej-ops2, on both the CLI and in IntelliJ:

[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.755 s <<< FAILURE! - in net.imagej.ops2.features.tamura2d.Tamura2dFeatureTest
[ERROR] testCoarsenessFeature  Time elapsed: 0.639 s  <<< ERROR!
java.lang.ArrayIndexOutOfBoundsException
	at net.imagej.ops2@0-SNAPSHOT/net.imagej.ops2.features.tamura2d.Tamura2dFeatureTest.testCoarsenessFeature(Tamura2dFeatureTest.java:65)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 10000 out of bounds for length 10000

According to the stack trace, it's still trying to log the output of individual pixels. It's like the hint isn't being respected.

@ctrueden
ctrueden self-requested a review October 21, 2022 21:36

@ctrueden ctrueden left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On second thought, I think we better get to the bottom of the hint-propagation failure before merging this. I strongly considered just commenting out the offending test, but it's going to bite us sooner rather than later if we do that.

@ctrueden
ctrueden force-pushed the scijava/scijava-ops-engine/skip-map-history branch from 938d24c to 8af31af Compare October 21, 2022 21:40
@hinerm
hinerm force-pushed the scijava/scijava-ops-engine/skip-map-history branch from 8af31af to 0ecefdd Compare October 24, 2022 15:33
The ProvenanceTest relies on history being recorded and this always
skipping seems aggressive.
History.SKIP_RECORDING hint is used to skip history recording in the
AbstractRichOp constructor, so this hint should also be respected in the
postprocess step of recording output.
This isn't the center-aware impl
@hinerm
hinerm force-pushed the scijava/scijava-ops-engine/skip-map-history branch from 0ecefdd to 3253890 Compare October 24, 2022 16:01
@hinerm

hinerm commented Oct 24, 2022

Copy link
Copy Markdown
Member Author

On second thought, I think we better get to the bottom of the hint-propagation failure before merging this. I strongly considered just commenting out the offending test, but it's going to bite us sooner rather than later if we do that.

@ctrueden see scijava/scijava#101 (comment)

There are cases we want to control what the Hints are during some stages
of matching.
Per-pixel operations with imglib2 we can reasonably assume to be
operating on Type wrappers that have some level of reuse. These are not
thread-safe to use as output keys in our history, thus we cannot be
recording per-pixel history.
@hinerm
hinerm force-pushed the scijava/scijava-ops-engine/skip-map-history branch from 3253890 to 94ae742 Compare October 31, 2022 19:12
@hinerm

hinerm commented Oct 31, 2022

Copy link
Copy Markdown
Member Author

@ctrueden @gselzer I just added a commit allowing the updating of Hints in an OpMetadata.. these tests are passing consistently for me now. Can you test a few times on your end and see if you get any failures?

@ctrueden
ctrueden self-requested a review October 31, 2022 19:39

@ctrueden ctrueden left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With OpenJDK 17.0.3, there is an unrelated JPMS issue with scijava-persist
[INFO] Running org.scijava.persist.SerializationTests
IClassAdapters : 
IRunTimeClassAdapters : 
	 interface org.scijava.persist.Shape
	 	 class org.scijava.persist.Circle
	 	 class org.scijava.persist.Shapes$Diamond
	 	 class org.scijava.persist.Shapes$Rectangle
IClassAdapters : 
IRunTimeClassAdapters : 
	 interface org.scijava.persist.Shape
	 	 class org.scijava.persist.Circle
	 	 class org.scijava.persist.Shapes$Diamond
	 	 class org.scijava.persist.Shapes$Rectangle
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.213 s <<< FAILURE! - in org.scijava.persist.SerializationTests
[ERROR] testCircleObject  Time elapsed: 0.192 s  <<< ERROR!
java.lang.reflect.InaccessibleObjectException: Unable to make field int org.scijava.persist.Circle.radius accessible: module org.scijava.persist does not "opens org.scijava.persist" to module com.google.gson
	at org.scijava.persist@0-SNAPSHOT/org.scijava.persist.SerializationTests.testSerializationDeserialization(SerializationTests.java:76)
	at org.scijava.persist@0-SNAPSHOT/org.scijava.persist.SerializationTests.testCircleObject(SerializationTests.java:50)

[ERROR] testDrawingObject  Time elapsed: 0.012 s  <<< ERROR!
java.lang.reflect.InaccessibleObjectException: Unable to make field int org.scijava.persist.Circle.radius accessible: module org.scijava.persist does not "opens org.scijava.persist" to module com.google.gson
	at org.scijava.persist@0-SNAPSHOT/org.scijava.persist.SerializationTests.testSerializationDeserialization(SerializationTests.java:76)
	at org.scijava.persist@0-SNAPSHOT/org.scijava.persist.SerializationTests.testDrawingObject(SerializationTests.java:62)

With OpenJDK 11.0.16, I got the following results:

  1. BUILD SUCCESS on Ubuntu
  2. BUILD SUCCESS on macOS
  3. BUILD FAILURE on Ubuntu: org.scijava.ops.api.features.OpMatchingException: No MatchingRoutine was able to produce a match! at net.imagej.ops2@0-SNAPSHOT/net.imagej.ops2.identity.IdentityTest.testIdentity(IdentityTest.java:49)
  4. BUILD SUCCESS on macOS
  5. BUILD SUCCESS on Ubuntu

This No MatchingRoutine was able to produce a match! error is not new; it was happening intermittently previously, and will need investigation, but I do not think it should hold up merging of this PR, which does seem to address the AIOOBEs and likely performance issues of map operations.

@ctrueden
ctrueden merged commit bdb8df8 into main Oct 31, 2022
@ctrueden
ctrueden deleted the scijava/scijava-ops-engine/skip-map-history branch October 31, 2022 19:47
@gselzer gselzer mentioned this pull request Aug 28, 2023
5 tasks done
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TamuraFeature2dTest#testCoarsenessFeature AIOOBE

2 participants