You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The matcher should be able to find Ops that do not exactly match the inputs/outputs, but where requested input/output argument types can be converted to/from the op candidate's actual parameter types via Converter plugins.
I have been thinking about this issue recently, especially with respect to primitives. It would be quite nice if the matcher had the ability to convert Integers to Doubles or Longs on the fly (these are my two bothersome use cases. I am wondering, however, where and/or when we would want to be doing the converting. It would be most convenient if we could place this logic within isAssignable: this would be by far the best place to do so if you are looking for the conversion while using ops.run. In comparison with another method like implementing transformations, it would be faster, more flexible, and require less transformation boilerplate, however we might (probably would) run into ClassCastExceptions. So we might have to go with the transformations.
@gselzer and I talked through two possible approaches for implementing converters:
As an op transformer.
Checking within the "assignability" layer.
We concluded that (1) is not tenable: the targets may not be finite/closed. Suppose you have a RAI<FloatType> and want to pass it to an op that requires a II<DoubleType>. As a human being, we can see that this is a straightforward two-step conversion: use Converters.convert to go from RAI<FloatType> to RAI<DoubleType> and then call Views.flatIterable to go from RAI to II. But from the perspective of a conversion transformer: given a RAI<FloatType>: what is the list of destination generic types you support (see the Collection<OpRef> getRefsTransformingTo(OpRef targetRef) method of OpTransformer)? It may not be a finite answer; you can only state, for a given destination generic type, whether you could do it. But there could be arbitrarily many supported destination types.
We lean toward (2) now because it would be feasible for any given converter plugin to answer the question "can you convert type T to type U?", regardless of the breadth/constraints on that converter plugin.
That said, for many (all?) converter plugins, they can be expressed as java.util.function.Functions? I.e. the sorts of types upon which they operate are expressible via the Java generics grammar? Seems true, but we didn't think deeply about all the possible sorts of converters we want to support. There is also an issue of not knowing if the conversion will be lossy without knowing the specific source and destination types for a particular conversion. Think e.g. RAI<T> to RAI<U> for T extends RealType<T> and U extends RealType<U>—it might be lossy depending on the exact T and U.
If we have a closed set of source and destination types supported by each converter plugin, we could maintain a data structure of all (A -> B) conversion chains. When a new converter for (P -> Q) is added to system, assuming we can efficiently query the existing conversion chains, we can look up all (A -> P) chains (for all A), and all (Q -> Z) chains (for all Z), then append/prepend the (P -> Q) converter to those chains and register those as well in the (A -> Q) and (P -> Z) lists, respectively.
So maybe it's worth having two sorts of converters: the very general ones, and the ones that have closed sets of input and output types. This would be useful to avoid implementing a large number of converters across a range of types. Think about implementing converters for all six numeric primitive type wrappers to all other numeric type wrappers, and then the same for all RealType equivalents, and then again between each numeric type wrapper and RealType wrapper! Instead, we almost certainly will want to support implicit multi-step conversion.
The generics of converters are also tricky. Converters that operate solely on raw types are not expressive enough (we know this from experience with the current SJC convert framework). But for a "closed converter" (see above) with generic conversion (P -> Q) where P and Q are generic types, it becomes more challenging to efficiently query the system for all (A -> P') and (Q' -> Z) chains, because P != P' and Q != Q'; rather, we want only that P' is assignable to P and Q is assignable to Q. So we may be stuck looping through all converters in the system every time a new converter is added in order to maintain this index of conversion chains. 🤮
So maybe we want to impose a further restriction of only reified types for chainable converters? With this restriction, we could still chain convert e.g. Byte ➡️ Double ➡️ DoubleType. But sadly that would prevent chained conversion of e.g. RandomAccessibleInterval<FloatType> ➡️ RandomAccessibleInterval<DoubleType> ➡️ IterableInterval<DoubleType>.
All of this complexity gives me pause. In the previous incarnation of ImageJ Ops, the conversion capability leads to an open-ended set of possible legal op executions (type-wise). It is not possible to "list all method signatures of ops"—certainly not without an exponential explosion of signatures, even assuming all converters are closed. Even for one single op in the system add8(Double, Double, Double, Double, Double, Double, Double, Double), if there are converters capable of going from all five other numeric types to Double, we now have 6^8 > 1 million method signatures for this one op! 🤯
With the op transformer layer, we probably already have the same problem, so I guess this is not something to worry about.
The only way I know to move forward on a new conversion framework is to start trying to code one here in scijava-ops. I expect it will take quite some work to get right.
I've done some thinking about how this would be implemented, and am using this to record my thoughts. The steps would be as follows:
For every OpInfo that does not directly match the OpRef, record a list of the incompatible parameter indices. This information will need to be recorded (maybe we need an ImperfectOpCandidate?), and will require a good deal of change within DefaultOpMatcher and MatchingUtils. It would be nice if we can avoid changing the signatures of OpMatcher, but this may be unavoidable. This returned list should be sorted (probably by increasing number of incompatible indices).
If and only if no OpInfo matches the OpRef (either directly or through adaptation), we should turn to this list of ImperfectOpCandidates. I am going to argue that adaptation should be prioritized over conversion as adaptation does not modify any of the user's arguments, better preserving the intended behavior.
For each ImperfectOpCandidate, determine whether Converters exist for each incompatible parameter index.
If some ImperfectOpCandidate can be converted, use Javassist to create some class to do the conversion.
My attitude towards adaptation is that it should be thought of as a convenience layer instead of a core layer. Adaptation should be thought of as a way of saving a failing request, but should not be considered responsible for it (because we cannot possibly cover every adaptation the user might want). I will argue that conversion should be thought of in the same way, for the same reasons. It is the responsibility of the user to ensure that their request can be matched, not the responsibility of the matcher to massage the request into something that can be matched. To assume otherwise would be a futile task, since we cannot fathom the intentions of the user.
Activity
gselzer commentedon Aug 27, 2019
I have been thinking about this issue recently, especially with respect to primitives. It would be quite nice if the matcher had the ability to convert
Integers toDoubles orLongs on the fly (these are my two bothersome use cases. I am wondering, however, where and/or when we would want to be doing the converting. It would be most convenient if we could place this logic withinisAssignable: this would be by far the best place to do so if you are looking for the conversion while usingops.run. In comparison with another method like implementing transformations, it would be faster, more flexible, and require less transformation boilerplate, however we might (probably would) run intoClassCastExceptions. So we might have to go with the transformations.ctrueden commentedon Aug 30, 2019
@gselzer and I talked through two possible approaches for implementing converters:
We concluded that (1) is not tenable: the targets may not be finite/closed. Suppose you have a
RAI<FloatType>and want to pass it to an op that requires aII<DoubleType>. As a human being, we can see that this is a straightforward two-step conversion: useConverters.convertto go fromRAI<FloatType>toRAI<DoubleType>and then callViews.flatIterableto go fromRAItoII. But from the perspective of a conversion transformer: given aRAI<FloatType>: what is the list of destination generic types you support (see theCollection<OpRef> getRefsTransformingTo(OpRef targetRef)method ofOpTransformer)? It may not be a finite answer; you can only state, for a given destination generic type, whether you could do it. But there could be arbitrarily many supported destination types.We lean toward (2) now because it would be feasible for any given converter plugin to answer the question "can you convert type T to type U?", regardless of the breadth/constraints on that converter plugin.
That said, for many (all?) converter plugins, they can be expressed as
java.util.function.Functions? I.e. the sorts of types upon which they operate are expressible via the Java generics grammar? Seems true, but we didn't think deeply about all the possible sorts of converters we want to support. There is also an issue of not knowing if the conversion will be lossy without knowing the specific source and destination types for a particular conversion. Think e.g.RAI<T>toRAI<U>forT extends RealType<T>andU extends RealType<U>—it might be lossy depending on the exactTandU.If we have a closed set of source and destination types supported by each converter plugin, we could maintain a data structure of all (A -> B) conversion chains. When a new converter for (P -> Q) is added to system, assuming we can efficiently query the existing conversion chains, we can look up all (A -> P) chains (for all A), and all (Q -> Z) chains (for all Z), then append/prepend the (P -> Q) converter to those chains and register those as well in the (A -> Q) and (P -> Z) lists, respectively.
So maybe it's worth having two sorts of converters: the very general ones, and the ones that have closed sets of input and output types. This would be useful to avoid implementing a large number of converters across a range of types. Think about implementing converters for all six numeric primitive type wrappers to all other numeric type wrappers, and then the same for all
RealTypeequivalents, and then again between each numeric type wrapper andRealTypewrapper! Instead, we almost certainly will want to support implicit multi-step conversion.The generics of converters are also tricky. Converters that operate solely on raw types are not expressive enough (we know this from experience with the current SJC convert framework). But for a "closed converter" (see above) with generic conversion (P -> Q) where P and Q are generic types, it becomes more challenging to efficiently query the system for all (A -> P') and (Q' -> Z) chains, because P != P' and Q != Q'; rather, we want only that P' is assignable to P and Q is assignable to Q. So we may be stuck looping through all converters in the system every time a new converter is added in order to maintain this index of conversion chains. 🤮
So maybe we want to impose a further restriction of only reified types for chainable converters? With this restriction, we could still chain convert e.g.
Byte➡️Double➡️DoubleType. But sadly that would prevent chained conversion of e.g.RandomAccessibleInterval<FloatType>➡️RandomAccessibleInterval<DoubleType>➡️IterableInterval<DoubleType>.All of this complexity gives me pause. In the previous incarnation of ImageJ Ops, the conversion capability leads to an open-ended set of possible legal op executions (type-wise). It is not possible to "list all method signatures of ops"—certainly not without an exponential explosion of signatures, even assuming all converters are closed. Even for one single op in the system
add8(Double, Double, Double, Double, Double, Double, Double, Double), if there are converters capable of going from all five other numeric types toDouble, we now have 6^8 > 1 million method signatures for this one op! 🤯With the op transformer layer, we probably already have the same problem, so I guess this is not something to worry about.
The only way I know to move forward on a new conversion framework is to start trying to code one here in scijava-ops. I expect it will take quite some work to get right.
gselzer commentedon Sep 28, 2020
I've done some thinking about how this would be implemented, and am using this to record my thoughts. The steps would be as follows:
For every
OpInfothat does not directly match theOpRef, record a list of the incompatible parameter indices. This information will need to be recorded (maybe we need anImperfectOpCandidate?), and will require a good deal of change withinDefaultOpMatcherandMatchingUtils. It would be nice if we can avoid changing the signatures ofOpMatcher, but this may be unavoidable. This returned list should be sorted (probably by increasing number of incompatible indices).If and only if no
OpInfomatches theOpRef(either directly or through adaptation), we should turn to this list ofImperfectOpCandidates. I am going to argue that adaptation should be prioritized over conversion as adaptation does not modify any of the user's arguments, better preserving the intended behavior.For each
ImperfectOpCandidate, determine whetherConvertersexist for each incompatible parameter index.If some
ImperfectOpCandidatecan be converted, use Javassist to create some class to do the conversion.My attitude towards adaptation is that it should be thought of as a convenience layer instead of a core layer. Adaptation should be thought of as a way of saving a failing request, but should not be considered responsible for it (because we cannot possibly cover every adaptation the user might want). I will argue that conversion should be thought of in the same way, for the same reasons. It is the responsibility of the user to ensure that their request can be matched, not the responsibility of the matcher to massage the request into something that can be matched. To assume otherwise would be a futile task, since we cannot fathom the intentions of the user.
gselzer commentedon May 6, 2021
@ctrueden this issue is solved by scijava/incubator#25, yeah?
gselzer commentedon Jun 16, 2023
Type conversion works via the simplification paradigm, so I'm considering this completed.