Skip to content
This repository was archived by the owner on Oct 26, 2026. It is now read-only.

Op Discovery Through YAML declaration - #67

Merged
gselzer merged 9 commits into
mainfrom
scijava/scijava-ops-engine/yaml-discovery
Jan 13, 2023
Merged

gselzer merged 9 commits into
mainfrom
scijava/scijava-ops-engine/yaml-discovery

Conversation

@gselzer

@gselzer gselzer commented Dec 22, 2022 •

Copy link
Copy Markdown
Member

This PR introduces a new YAMLDiscoverer that provides the ability to describe Ops via YAML declaration. Currently, Classes implementing a functional interface, final Fields and static Methods can be wrapped into Ops via YAML. This is done by writing a file ops.yaml within the resources package of a project. At runtime, the ClassLoader obtains all of these files on the class/module path, adding them to Ops (assuming that the functionality is exposed through JPMS (?)). An example is included below:

info:
  source: Java
  package: SciJava Ops YAML
  version: 0-SNAPSHOT

ops:
  - op:
      name: example.add
      priority: 0.0
      source: org.scijava.ops.engine.yaml.ops.YAMLClassOp
      parameters:
        - input: in1
          description: The first input
        - input: in2
          description: The second input
        - output: out
          description: The sum

  - op:
      name: example.sub
      priority: 0.0
      source: org.scijava.ops.engine.yaml.ops.YAMLMethodOp.subtract(java.lang.Double, java.lang.Double)
      type: java.util.function.BiFunction
      parameters:
        - input: in1
          description: The first input
        - input: in2
          description: The second input
        - output: out
          description: The difference

  - op:
      name: example.mul
      priority: 0.0
      source: org.scijava.ops.engine.yaml.ops.YAMLFieldOp$multiply
      parameters:
        - input: in1
          description: The first input
        - input: in2
          description: The second input
        - output: out
          description: The product

The info section declares metadata pertaining to all Ops in the package:

  • The source tells us the language the functionality is written in, and allows us to later target e.g. Python
  • The package tells us the package the Ops are described in. I'm not sure we need this, actually.
  • The version is used for provenance, telling us the code version.

The ops section iterates over all ops. One Op is created for each op in the list.

  • The name is the term(s) that the Op can be searched by
  • The priority is the Op's priority
  • The source is a term that Ops could use to get a handle on the functionality. For Java, this is either
    • A fully qualified class name
    • A fully qualified class name plus a string describing a static method (with parameters, to protect against overloading)
    • A fully qualified class name plus a field name
  • The parameters section provides information on the parameters. We aren't using this right now, and might want to delete it.

Discussion points:

  • Is it worth keeping the info -> package tag, and the parameters tags? This PR doesn't use either, but I can see us wanting to use it later (e.g. for Python functionality).
  • Should this functionality be moved into a separate module? How much of it should be moved?

@gselzer gselzer added the enhancement New feature or request label Dec 22, 2022
@gselzer
gselzer requested review from ctrueden and hinerm December 22, 2022 16:59
@gselzer gselzer self-assigned this Dec 22, 2022
@gselzer
gselzer force-pushed the scijava/scijava-ops-engine/yaml-discovery branch from b6e5745 to dd41241 Compare December 22, 2022 17:08

@hinerm hinerm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth keeping the info -> package tag, and the parameters tags?

These feel worthwhile to me. I would keep them.

Should this functionality be moved into a separate module? How much of it should be moved?

How much of it can be moved? Based on my understanding from our previous discussion, I'm in favor of consolidating code in scijava-ops-engine instead of submodules, since those modules aren't true "extensions" and require fundamental changes to the core module.

@gselzer
gselzer force-pushed the scijava/scijava-ops-engine/yaml-discovery branch 6 times, most recently from fbf253d to aec41f3 Compare December 27, 2022 17:48
@gselzer gselzer mentioned this pull request Dec 28, 2022
@hinerm

hinerm commented Jan 4, 2023

Copy link
Copy Markdown
Member

@gselzer I added two commits to tweak the inner-class vs. method discovery. I appreciate trying to load the class in the JavaClassYAMLOpInfoCreator, but that created a situation where we're relying on the Class info creator to be discovered before the Method info creator. This felt fragile, so I added the class load check to the Method creator as well, and avoid loading the class if there is no ambiguity.

If you agree with those commits: could you just rewrite your WIPs? Then we'll merge!

@ctrueden

ctrueden commented Jan 5, 2023

Copy link
Copy Markdown
Member

How much of it can be moved? Based on my understanding from our previous discussion, I'm in favor of consolidating code in scijava-ops-engine instead of submodules, since those modules aren't true "extensions" and require fundamental changes to the core module.

Is the code in question the YAML handling code? If so, I propose that it might not be specific to scijava-ops-engine, but more about the metadata. Imagine you are in Python and would like to invoke some convenience function (invoked via scyjava) to reason about ops, but without the full heavyweight matcher. I don't have a strong opinion—such YAML handling code could maybe be written directly in Python by leaning on the yaml package, although I think there would be some code duplication—but it's worth a discussion I think.

@hinerm
hinerm force-pushed the scijava/scijava-ops-engine/yaml-discovery branch from dd4f33b to 01c02e1 Compare January 6, 2023 13:57
@gselzer
gselzer force-pushed the scijava/scijava-ops-engine/yaml-discovery branch from 01c02e1 to 20226f1 Compare January 6, 2023 15:37
@gselzer

gselzer commented Jan 6, 2023

Copy link
Copy Markdown
Member Author

Is the code in question the YAML handling code? If so, I propose that it might not be specific to scijava-ops-engine, but more about the metadata.

There is some general YAML logic, but I think right now we should leave it in the engine. It's pretty intertwined with the Ops-specific logic, and I don't yet know that we have a good use case to separate it. I say YAGNI

@gselzer
gselzer force-pushed the scijava/scijava-ops-engine/yaml-discovery branch from 20226f1 to 9ac4bc3 Compare January 6, 2023 16:23
@gselzer
gselzer force-pushed the scijava/scijava-ops-engine/yaml-discovery branch from 9ac4bc3 to 62b4c02 Compare January 6, 2023 16:32
@gselzer gselzer linked an issue Jan 6, 2023 that may be closed by this pull request
@gselzer
gselzer marked this pull request as ready for review January 13, 2023 15:21
@gselzer
gselzer merged commit 413d1be into main Jan 13, 2023
@gselzer
gselzer deleted the scijava/scijava-ops-engine/yaml-discovery branch January 13, 2023 15:23
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SciJava Ops: create external configuration mechanism

3 participants