Skip to content
This repository was archived by the owner on Oct 26, 2026. It is now read-only.

Progress Reporting Implementation - #43

Merged
ctrueden merged 14 commits into
mainfrom
scijava/scijava-ops-engine/progress-reporting
Apr 13, 2022
Merged

ctrueden merged 14 commits into
mainfrom
scijava/scijava-ops-engine/progress-reporting

Conversation

@gselzer

@gselzer gselzer commented Sep 30, 2021

Copy link
Copy Markdown
Member

This PR introduces a mechanism for Progress Reporting within the Ops framework. The core of this mechanism is the static Progress class, which is used both for Ops to broadcast updates to their progress and for users to be notified of updates to Ops' progress. The Progress class maintains a set of Tasks, describing all concurrently-executing processing hierarchies. Below details the (current) methods by which each party interfaces with the Progress class:

1: Ops Implementing Progress Reporting

Ops obtained via the matcher are automagically registered as Tasks, recorded within Progress. Therefore, in order to report numerical progress, Ops must:

  1. Define the number of stages it will run, and the number of subtasks (using the command Progress.defineTotalProgress(int numStages, int numSubTasks)):
    a. A stage is defined as a section of Op processing done within the Op itself, and are customizable within the Op
    b. A subtask is defined as a standalone task that will report its progress to this Op. Any Op dependency execution is automatically registered as a subtask.
  2. For each stage, the Op must provide the number of updates it plans to record. This can be done calling Progress.setStageMax(long max)
    After this setup, the Op can report its progress by calling Progress.update().
    Here is an example of how an Op might define progress:
@Plugin(type = Op.class, name = "create.array")
public class ArrayCreator implements BiFunction<Integer, Integer, Integer[]> {

  public Integer[] apply(Integer size, Integer initialValue) {
    Progress.defineTotalProgress(1, 0); // 1 Op stage, 0 Op Subtasks
    Progress.setStageMax(size); // Set the progress maximum to size
    Integer[] arr = new Integer[size];
    for (int i = 0; i < arr.length; i++) {
      arr[i] = initialValue;
      Progress.update(); //Record update to the total progress
    }
  }
}

Ops can also update their status, without any setup, using the method Progress.setStatus(String status):

@Plugin(type = Op.class, name = "create.array")
public class ArrayCreator implements BiFunction<Integer, Integer, Integer[]> {

  public Integer[] apply(Integer size, Integer initialValue) {
    Integer[] arr = new Integer[size];
    for (int i = 0; i < arr.length; i++) {
      arr[i] = initialValue;
      Progress.setStatus("Initialized element " + i); //Update status
    }
  }
}

2: The ProgressListener interface as an interrupt mechanism for progress updates

Progress updates can trigger interrupt code via the ProgressListener interface. This interface is a FunctionalInterface, with a functional method acknowledgeUpdate(Task task) (allowing a ProgressListener to be defined as a lambda). ProgressListeners can be registered using the method Progress.addListener(Object progressible, ProgressListener p); an example of a ProgressListener interrupt is shown below:

BiFunction<Integer, Integer, Integer> op = ops.op("create.array").inTypes(Integer.class, Integer.class).outType(Integer.class).function();
Progress.addListener(op, (task) -> {
  System.out.println("Progress: " + task.progress());
});
op.apply(400, 1);

During the op call, the lambda-defined ProgressListener will function as a callback Object, with its acknowledgeUpdate(Task task) functional method being called every time the Op calls Progress.update(). The method Task.status() can also be called if the listener is interested in the status of the Op.

@gselzer
gselzer force-pushed the scijava/scijava-ops-engine/progress-reporting branch 4 times, most recently from f2e15d4 to e5db14b Compare October 4, 2021 18:35
@gselzer

gselzer commented Oct 4, 2021

Copy link
Copy Markdown
Member Author

Update: Progress and related code have been relocated into a new module, scijava-progress

@gselzer
gselzer force-pushed the scijava/scijava-ops-engine/progress-reporting branch from e5db14b to 3c769ed Compare October 4, 2021 19:05
@gselzer

gselzer commented Nov 11, 2021

Copy link
Copy Markdown
Member Author

Closes scijava/scijava#42

@gselzer
gselzer force-pushed the scijava/scijava-ops-engine/progress-reporting branch from 3c769ed to a13f9b2 Compare November 16, 2021 19:18
@ctrueden ctrueden self-assigned this Nov 22, 2021
@ctrueden
ctrueden force-pushed the scijava/scijava-ops-engine/progress-reporting branch 2 times, most recently from ba6315b to 10395e7 Compare December 9, 2021 23:30
gselzer added 14 commits April 13, 2022 11:46
Signed-off-by: Curtis Rueden <ctrueden@wisc.edu>
This allows a progress() check to return 0 instead of NaN
This allows the dependencies to update the progress of their parent Task

This commit also fixes some expected answers within ProvenanceTest.
Previously, these tests asserted that only the outside (mapper) function
altered the output. But, in this case, the output was directly created
by the mapped function. Thus there were really two Ops that directly
altered the output. Now, the test reflects that.
Also, we no longer need to run the ops on separate threads since we are
interrupting instead of polling
This commit also changes the name of the ProgressListener's functional
method to acknowledgeUpdate, instead of updateProgress, as an update
does not necessarily imply that the PROGRESS is being updated.
@ctrueden
ctrueden force-pushed the scijava/scijava-ops-engine/progress-reporting branch from 10395e7 to 11f50d2 Compare April 13, 2022 16:49

@ctrueden ctrueden left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a lot of problems trying to test and review this code from Eclipse. I cleaned up many of the problems and merged the work to main, then rebased this branch over it, but there will continue to be problems until #57 is merged due to Eclipse's failure surrounding annotation processing. Even with the EclipseHelper. I played with augmenting the Ruby script that merges main and test annotations, to make it also clean up after Eclipse, but couldn't make it work out of the box no matter what I tried. Even with building from the command line via mvn clean install followed by a refresh in Eclipse, the unit tests in Eclipse still are crazy and can't find plugins and a bunch of them fail. So I'm giving up: I'm just going to push through all these PRs as fast as I can until after #57. Then I hope it will become possible to develop this project in Eclipse without so much pain.

if (!metadata.hints().contains(History.SKIP_RECORDING)) {
metadata.history().addExecution(this, output);
}
metadata.history().logOutput(this, output);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you change this to metadata().history()..., this could be a default method of RichOp interface.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh...by default I'd prefer to do nothing in RichOp. Do you strongly about this @ctrueden?

Comment thread scijava/scijava-progress/LICENSE.txt Resolved
@ctrueden
ctrueden merged commit 2581611 into main Apr 13, 2022
@ctrueden
ctrueden deleted the scijava/scijava-ops-engine/progress-reporting branch April 13, 2022 16:57
gselzer added a commit that referenced this pull request Oct 17, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

No open projects

Development

Successfully merging this pull request may close these issues.

2 participants