You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/ops/doc/Benchmarks.rst
+33-32Lines changed: 33 additions & 32 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,23 +1,22 @@
1
1
SciJava Ops Benchmarks
2
2
======================
3
3
4
-
This page describes a quantitative analysis of the SciJava Ops framework, and is heavily inspired by a similar comparison of `ImgLib2 <https://imagej.net/libs/imglib2/benchmarks>`_.
4
+
This page describes a quantitative analysis of the SciJava Ops framework, and is heavily inspired by a similar comparison of `ImgLib2 <https://imagej.net/libs/imglib2/benchmarks>`_. In all figures, benchmark times are displayed using bar charts (describing mean execution times, in microseconds) with error bars (used to denote the range of observed execution times).
5
5
6
6
Hardware and Software
7
7
---------------------
8
8
9
9
This analysis was performed with the following hardware:
10
10
11
-
* Dell Precision 7770
12
-
* 12th Gen Intel i9-12950HX (24) @ 4.900GHz
13
-
* 32 GB 4800 MT/s DDR5 RAM
11
+
* 2021 Dell OptiPlex 5090 Small Form Factor
12
+
* Intel(R) Core(TM) i7-10700 CPU @ 2.90GHz
13
+
* 64 GB 3200 MHz DDR4 RAM
14
14
15
15
The following software components were used:
16
16
17
-
* Ubuntu 23.10
18
-
* Kernel 6.5.0-26-generic
19
-
* OpenJDK 64-Bit Server VM GraalVM CE 21.0.2+13.1 (build 21.0.2+13-jvmci-23.1-b30, mixed mode, sharing)
All benchmarks are executed using the `Java Microbenchmark Harness <https://github.com/openjdk/jmh>`_, using the following parameters:
@@ -31,7 +30,7 @@ All benchmarks are executed using the `Java Microbenchmark Harness <https://gith
31
30
Op Matching
32
31
-----------
33
32
34
-
We first analyze the performance of executing the following static method:
33
+
We first analyze the performance of executing the following static method, written to contain the *fewest* instructions possible while also avoiding code removal by the Just In Time compiler:
35
34
36
35
.. code-block:: java
37
36
@@ -45,31 +44,33 @@ We first analyze the performance of executing the following static method:
45
44
data[0]++;
46
45
}
47
46
48
-
We first benchmark the base penalty of executing this method using SciJava Ops, compared to direct execution of the static method. This method mutates a data structure in place, meaning the Ops engine can match it directly as an inplace Op, or adapt it to a function Op. Thus, we test the benchmark the following three scenarios:
47
+
We first benchmark the overhead of executing this method through SciJava Ops, compared to direct execution of the static method. This method mutates a data structure in place, meaning the Ops engine can match it directly as an inplace Op, or **adapt** it to a function Op. Thus, we test the benchmark the following four scenarios:
The results are shown in **Figure 1**. We find Op execution through the SciJava Ops framework adds a few milliseconds of additional overhead. A few additional milliseconds of overhead are observed when SciJava Ops is additionally tasked with creating an output buffer.
54
+
The results are shown in **Figure 1**. We find Op execution through the SciJava Ops framework adds approximately 100 microseconds of additional overhead. An additional millisecond of overhead is observed when SciJava Ops additionally creates an output buffer.
55
55
56
56
.. chart:: ../images/BenchmarkMatching.json
57
57
58
58
**Figure 1:** Algorithm execution performance (lower is better)
59
59
60
-
Note that the above requests are benchmarked without assistance from the Op cache, i.e. they are designed to model the full matching process. As repeated Op requests will utilize the Op cache, we benchmark cached Op retrieval separately, with results shown in **Figure 2**. These benchmarks suggest Op caching helps avoid the additional overhead of Op adaptation as its performance approaches that of normal Op execution.
60
+
Note that the above requests are benchmarked without assistance from the Op cache, measuring the overhead of full matching process. As repeated Op requests will utilize the Op cache, we benchmark cached Op execution separately, with results shown in **Figure 2**. From these results, we conlude that Op matching comprises the majority of SciJava Ops overhead, and repeated executions add only a few microseconds of overhead.
61
61
62
62
.. chart:: ../images/BenchmarkCaching.json
63
63
64
64
**Figure 2:** Algorithm execution performance with Op caching (lower is better)
65
65
66
66
Finally, we benchmark the overhead of SciJava Ops parameter conversion. Suppose we instead wish to operate upon a ``double[]`` - we must convert it to ``byte[]`` to call our Op. We consider the following procedures:
The results are shown in **Figure 3**; note the Op cache is **not** enabled. We observe overheads on the order of 10 milliseconds to perform Op conversion with and without Op adaptation.
73
+
The results are shown in **Figure 3**; note the Op cache is **not** enabled. We find that parameter conversion imposes additional overhead of approximately 1 millisecond, and when both parameter conversion and Op adaptation are used the overhead (~2 milliseconds) is *additive*.
73
74
74
75
.. chart:: ../images/BenchmarkConversion.json
75
76
@@ -78,35 +79,35 @@ The results are shown in **Figure 3**; note the Op cache is **not** enabled. We
78
79
Framework Comparison
79
80
--------------------
80
81
81
-
To validate our development efforts atop the original `ImageJ Ops <https://imagej.net/libs/imagej-ops/>`_ framework, we benchmark executions of the following method:
82
+
To validate our development efforts atop the original `ImageJ Ops <https://imagej.net/libs/imagej-ops/>`_ framework, we additionally wrap the above static method within ImageJ Ops:
82
83
83
84
.. code-block:: java
84
85
85
-
/**
86
-
* @param data the data to invert
87
-
* @implNote op name="benchmark.invert",type=Inplace1
We then benchmark the performance of executing this code using the following pathways:
98
+
We then benchmark the performance of executing the static method using the following pathways:
98
99
99
100
* Static method invocation
100
101
* SciJava Ops invocation
101
-
* ImageJ Ops invocation (using a ``Class`` wrapper to make the method discoverable within ImageJ Ops)
102
+
* ImageJ Ops invocation (using the above wrapper)
102
103
103
-
The results are shown in **Figure 4**. When algorithm matching dominates execution time, the SciJava Ops matching framework provides significant improvement in matching performance in comparison with the original ImageJ Ops framework.
104
+
The results are shown in **Figure 4**. From this figure we can see that the "Op overhead" from ImageJ Ops is approximately 70x the "Op overhead" from SciJava Ops.
104
105
105
106
.. chart:: ../images/BenchmarkFrameworks.json
106
107
107
108
**Figure 4:** Algorithm execution performance by Framework (lower is better)
108
109
109
-
Finally, here is a figure combining all the metrics above:
110
+
We provide a final figure combining all the metrics above:
0 commit comments