Skip to content

Commit 92a93bf

Browse files
ctruedenclaude
andcommitted
scijava-io3: port ByteBufferByteBank, keeping direct banks off-heap
I had dropped this as adding no capability over ByteArrayByteBank, since both cap at Integer.MAX_VALUE. That was wrong: a direct ByteBuffer is off-heap, so its bytes can be handed to native code and to shared memory without a copy. That matters increasingly for Fiji as Appose's named shared memory blocks become more prominent. Ported with two additions that make the capability usable and observable: direct() factories, and an isDirect() accessor. It also fixes a bug that defeated the entire purpose. Growing the bank reallocated with ByteBuffer.allocate, so a direct bank quietly became a heap bank as soon as it outgrew its initial capacity -- and since the default capacity is 10,000 bytes, any real payload would trip it. Growth now goes through the bank's own provider, and a test asserts a direct bank is still direct after growing well past its initial capacity. NIOService stays dropped: it wraps two JDK calls behind a system-property switch, was inherited from old Bio-Formats code, and SCIFIO only exposes it as a gateway accessor without using it. Nothing in the ported architecture precludes FileChannel-based access later -- a handle may open a channel, and a byte bank may be backed by a mapped buffer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013gJpzGd3ksbME4CN6Tu2ni
1 parent 55a1f1e commit 92a93bf

4 files changed

Lines changed: 327 additions & 5 deletions

File tree

‎docs/migration-plan.md‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -415,10 +415,13 @@ No application context required by anything in this phase.
415415
Cancellation is thread interruption instead — see the cancellation
416416
section — so `copy` checks the flag once per block and throws
417417
`InterruptedIOException`.
418-
- `io.nio` is **not ported**: `NIOService` wraps two JDK calls and has no
419-
usages, and `ByteBufferByteBank` has none either and caps at
420-
`Integer.MAX_VALUE` just as `ByteArrayByteBank` does, so it adds no
421-
capability. See [migration.md](migration.md). This component is therefore
418+
- From `io.nio`, `ByteBufferByteBank` **is** ported: a direct buffer is
419+
off-heap, so its bytes reach native code and shared memory without a
420+
copy — which matters increasingly for Appose's named shared memory
421+
blocks. `NIOService` is dropped, being two JDK calls behind a
422+
system-property switch; nothing in the ported architecture precludes
423+
reintroducing `FileChannel` later, since a handle may open a channel and
424+
a bank may be backed by a mapped buffer. This component is therefore
422425
complete.
423426
- **`scijava-events`** (new, `org.scijava.events`): a clean-room event bus,
424427
no context, designed for typed topics and weak-reference subscribers. Not a

‎docs/migration.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,8 @@ A **dropped** row is a successful outcome, not a gap.
100100
| `io.location` | `org.scijava.io3.location`. `LocationService`/`DefaultLocationService` become `Locations`, backed by plain `ServiceLoader`; `LocationResolver` no longer extends `HandlerPlugin` and carries its own `priority()`. `URILocation`'s injected `LogService` was dead code and is gone | ported |
101101
| `ByteBank`, `ByteArrayByteBank` | `org.scijava.io3` | ported |
102102
| `io.handle` | `org.scijava.io3.handle`. `DataHandle` no longer extends `WrapperPlugin`: the contract was `getType()` plus `set`/`get`, now inlined as `locationType()`, `supports`, `set`, `get` and `priority()`. `DataHandleService` becomes `DataHandles`, backed by plain `ServiceLoader`. `DataHandles.copy` reports progress to a `LongConsumer` rather than a `Task`, and so no longer supports cancellation | ported |
103-
| `io.nio` | **not ported.** `NIOService`/`DefaultNIOService` wrap two JDK calls (`ByteBuffer.allocate` + `FileChannel.read`, or `FileChannel.map`) behind a system-property switch, and have no usages in the scijava or imagej repositories. `ByteBufferByteBank` also has none, and caps at `Integer.MAX_VALUE` like `ByteArrayByteBank`, so it adds no capability — only a pluggable allocator, e.g. `ByteBuffer::allocateDirect`. `ByteBank` is ported, so a direct-buffer bank is a small class to add when something needs one. NB: SCIFIO was not checked, having no local checkout | dropped |
103+
| `io.nio.ByteBufferByteBank` | `org.scijava.io3.ByteBufferByteBank`, with `direct()` factories and an `isDirect()` accessor. Off-heap direct buffers are its reason to exist alongside `ByteArrayByteBank`: bytes reach native code and shared memory without a copy, which matters increasingly for Appose's named shared memory blocks. **Bug fixed in the port:** growing a direct bank reallocated with `ByteBuffer.allocate`, quietly moving the bytes back onto the heap; it now grows through the bank's own provider | ported |
104+
| `io.nio.NIOService` | **not ported** — it wraps two JDK calls behind a system-property switch, was inherited from old Bio-Formats code, and SCIFIO only exposes it as a gateway accessor (`SCIFIO.nio()`) without using it. Nothing in the ported architecture precludes `FileChannel`-based handles later: a handle is free to open a channel, and a byte bank is free to be backed by a mapped buffer | dropped |
104105
| `io` (`IOPlugin`, `IOService`, `RecentFileService`) | `scijava-io3` (Phase 1), pending review of what is still used | open |
105106
| `event`, `event.bushe` | `org.scijava.events` — clean-room rewrite, not a port (Phase 1) | planned |
106107
| `parse` | Parsington (`org.scijava.parsington`) | planned |
Lines changed: 216 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,216 @@
1+
/*
2+
* #%L
3+
* Locations and data handles: a uniform way to address and read bytes.
4+
* %%
5+
* Copyright (C) 2026 SciJava developers.
6+
* %%
7+
* Redistribution and use in source and binary forms, with or without
8+
* modification, are permitted provided that the following conditions are met:
9+
*
10+
* 1. Redistributions of source code must retain the above copyright notice,
11+
* this list of conditions and the following disclaimer.
12+
* 2. Redistributions in binary form must reproduce the above copyright notice,
13+
* this list of conditions and the following disclaimer in the documentation
14+
* and/or other materials provided with the distribution.
15+
*
16+
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
17+
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
18+
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
19+
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE
20+
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
21+
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
22+
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
23+
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
24+
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
25+
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
26+
* POSSIBILITY OF SUCH DAMAGE.
27+
* #L%
28+
*/
29+
30+
package org.scijava.io3;
31+
32+
import java.nio.ByteBuffer;
33+
import java.util.function.Function;
34+
35+
36+
/**
37+
* A {@link ByteBank} backed by a {@link ByteBuffer}. Self-growing up to a
38+
* maximal capacity of {@link Integer#MAX_VALUE}
39+
*
40+
* @author Gabriel Einsdorf
41+
*/
42+
public class ByteBufferByteBank implements ByteBank {
43+
44+
private static final int DEFAULT_CAPACITY = 10_000;
45+
46+
/**
47+
* Creates a bank backed by an <em>off-heap</em> direct buffer.
48+
* <p>
49+
* Direct buffers are what make this bank worth having alongside
50+
* {@link ByteArrayByteBank}: bytes live outside the Java heap, so they can
51+
* be handed to native code and to shared memory without a copy. Appose's
52+
* named shared memory blocks are the motivating case.
53+
* </p>
54+
*/
55+
public static ByteBufferByteBank direct() {
56+
return new ByteBufferByteBank(ByteBuffer::allocateDirect);
57+
}
58+
59+
/**
60+
* Creates a bank backed by an <em>off-heap</em> direct buffer of the given
61+
* initial capacity.
62+
*
63+
* @see #direct()
64+
*/
65+
public static ByteBufferByteBank direct(final int initialCapacity) {
66+
return new ByteBufferByteBank(ByteBuffer::allocateDirect, initialCapacity);
67+
}
68+
69+
private ByteBuffer buffer;
70+
71+
private int size;
72+
73+
private Function<Integer, ByteBuffer> provider;
74+
75+
public ByteBufferByteBank() {
76+
provider = ByteBuffer::allocate;
77+
buffer = provider.apply(DEFAULT_CAPACITY);
78+
}
79+
80+
public ByteBufferByteBank(final Function<Integer, ByteBuffer> provider) {
81+
this.provider = provider;
82+
buffer = provider.apply(DEFAULT_CAPACITY);
83+
}
84+
85+
public ByteBufferByteBank(final Function<Integer, ByteBuffer> provider,
86+
final int initialCapacity)
87+
{
88+
this.provider = provider;
89+
buffer = provider.apply(initialCapacity);
90+
}
91+
92+
public ByteBufferByteBank(final int initialCapacity) {
93+
provider = ByteBuffer::allocate;
94+
buffer = provider.apply(initialCapacity);
95+
}
96+
97+
/**
98+
* Gets whether this bank's bytes live off the Java heap, and so can reach
99+
* native code and shared memory without a copy.
100+
*
101+
* @see #direct()
102+
*/
103+
public boolean isDirect() {
104+
return buffer.isDirect();
105+
}
106+
107+
@Override
108+
public long getMaxBufferSize() {
109+
return Integer.MAX_VALUE;
110+
}
111+
112+
@Override
113+
public void setBytes(final long startpos, final byte[] bytes,
114+
final int offset, final int length)
115+
{
116+
// ensure we have space
117+
checkWritePos(startpos, startpos + length);
118+
final int neededCapacity = size + length;
119+
ensureCapacity(neededCapacity);
120+
121+
// copy the data
122+
buffer.position((int) startpos);
123+
buffer.put(bytes, offset, length);
124+
125+
// update the maxpos
126+
updateSize(startpos + length);
127+
}
128+
129+
@Override
130+
public void setByte(final long pos, final byte b) {
131+
checkWritePos(pos, pos);
132+
if (pos == buffer.capacity()) {
133+
ensureCapacity((int) pos + 1);
134+
}
135+
buffer.put((int) pos, b);
136+
updateSize(pos + 1);
137+
}
138+
139+
@Override
140+
public void clear() {
141+
buffer.clear();
142+
size = 0;
143+
}
144+
145+
@Override
146+
public byte getByte(final long pos) {
147+
checkReadPos(pos, pos);
148+
// the buffer might contain bytes with negative value
149+
// we need to flip the sign to positive to satisfy the contract of this
150+
// method
151+
return buffer.get((int) pos);
152+
}
153+
154+
@Override
155+
public int getBytes(final long startPos, final byte[] b, final int offset,
156+
final int length)
157+
{
158+
checkReadPos(startPos, startPos + length);
159+
// ensure we don't try to read data which is not in the buffer
160+
final int readLength = (int) Math.min(size() - startPos, length);
161+
buffer.position((int) startPos);
162+
buffer.get(b, offset, readLength);
163+
164+
return readLength;
165+
}
166+
167+
@Override
168+
public long size() {
169+
return size;
170+
}
171+
172+
@Override
173+
public boolean isReadOnly() {
174+
// NB: Some ByteBuffers are read-only. But there is no API to check it.
175+
// Therefore, we make a "best effort" guess based on known read-only types.
176+
// Since these read-only types are not public, we compare class names rather
177+
// than checking for type equality or using instanceof.
178+
final String className = buffer.getClass().getName();
179+
return className.equals("java.nio.HeapByteBufferR") ||
180+
className.equals("java.nio.DirectByteBufferR");
181+
}
182+
183+
// -- Helper methods --
184+
185+
private void ensureCapacity(final int minCapacity) {
186+
final int oldCapacity = buffer.capacity();
187+
if (minCapacity <= oldCapacity) return; // no need to grow
188+
189+
// grow the array by up to 50% (plus a small constant)
190+
final int growth = Math.min(oldCapacity / 2 + 16, Integer.MAX_VALUE);
191+
final int newCapacity;
192+
if (growth > Integer.MAX_VALUE - oldCapacity) {
193+
// growth would push array over the maximum array size
194+
newCapacity = Integer.MAX_VALUE;
195+
}
196+
else newCapacity = oldCapacity + growth;
197+
// ensure the array grows by at least the requested minimum capacity
198+
final int newLength = Math.max(minCapacity, newCapacity);
199+
200+
// copy the data into a new buffer
201+
buffer.position(0);
202+
// NB: allocate through the provider, not ByteBuffer.allocate. Growing a
203+
// direct bank must yield another direct buffer: SciJava Common's version
204+
// quietly moved the bytes back onto the heap here, losing the off-heap
205+
// property that is the whole reason to choose this bank.
206+
final ByteBuffer newBuffer = provider.apply(newLength);
207+
newBuffer.order(buffer.order());
208+
newBuffer.put(buffer);
209+
buffer = newBuffer;
210+
}
211+
212+
private void updateSize(final long newSize) {
213+
size = (int) (newSize > size ? newSize : size);
214+
}
215+
216+
}
Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
/*
2+
* #%L
3+
* Locations and data handles: a uniform way to address and read bytes.
4+
* %%
5+
* Copyright (C) 2026 SciJava developers.
6+
* %%
7+
* Redistribution and use in source and binary forms, with or without
8+
* modification, are permitted provided that the following conditions are met:
9+
*
10+
* 1. Redistributions of source code must retain the above copyright notice,
11+
* this list of conditions and the following disclaimer.
12+
* 2. Redistributions in binary form must reproduce the above copyright notice,
13+
* this list of conditions and the following disclaimer in the documentation
14+
* and/or other materials provided with the distribution.
15+
*
16+
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
17+
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
18+
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
19+
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE
20+
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
21+
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
22+
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
23+
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
24+
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
25+
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
26+
* POSSIBILITY OF SUCH DAMAGE.
27+
* #L%
28+
*/
29+
30+
package org.scijava.io3;
31+
32+
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
33+
import static org.junit.jupiter.api.Assertions.assertEquals;
34+
import static org.junit.jupiter.api.Assertions.assertTrue;
35+
36+
import java.nio.ByteBuffer;
37+
38+
import org.junit.jupiter.api.Test;
39+
40+
/**
41+
* Tests {@link ByteBufferByteBank}, in particular that an off-heap bank stays
42+
* off-heap - the reason to choose it over {@link ByteArrayByteBank}, since
43+
* direct bytes can reach native code and shared memory without a copy.
44+
*
45+
* @author Curtis Rueden
46+
*/
47+
public class ByteBufferByteBankTest {
48+
49+
@Test
50+
public void testReadWrite() {
51+
final ByteBufferByteBank bank = new ByteBufferByteBank();
52+
final byte[] data = { 1, 2, 3, 4, 5 };
53+
bank.setBytes(0, data, 0, data.length);
54+
assertEquals(data.length, bank.size());
55+
final byte[] read = new byte[data.length];
56+
bank.getBytes(0, read);
57+
assertArrayEquals(data, read);
58+
}
59+
60+
@Test
61+
public void testDirectIsOffHeap() {
62+
assertTrue(ByteBufferByteBank.direct().isDirect());
63+
assertTrue(ByteBufferByteBank.direct(64).isDirect());
64+
}
65+
66+
/**
67+
* Growing a direct bank must keep it direct. SciJava Common reallocated
68+
* with ByteBuffer.allocate here, quietly moving the bytes onto the heap.
69+
*/
70+
@Test
71+
public void testDirectStaysDirectWhenGrown() {
72+
final ByteBufferByteBank bank = ByteBufferByteBank.direct(8);
73+
final byte[] data = new byte[1024];
74+
for (int i = 0; i < data.length; i++)
75+
data[i] = (byte) i;
76+
77+
// NB: far beyond the initial capacity, so the bank must grow.
78+
bank.setBytes(0, data, 0, data.length);
79+
assertEquals(data.length, bank.size());
80+
81+
final byte[] read = new byte[data.length];
82+
bank.getBytes(0, read);
83+
assertArrayEquals(data, read);
84+
assertTrue(bank.isDirect(), "growing moved the bank back onto the heap");
85+
}
86+
87+
@Test
88+
public void testHeapBankIsNotDirect() {
89+
final ByteBufferByteBank bank = new ByteBufferByteBank(8);
90+
bank.setBytes(0, new byte[1024], 0, 1024);
91+
assertTrue(!bank.isDirect());
92+
}
93+
94+
@Test
95+
public void testCustomProvider() {
96+
final ByteBufferByteBank bank = new ByteBufferByteBank(
97+
ByteBuffer::allocateDirect, 16);
98+
bank.setBytes(0, new byte[] { 9 }, 0, 1);
99+
assertEquals(9, bank.getByte(0));
100+
assertTrue(bank.isDirect());
101+
}
102+
}

0 commit comments

Comments
 (0)