As mentioned on the Is LMDB A Good Fit? page, LMDB is a key-value store that works with arbitrary arrays of bytes that you define. These arrays of bytes need be presented to LMDB for storage, and eventually retrieved through its various get and cursor operations.

In Java it is commonplace to use a "buffer" class to cleanly abstract such arrays of bytes. Such "buffer" classes provide convenience methods to easily read and write buffer contents (eg numbers and strings). More importantly they prevent reads and writes outside the memory range allocated to the buffer. Buffer classes also assist with housekeeping such as allocating memory, deallocating memory, and converting the buffer contents into other forms (eg byte arrays or other buffer classes).

There are many buffers classes available to Java applications. LmdbJava was therefore designed from the outset to allow any buffer class to be used without performance penalty. Indeed one of the reasons that LmdbJava exists is because an earlier Java LMDB wrapper (LmdbJni) provided a buffer class that couldn’t easily be updated to the latest version of that buffer class, plus other community members preferred a different buffer entirely. We’ve found it useful being buffer agnostic in LmdbJava, especially as the community sought support for additional buffers over the years (ie Netty ByteBuf and byte[] were not originally provided).

This wiki page tries to cover some of the differences between the supported buffer classes and how they are integrated into LmdbJava.

LMDB - like most C libraries - expects applications to work directly with memory pointers. Indeed one of LMDB’s most ubiquitous structures is named MDB_val. It is used any time that user data needs to be moved between LMDB and an application. The structure is defined as follows:

typedef struct MDB_val {
        size_t           mv_size;       /**< size of the data item */
        void            *mv_data;       /**< address of the data item */
} MDB_val;

Because LmdbJava supports flexible, memory-safe buffer classes, MDB_val is never exposed to Java applications. Instead users select one of the buffers below when instantiating Env, causing LmdbJava to use that buffer type for all subsequent interactions with application code. As the choice of buffer has no bearing on the way that data is persisted in LMDB, it is perfectly fine to change buffer types at any time (or even use them concurrently in different Envs).

ByteBuffer is Java’s standard buffer class. Users construct a ByteBuffer by allocating it memory. This memory can either be on-heap or off-heap. On-heap means the memory is allocated within the usual Java garbage collected memory regions. Off-heap means the memory is outside the garbage collected memory regions. It is more expensive (latency-wise) to allocate off-heap memory, so most of the time you will want to allocate off-heap memory and reuse the resulting ByteBuffer repeatedly during the application lifecycle. If you intend to present the memory to LmdbJava, off-heap memory must be used (an exception will be raised if this is not the case).

One of the issues with ByteBuffer is that memory is always cleared when being allocated. This has a small performance impact. A further issue is the memory is limited to 2 GB, although this isn’t a practical problem for the size of values expected in a key-value store like LMDB.

Because the Java programming language has always maintained such a strong emphasis on safety, unsurprisingly there is no official support for extracting the memory address allocated to a ByteBuffer's off-heap memory. This presents a challenge for LmdbJava, because it often needs to "point" an MDB_val at that off-heap memory for operations like seeking to a particular key. While we could copy the data from the Java ByteBuffer to some separately-allocated C-side memory, this imposes a data copy cost. A more significant issue is that every read operation requires we provided applications with a ByteBuffer that points to whatever off-heap memory location was nominated in the MDB_val. It’s a more significant problem because we not only have the copy cost, but we also have the allocation cost of another ByteBuffer, and the GC pressure of deallocating old ByteBuffers.

Given that minimising latency is one of the key goals of LmdbJava, we resolve the above issues by using "illegal" Java APIs to get or set a ByteBuffer’s underlying memory address and size. We implement this through private field reflection or using the Unsafe class. Such techniques are widely used in low latency Java programming and as such are unlikely to be abruptly removed in the future. In the worst case scenario of their complete removal, we can always fall back to instantiating ByteBuffers on each operation and copying memory.

To use ByteBuffer with LmdbJava, there are two buffer proxies available. The safer, reflection-based option is obtained by instantiating Env with ByteBufferProxy.PROXY_SAFE. To use the faster option that relies on the Unsafe class, use ByteBufferProxy.PROXY_OPTIMAL.

LmdbJava can also work with byte arrays. However, support for such arrays is limited in two important ways:

1. Slower performance
2. No support for ranges

The performance limitation is because a byte[] cannot be modified to point to an arbitrary memory location. Unlike every other buffer type, where there’s some kind of "memory address" and "length" field which we can get and set, a byte[] cannot be similarly manipulated. As such LmdbJava is required to copy data between a byte[] and off-heap locations used by LMDB. This means using byte[] with LmdbJava will always be slower than correctly using any other buffer type.

The byte[] range limitation refers to the absence of starting offset and length parameters when presenting a byte[] to LmdbJava methods. While most idiomatic Java methods that deal with arrays will accept such parameters, we decided against this in LmdbJava’s case because doing so would require a series of overloaded methods that are unnecessary for every other buffer type. This is because every other buffer type provides an efficient option for indicating an offset and length within their buffer instances. As such it’s a design decision to not over-complicate the LmdbJava API.

While byte[] ranges are not supported by LmdbJava methods, users are directed to Arrays.copyOfRange(byte[], int, int) to efficiently create a new array containing a subset of source array data if required.

Despite the limitations of using byte[] as a buffer type, it is a pragmatic option in that it avoids the learning curve of another buffer API. It also avoids the "illegal" operations necessary for other buffer types to deliver optimal performance (private field reflection or using Unsafe). For applications that are not especially latency sensitive, byte[] is an attractive option.

To use a byte array buffer, instantiate Env with ByteArrayProxy.PROXY_BA.

Real Logic maintains several high quality, low latency open source projects, including Simple Binary Encoding (a serialisation mechanism) and Aeron (an ordered message transport). Standalone, reusable types are extracted into the Agrona project, and this includes several buffer implementations.

The Agrona buffers are used in both SBE and Aeron. There is a read-only buffer interface named DirectBuffer, and an extended writable buffer interface named MutableDirectBuffer. Separation of these read-only and read-write interfaces makes API contracts clearer by exposing the intended mutability characteristics of the buffer.

Each interface offers methods for reading (and if applicable writing) various data types in the buffer, such as strings (Unicode and ASCII), numbers (of many sizes and both Endianness) and arbitrary bytes. There are also methods to wrap the buffer instance around some other arbitrary buffer, such as a specific memory location, ByteBuffer, another Agrona buffer slice etc.

LmdbJava supports Agrona and it is recommended for applications that need a flexible, low latency buffer. We defer to Agrona for all buffer wrapping operations and use its UnsafeBuffer to provide a pool of MutableDirectBuffer instances to applications. This means Agrona encapsulates the required "illegal" operations to wrap the MDB_val-specified memory location. When instantiating a MutableDirectBuffer implementation to present to LmdbJava, it is a requirement that it is backed by off-heap memory.

To use Agrona buffers with LmdbJava, instantiate Env with DirectBufferProxy.PROXY_DB.

Netty is a popular and mature framework for building network applications. It includes its own buffer type and offers mechanisms for creating, pooling and using those buffers.

Netty buffers wrap either byte[] or ByteBuffers. In a similar approach to that used for Agrona, LmdbJava uses Netty’s PooledUnsafeDirectByteBuf to access to MDB_val memory locations. Unsafe is used to modify the internal memory address being used by Netty’s buffer.

To use Netty with LmdbJava, instantiate Env with ByteBufProxy.PROXY_NETTY.

LmdbJava allows applications to select between four types of buffers, each with their own trade-offs. With the exception of byte[], each of the supported buffers will deliver very similar overall performance. That’s because the LmdbJava implementation internals pool these buffers and avoid any allocation or copying costs when interacting with MDB_val. Any subtle implementation differences between the buffers are unlikely to be measurable in real-world workloads.

If you are using an unsupported buffer, you can write your own BufferProxy implementation and present it to Env like any other supported buffer type. Pull requests that contribute new buffer proxies for popular open source buffers or enhance existing buffer proxies are most welcome.