Skip to content

msgpack-jackson3

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History
 
 

msgpack-jackson3

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
src
src
 
 
README.md
README.md
 
 

README.md

jackson-dataformat-msgpack-jackson3

This Jackson 3.x extension library is a component to easily read and write MessagePack encoded data through jackson-databind API.

It extends standard Jackson streaming API (JsonFactory, JsonParser, JsonGenerator), and as such works seamlessly with all the higher level data abstractions (data binding, tree model, and pluggable extensions).

Requirements: Java 17+ and Jackson 3.x. For the Jackson 2.x compatible version, see msgpack-jackson.

Note on imports: Jackson 3 moved its core and databind packages from com.fasterxml.jackson to tools.jackson. User-facing annotations (@JsonProperty, @JsonFormat, etc.) remain in com.fasterxml.jackson.annotation for backward compatibility.

Install

Maven

<dependency>
  <groupId>org.msgpack</groupId>
  <artifactId>jackson-dataformat-msgpack-jackson3</artifactId>
  <version>(version)</version>
</dependency>

Sbt

libraryDependencies += "org.msgpack" % "jackson-dataformat-msgpack-jackson3" % "(version)"

Gradle

repositories {
    mavenCentral()
}

dependencies {
    implementation 'org.msgpack:jackson-dataformat-msgpack-jackson3:(version)'
}

Basic usage

Serialization/Deserialization of POJO

Only thing you need to do is to instantiate MessagePackFactory and pass it to the constructor of tools.jackson.databind.ObjectMapper. And then, you can use it for MessagePack format data in the same way as jackson-databind.

// Instantiate ObjectMapper for MessagePack
ObjectMapper objectMapper = new ObjectMapper(new MessagePackFactory());

// Serialize a Java object to byte array
ExamplePojo pojo = new ExamplePojo("komamitsu");
byte[] bytes = objectMapper.writeValueAsBytes(pojo);

// Deserialize the byte array to a Java object
ExamplePojo deserialized = objectMapper.readValue(bytes, ExamplePojo.class);
System.out.println(deserialized.getName()); // => komamitsu

Or more easily:

ObjectMapper objectMapper = new MessagePackMapper();

We strongly recommend calling MessagePackMapper.Builder#handleBigIntegerAndBigDecimalAsString() if you serialize and/or deserialize BigInteger/BigDecimal values. See Serialize and deserialize BigDecimal as str type internally in MessagePack format for details.

ObjectMapper objectMapper = MessagePackMapper.builder().handleBigIntegerAndBigDecimalAsString().build();

Serialization/Deserialization of List

// Instantiate ObjectMapper for MessagePack
ObjectMapper objectMapper = new MessagePackMapper();

// Serialize a List to byte array
List<Object> list = new ArrayList<>();
list.add("Foo");
list.add("Bar");
list.add(42);
byte[] bytes = objectMapper.writeValueAsBytes(list);

// Deserialize the byte array to a List
List<Object> deserialized = objectMapper.readValue(bytes, new TypeReference<List<Object>>() {});
System.out.println(deserialized); // => [Foo, Bar, 42]

Serialization/Deserialization of Map

// Instantiate ObjectMapper for MessagePack
ObjectMapper objectMapper = new MessagePackMapper();

// Serialize a Map to byte array
Map<String, Object> map = new HashMap<>();
map.put("name", "komamitsu");
map.put("age", 42);
byte[] bytes = objectMapper.writeValueAsBytes(map);

// Deserialize the byte array to a Map
Map<String, Object> deserialized = objectMapper.readValue(bytes, new TypeReference<Map<String, Object>>() {});
System.out.println(deserialized); // => {name=komamitsu, age=42}

Example of Serialization/Deserialization over multiple languages

Java

// Serialize
Map<String, Object> obj = new HashMap<String, Object>();
obj.put("foo", "hello");
obj.put("bar", "world");
byte[] bs = objectMapper.writeValueAsBytes(obj);
// bs => [-126, -93, 102, 111, 111, -91, 104, 101, 108, 108, 111,
//        -93, 98, 97, 114, -91, 119, 111, 114, 108, 100]

Ruby

require 'msgpack'

# Deserialize
xs = [-126, -93, 102, 111, 111, -91, 104, 101, 108, 108, 111,
      -93, 98, 97, 114, -91, 119, 111, 114, 108, 100]
MessagePack.unpack(xs.pack("C*"))
# => {"foo"=>"hello", "bar"=>"world"}

# Serialize
["zero", 1, 2.0, nil].to_msgpack.unpack('C*')
# => [148, 164, 122, 101, 114, 111, 1, 203, 64, 0, 0, 0, 0, 0, 0, 0, 192]

Java

// Deserialize
bs = new byte[] {(byte) 148, (byte) 164, 122, 101, 114, 111, 1,
                 (byte) 203, 64, 0, 0, 0, 0, 0, 0, 0, (byte) 192};
TypeReference<List<Object>> typeReference = new TypeReference<List<Object>>(){};
List<Object> xs = objectMapper.readValue(bs, typeReference);
// xs => [zero, 1, 2.0, null]

Advanced usage

Serialize multiple values without closing an output stream

tools.jackson.databind.ObjectMapper closes an output stream by default after it writes a value. If you want to serialize multiple values in a row without closing an output stream, disable StreamWriteFeature.AUTO_CLOSE_TARGET.

OutputStream out = new FileOutputStream(tempFile);
ObjectMapper objectMapper = MessagePackMapper.builder()
        .disable(StreamWriteFeature.AUTO_CLOSE_TARGET)
        .build();

objectMapper.writeValue(out, 1);
objectMapper.writeValue(out, "two");
objectMapper.writeValue(out, 3.14);
out.close();

MessageUnpacker unpacker = MessagePack.newDefaultUnpacker(new FileInputStream(tempFile));
System.out.println(unpacker.unpackInt());      // => 1
System.out.println(unpacker.unpackString());   // => two
System.out.println(unpacker.unpackFloat());    // => 3.14

Deserialize multiple values without closing an input stream

tools.jackson.databind.ObjectMapper closes an input stream by default after it reads a value. If you want to deserialize multiple values in a row without closing an input stream, disable StreamReadFeature.AUTO_CLOSE_SOURCE.

MessagePacker packer = MessagePack.newDefaultPacker(new FileOutputStream(tempFile));
packer.packInt(42);
packer.packString("Hello");
packer.close();

FileInputStream in = new FileInputStream(tempFile);
ObjectMapper objectMapper = MessagePackMapper.builder()
        .disable(StreamReadFeature.AUTO_CLOSE_SOURCE)
        .build();
System.out.println(objectMapper.readValue(in, Integer.class));
System.out.println(objectMapper.readValue(in, String.class));
in.close();

Serialize not using str8 type

Old msgpack-java (e.g 0.6.7) doesn't support MessagePack str8 type. When your application needs to communicate with such an old MessagePack library, you can disable the data type like this:

MessagePack.PackerConfig config = new MessagePack.PackerConfig().withStr8FormatSupport(false);
ObjectMapper objectMapper = new MessagePackMapper(new MessagePackFactory(config));
// This string is serialized as bin8 type
byte[] resultWithoutStr8Format = objectMapper.writeValueAsBytes(str8LengthString);

Serialize using non-String as a key of Map

When you want to use non-String value as a key of Map, use MessagePackKeySerializer for key serialization.

@JsonSerialize(keyUsing = MessagePackKeySerializer.class)
private Map<Integer, String> intMap = new HashMap<>();

  :

intMap.put(42, "Hello");

ObjectMapper objectMapper = new MessagePackMapper();
byte[] bytes = objectMapper.writeValueAsBytes(intMap);

Map<Integer, String> deserialized = objectMapper.readValue(bytes, new TypeReference<Map<Integer, String>>() {});
System.out.println(deserialized);   // => {42=Hello}

Serialize and deserialize BigDecimal as str type internally in MessagePack format

jackson-dataformat-msgpack-jackson3 represents BigDecimal values as float type in MessagePack format by default for backward compatibility. But the default behavior could fail when handling too large value for double type. So we strongly recommend calling MessagePackMapper.Builder#handleBigIntegerAndBigDecimalAsString() to internally handle BigDecimal values as String.

ObjectMapper objectMapper = MessagePackMapper.builder().handleBigIntegerAndBigDecimalAsString().build();

Pojo obj = new Pojo();
// This value is too large to be serialized as double
obj.value = new BigDecimal("1234567890.98765432100");

byte[] converted = objectMapper.writeValueAsBytes(obj);

System.out.println(objectMapper.readValue(converted, Pojo.class));   // => Pojo{value=1234567890.98765432100}

MessagePackMapper.Builder#handleBigIntegerAndBigDecimalAsString() is equivalent to the following configuration.

ObjectMapper objectMapper = new ObjectMapper(new MessagePackFactory());
objectMapper.configOverride(BigInteger.class).setFormat(JsonFormat.Value.forShape(JsonFormat.Shape.STRING));
objectMapper.configOverride(BigDecimal.class).setFormat(JsonFormat.Value.forShape(JsonFormat.Shape.STRING));

Serialize and deserialize Instant instances as MessagePack extension type

timestamp extension type is defined in MessagePack as type:-1. Registering TimestampExtensionModule.INSTANCE module enables automatic serialization and deserialization of java.time.Instant to/from the MessagePack extension type.

ObjectMapper objectMapper = MessagePackMapper.builder()
        .addModule(TimestampExtensionModule.INSTANCE)
        .build();
Pojo pojo = new Pojo();
// The type of `timestamp` variable is Instant
pojo.timestamp = Instant.now();
byte[] bytes = objectMapper.writeValueAsBytes(pojo);

// The Instant instance is serialized as MessagePack extension type (type: -1)

Pojo deserialized = objectMapper.readValue(bytes, Pojo.class);
System.out.println(deserialized);   // "2022-09-14T08:47:24.922Z"

Deserialize extension types with ExtensionTypeCustomDeserializers

ExtensionTypeCustomDeserializers helps you to deserialize your own custom extension types easily.

Deserialize extension type value directly

// In this application, extension type 59 is used for byte[]
byte[] bytes;
{
    // This ObjectMapper is just for temporary serialization
    ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
    MessagePacker packer = MessagePack.newDefaultPacker(outputStream);

    packer.packExtensionTypeHeader((byte) 59, hexspeak.length);
    packer.addPayload(hexspeak);
    packer.close();

    bytes = outputStream.toByteArray();
}

// Register the type and a deserializer to ExtensionTypeCustomDeserializers
ExtensionTypeCustomDeserializers extTypeCustomDesers = new ExtensionTypeCustomDeserializers();
extTypeCustomDesers.addCustomDeser((byte) 59, data -> {
    if (Arrays.equals(data,
              new byte[] {(byte) 0xCA, (byte) 0xFE, (byte) 0xBA, (byte) 0xBE})) {
        return "Java";
    }
    return "Not Java";
});

ObjectMapper objectMapper = new ObjectMapper(
        new MessagePackFactory().setExtTypeCustomDesers(extTypeCustomDesers));

System.out.println(objectMapper.readValue(bytes, Object.class));
  // => Java

Use extension type as Map key

static class TripleBytesPojo
{
  public byte first;
  public byte second;
  public byte third;

  public TripleBytesPojo(byte first, byte second, byte third)
  {
    this.first = first;
    this.second = second;
    this.third = third;
  }

  @Override
  public boolean equals(Object o)
  {
    :
  }

  @Override
  public int hashCode()
  {
    :
  }

  @Override
  public String toString()
  {
    // This key format is used when serialized as map key
    return String.format("%d-%d-%d", first, second, third);
  }

  static class KeyDeserializer
      extends tools.jackson.databind.KeyDeserializer
  {
    @Override
    public Object deserializeKey(String key, DeserializationContext ctxt)
    {
      String[] values = key.split("-");
      return new TripleBytesPojo(Byte.parseByte(values[0]), Byte.parseByte(values[1]), Byte.parseByte(values[2]));
    }
  }

  static TripleBytesPojo deserialize(byte[] bytes)
  {
    return new TripleBytesPojo(bytes[0], bytes[1], bytes[2]);
  }
}

:

byte extTypeCode = 42;

ExtensionTypeCustomDeserializers extTypeCustomDesers = new ExtensionTypeCustomDeserializers();
extTypeCustomDesers.addCustomDeser(extTypeCode, new ExtensionTypeCustomDeserializers.Deser()
{
  @Override
  public Object deserialize(byte[] value)
        throws IOException
  {
    return TripleBytesPojo.deserialize(value);
  }
});

SimpleModule module = new SimpleModule();
module.addKeyDeserializer(TripleBytesPojo.class, new TripleBytesPojo.KeyDeserializer());
ObjectMapper objectMapper = MessagePackMapper.builder(
        new MessagePackFactory().setExtTypeCustomDesers(extTypeCustomDesers))
        .addModule(module)
        .build();

Map<TripleBytesPojo, Integer> deserializedMap =
        objectMapper.readValue(serializedData,
            new TypeReference<Map<TripleBytesPojo, Integer>>() {});

Use extension type as Map value

static class TripleBytesPojo
{
  public byte first;
  public byte second;
  public byte third;

  public TripleBytesPojo(byte first, byte second, byte third)
  {
    this.first = first;
    this.second = second;
    this.third = third;
  }

  static class Deserializer
      extends StdDeserializer<TripleBytesPojo>
  {
    protected Deserializer()
    {
      super(TripleBytesPojo.class);
    }

    @Override
    public TripleBytesPojo deserialize(JsonParser p, DeserializationContext ctxt)
    {
      return TripleBytesPojo.deserialize(p.getBinaryValue());
    }
  }

  static TripleBytesPojo deserialize(byte[] bytes)
  {
    return new TripleBytesPojo(bytes[0], bytes[1], bytes[2]);
  }
}

:

byte extTypeCode = 42;

ExtensionTypeCustomDeserializers extTypeCustomDesers = new ExtensionTypeCustomDeserializers();
extTypeCustomDesers.addCustomDeser(extTypeCode, new ExtensionTypeCustomDeserializers.Deser()
{
  @Override
  public Object deserialize(byte[] value)
      throws IOException
  {
    return TripleBytesPojo.deserialize(value);
  }
});

SimpleModule module = new SimpleModule();
module.addDeserializer(TripleBytesPojo.class, new TripleBytesPojo.Deserializer());
ObjectMapper objectMapper = MessagePackMapper.builder(
        new MessagePackFactory().setExtTypeCustomDesers(extTypeCustomDesers))
        .addModule(module)
        .build();

Map<String, TripleBytesPojo> deserializedMap =
        objectMapper.readValue(serializedData,
            new TypeReference<Map<String, TripleBytesPojo>>() {});

Serialize a nested object that also serializes

When you serialize an object that has a nested object also serializing with ObjectMapper and MessagePackFactory like the following code, it throws NullPointerException since the nested MessagePackFactory modifies a shared state stored in ThreadLocal.

@Test
public void testNestedSerialization() throws Exception
{
    ObjectMapper objectMapper = new MessagePackMapper();
    objectMapper.writeValueAsBytes(new OuterClass());
}

public class OuterClass
{
  public String getInner() throws JacksonException
  {
    ObjectMapper m = new MessagePackMapper();
    m.writeValueAsBytes(new InnerClass());
    return "EFG";
  }
}

public class InnerClass
{
  public String getName()
  {
    return "ABC";
  }
}

There are a few options to fix this issue, but they introduce performance degradations while this usage is a corner case. A workaround that doesn't affect performance is to call MessagePackFactory#setReuseResourceInGenerator(false). It might be inconvenient to call the API for users, but it's a reasonable tradeoff with performance for now.

ObjectMapper objectMapper = new ObjectMapper(
  new MessagePackFactory().setReuseResourceInGenerator(false));