MESSAGEPACK - AN EFFICIENT
BINARY SERIALIZATION FORMAT
Larry Nung
AGENDA
Introduction
Format
MsgPack.Cli
Messagepack VS Protobuf
Reference
Q & A
2
INTRODUCTION
3
INTRODUCTION
 An efficient binary serialization format
 Like JSON. but fast and small.
4
FORMAT
5
FORMAT
6
FORMAT
7
FORMAT
format name first byte (in binary) first byte (in hex)
positive fixint 0xxxxxxx 0x00 - 0x7f
fixmap 1000xxxx 0x80 - 0x8f
fixarray 1001xxxx 0x90 - 0x9f
fixstr 101xxxxx 0xa0 - 0xbf
nil 11000000 0xc0
(never used) 11000001 0xc1
false 11000010 0xc2
true 11000011 0xc3
bin 8 11000100 0xc4
bin 16 11000101 0xc5
bin 32 11000110 0xc6 8
FORMAT
format name first byte (in binary) first byte (in hex)
ext 8 11000111 0xc7
ext 16 11001000 0xc8
ext 32 11001001 0xc9
float 32 11001010 0xca
float 64 11001011 0xcb
uint 8 11001100 0xcc
uint 16 11001101 0xcd
uint 32 11001110 0xce
uint 64 11001111 0xcf
int 8 11010000 0xd0
int 16 11010001 0xd1 9
FORMAT
format name first byte (in binary) first byte (in hex)
int 32 11010010 0xd2
int 64 11010011 0xd3
fixext 1 11010100 0xd4
fixext 2 11010101 0xd5
fixext 4 11010110 0xd6
fixext 8 11010111 0xd7
fixext 16 11011000 0xd8
str 8 11011001 0xd9
str 16 11011010 0xda
str 32 11011011 0xdb
array 16 11011100 0xdc 10
FORMAT
format name first byte (in binary) first byte (in hex)
array 32 11011101 0xdd
map 16 11011110 0xde
map 32 11011111 0xdf
negative fixint 111xxxxx 0xe0 - 0xff
11
FORMAT
12
 Fixed length types
 Integer
 Floating point
 Boolean
 Nil
FORMAT
13
 Nil => 11000000 => 0xc0
 Boolean
 False => 11000010 => 0xc2
 True => 11000011 => 0xc3
 Positive fixint
 0 => 00000000 => 0x00
 1 => 00000001 => 0x01
FORMAT
14
 Variable length types
 Raw bytes
 Array
 Map
FORMAT
 Fixstr
 Compact => 10100007 01100011 01101111 01101101
01110000 01100001 01100011 01110100 => a7 63 6f 6d
70 61 63 74
15
MSGPACK.CLI
16
MSGPACK.CLI
 Install-Package MsgPack.Cli
17
MSGPACK.CLI
using MsgPack.Serialization;
...
public static byte[] Serialize<T>(T thisObj) {
var serializer = SerializationContext.Default.GetSerializer<T>();
using (var ms = new MemoryStream()) {
serializer.Pack(ms, thisObj);
return ms.ToArray();
}
}
public static T Deserialize<T>(byte[] bytes) {
var serializer = SerializationContext.Default.GetSerializer<T>();
using (var byteStream = new MemoryStream(bytes)) {
return serializer.Unpack(byteStream);
}
} 18
MSGPACK.CLI
public class OldPerson {
[MessagePackMember(0)]
public String Name { get; set; }
[MessagePackMember(1)]
public String NickName { get; set; }
[MessagePackIgnore]
public Object Tag { get; set; }
}
public class NewPerson {
[MessagePackMember(2)]
public String ID { get; set; }
[MessagePackMember(0)]
public String Name { get; set; }
[MessagePackMember(1)]
public String NickName { get; set; }
}
19
MSGPACK.CLI
…
var larry = new OldPerson {
Name = "Larry Nung",
NickName = "Larry"
};
var bytes = Serialize(larry);
var person = Deserialize<NewPerson>(bytes);
Console.WriteLine("{0} ({1})", person.NickName,
person.Name);
…
20
MESSAGEPACK VS PROTOBUF
21
MESSAGEPACK VS PROTOBUF
private static void SerializeToStream<T>(Stream stream, T obj) {
stream.Seek(0, SeekOrigin.Begin);
Serializer.Serialize(stream, obj);
}
private static T DeSerializeFromStream<T>(Stream stream) {
stream.Seek(0, SeekOrigin.Begin);
return Serializer.Deserialize<T>(stream);
}
public static void Serialize<T>(Stream stream, T thisObj) {
stream.Seek(0, SeekOrigin.Begin);
var serializer = SerializationContext.Default.GetSerializer<T>();
serializer.Pack(stream, thisObj);
}
public static T Deserialize<T>(Stream stream) {
stream.Seek(0, SeekOrigin.Begin);
var serializer = SerializationContext.Default.GetSerializer<T>();
return serializer.Unpack(stream);
}
22
MESSAGEPACK VS PROTOBUF
var counts = new int[] { 100, 1000, 10000, 100000, 1000000, 10000000, 100000000 };
...
Serialize(ms, larry);
Console.WriteLine("MsgPack Size: {0}", ms.Length);
SerializeToStream(ms, larry);
Console.WriteLine("ProtoBuf Size: {0}", ms.Length);
...
foreach (var count in counts)
{
…
Console.WriteLine("MsgPack Serialize: {0} ms", DoTest(count, () => { Serialize(ms,
larry); }));
Console.WriteLine("ProtoBuf Serialize: {0} ms", DoTest(count, () =>
{ SerializeToStream(ms, larry); }));
Console.WriteLine("MsgPack DeSerialize: {0} ms", DoTest(count, () =>
{ Deserialize<Person>(ms); }));
Console.WriteLine("ProtoBuf DeSerialize: {0} ms", DoTest(count, () =>
{ DeSerializeFromStream<Person>(ms); }));
}
23
MESSAGEPACK VS PROTOBUF
24
MESSAGEPACK VS PROTOBUF
Serialize
25
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
100 1000 10000 100000 1000000 10000000 100000000
MsgPack
Protobuf
MESSAGEPACK VS PROTOBUF
Deserialize
26
0
10000
20000
30000
40000
50000
60000
70000
80000
100 1000 10000 100000 1000000 10000000 100000000
MsgPack
Protobuf
REFERENCE
27
REFERENCE
 MessagePack: It's like JSON. but fast and small.
 http://msgpack.org/
28
Q&A
29
QUESTION & ANSWER
30

MessagePack - An efficient binary serialization format

  • 1.
    MESSAGEPACK - ANEFFICIENT BINARY SERIALIZATION FORMAT Larry Nung
  • 2.
  • 3.
  • 4.
    INTRODUCTION  An efficientbinary serialization format  Like JSON. but fast and small. 4
  • 5.
  • 6.
  • 7.
  • 8.
    FORMAT format name firstbyte (in binary) first byte (in hex) positive fixint 0xxxxxxx 0x00 - 0x7f fixmap 1000xxxx 0x80 - 0x8f fixarray 1001xxxx 0x90 - 0x9f fixstr 101xxxxx 0xa0 - 0xbf nil 11000000 0xc0 (never used) 11000001 0xc1 false 11000010 0xc2 true 11000011 0xc3 bin 8 11000100 0xc4 bin 16 11000101 0xc5 bin 32 11000110 0xc6 8
  • 9.
    FORMAT format name firstbyte (in binary) first byte (in hex) ext 8 11000111 0xc7 ext 16 11001000 0xc8 ext 32 11001001 0xc9 float 32 11001010 0xca float 64 11001011 0xcb uint 8 11001100 0xcc uint 16 11001101 0xcd uint 32 11001110 0xce uint 64 11001111 0xcf int 8 11010000 0xd0 int 16 11010001 0xd1 9
  • 10.
    FORMAT format name firstbyte (in binary) first byte (in hex) int 32 11010010 0xd2 int 64 11010011 0xd3 fixext 1 11010100 0xd4 fixext 2 11010101 0xd5 fixext 4 11010110 0xd6 fixext 8 11010111 0xd7 fixext 16 11011000 0xd8 str 8 11011001 0xd9 str 16 11011010 0xda str 32 11011011 0xdb array 16 11011100 0xdc 10
  • 11.
    FORMAT format name firstbyte (in binary) first byte (in hex) array 32 11011101 0xdd map 16 11011110 0xde map 32 11011111 0xdf negative fixint 111xxxxx 0xe0 - 0xff 11
  • 12.
    FORMAT 12  Fixed lengthtypes  Integer  Floating point  Boolean  Nil
  • 13.
    FORMAT 13  Nil =>11000000 => 0xc0  Boolean  False => 11000010 => 0xc2  True => 11000011 => 0xc3  Positive fixint  0 => 00000000 => 0x00  1 => 00000001 => 0x01
  • 14.
    FORMAT 14  Variable lengthtypes  Raw bytes  Array  Map
  • 15.
    FORMAT  Fixstr  Compact=> 10100007 01100011 01101111 01101101 01110000 01100001 01100011 01110100 => a7 63 6f 6d 70 61 63 74 15
  • 16.
  • 17.
  • 18.
    MSGPACK.CLI using MsgPack.Serialization; ... public staticbyte[] Serialize<T>(T thisObj) { var serializer = SerializationContext.Default.GetSerializer<T>(); using (var ms = new MemoryStream()) { serializer.Pack(ms, thisObj); return ms.ToArray(); } } public static T Deserialize<T>(byte[] bytes) { var serializer = SerializationContext.Default.GetSerializer<T>(); using (var byteStream = new MemoryStream(bytes)) { return serializer.Unpack(byteStream); } } 18
  • 19.
    MSGPACK.CLI public class OldPerson{ [MessagePackMember(0)] public String Name { get; set; } [MessagePackMember(1)] public String NickName { get; set; } [MessagePackIgnore] public Object Tag { get; set; } } public class NewPerson { [MessagePackMember(2)] public String ID { get; set; } [MessagePackMember(0)] public String Name { get; set; } [MessagePackMember(1)] public String NickName { get; set; } } 19
  • 20.
    MSGPACK.CLI … var larry =new OldPerson { Name = "Larry Nung", NickName = "Larry" }; var bytes = Serialize(larry); var person = Deserialize<NewPerson>(bytes); Console.WriteLine("{0} ({1})", person.NickName, person.Name); … 20
  • 21.
  • 22.
    MESSAGEPACK VS PROTOBUF privatestatic void SerializeToStream<T>(Stream stream, T obj) { stream.Seek(0, SeekOrigin.Begin); Serializer.Serialize(stream, obj); } private static T DeSerializeFromStream<T>(Stream stream) { stream.Seek(0, SeekOrigin.Begin); return Serializer.Deserialize<T>(stream); } public static void Serialize<T>(Stream stream, T thisObj) { stream.Seek(0, SeekOrigin.Begin); var serializer = SerializationContext.Default.GetSerializer<T>(); serializer.Pack(stream, thisObj); } public static T Deserialize<T>(Stream stream) { stream.Seek(0, SeekOrigin.Begin); var serializer = SerializationContext.Default.GetSerializer<T>(); return serializer.Unpack(stream); } 22
  • 23.
    MESSAGEPACK VS PROTOBUF varcounts = new int[] { 100, 1000, 10000, 100000, 1000000, 10000000, 100000000 }; ... Serialize(ms, larry); Console.WriteLine("MsgPack Size: {0}", ms.Length); SerializeToStream(ms, larry); Console.WriteLine("ProtoBuf Size: {0}", ms.Length); ... foreach (var count in counts) { … Console.WriteLine("MsgPack Serialize: {0} ms", DoTest(count, () => { Serialize(ms, larry); })); Console.WriteLine("ProtoBuf Serialize: {0} ms", DoTest(count, () => { SerializeToStream(ms, larry); })); Console.WriteLine("MsgPack DeSerialize: {0} ms", DoTest(count, () => { Deserialize<Person>(ms); })); Console.WriteLine("ProtoBuf DeSerialize: {0} ms", DoTest(count, () => { DeSerializeFromStream<Person>(ms); })); } 23
  • 24.
  • 25.
  • 26.
    MESSAGEPACK VS PROTOBUF Deserialize 26 0 10000 20000 30000 40000 50000 60000 70000 80000 1001000 10000 100000 1000000 10000000 100000000 MsgPack Protobuf
  • 27.
  • 28.
    REFERENCE  MessagePack: It'slike JSON. but fast and small.  http://msgpack.org/ 28
  • 29.
  • 30.