Skip to content

Commit 932f883

Browse files
estespestesp
authored
Merge pull request containerd#3152 from mxpv/bench
Add snapshotters benchmark
2 parents 341b99d + d9526f5 commit 932f883

File tree

3 files changed

+539
-0
lines changed

3 files changed

+539
-0
lines changed

‎contrib/aws/snapshotter_bench_cf.yml‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
AWSTemplateFormatVersion: "2010-09-09"
2+
3+
Description: >
4+
This templates spin ups an EC2 instance with EBS volumes suitable for containerd snapshotters benchmarking.
5+
The template will create EBS volumes for benchmarking (/dev/sdb, /dev/sdc, and /dev/sdd) with same performance characteristics.
6+
/dev/sde volume will be created and used for device mapper thin-pool device.
7+
8+
Parameters:
9+
Key:
10+
Type: AWS::EC2::KeyPair::KeyName
11+
Description: SSH key to use
12+
13+
AMI:
14+
Type: AWS::EC2::Image::Id
15+
Description: AMI ID to use for the EC2 instance. Must be Amazon Linux 2.
16+
Default: "ami-032509850cf9ee54e"
17+
18+
SecurityGroups:
19+
Type: List<AWS::EC2::SecurityGroup::Id>
20+
Description: List of security groups to add to EC2 instance
21+
22+
InstanceType:
23+
Type: String
24+
Default: m4.xlarge
25+
Description: EC2 instance type to use
26+
27+
VolumesIOPS:
28+
Type: Number
29+
Default: 1000
30+
MinValue: 100
31+
MaxValue: 20000
32+
Description: The number of I/O operations per second (IOPS) to reserve for EBS volumes.
33+
34+
VolumesSize:
35+
Type: Number
36+
Default: 20
37+
MinValue: 4
38+
MaxValue: 16384
39+
Description: EBS volumes size, in gibibytes (GiB)
40+
41+
VolumeType:
42+
Type: String
43+
Default: io1
44+
AllowedValues:
45+
- io1
46+
- gp2
47+
- sc1
48+
- st1
49+
Description: >
50+
Volume type to use for EBS volumes (io1 is recommended).
51+
More information on volume types https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html
52+
53+
ContainerStorageSetup:
54+
Type: String
55+
Default: https://github.com/projectatomic/container-storage-setup/archive/v0.6.0.tar.gz
56+
Description: container-storage-setup tool version to install (more details at https://github.com/projectatomic/container-storage-setup)
57+
58+
Resources:
59+
Instance:
60+
Type: AWS::EC2::Instance
61+
Properties:
62+
EbsOptimized: true
63+
InstanceType: !Ref InstanceType
64+
KeyName: !Ref Key
65+
ImageId: !Ref AMI
66+
SecurityGroupIds: !Ref SecurityGroups
67+
BlockDeviceMappings:
68+
- DeviceName: "/dev/xvda" # Root volume
69+
Ebs:
70+
VolumeSize: 64
71+
VolumeType: io1
72+
Iops: 1000
73+
DeleteOnTermination: true
74+
- DeviceName: "/dev/sdb"
75+
Ebs:
76+
VolumeSize: !Ref VolumesSize
77+
VolumeType: !Ref VolumeType
78+
Iops: !Ref VolumesIOPS
79+
DeleteOnTermination: true
80+
- DeviceName: "/dev/sdc"
81+
Ebs:
82+
VolumeSize: !Ref VolumesSize
83+
VolumeType: !Ref VolumeType
84+
Iops: !Ref VolumesIOPS
85+
DeleteOnTermination: true
86+
- DeviceName: "/dev/sdd"
87+
Ebs:
88+
VolumeSize: !Ref VolumesSize
89+
VolumeType: !Ref VolumeType
90+
Iops: !Ref VolumesIOPS
91+
DeleteOnTermination: true
92+
- DeviceName: "/dev/sde"
93+
Ebs:
94+
VolumeSize: !Ref VolumesSize
95+
VolumeType: !Ref VolumeType
96+
Iops: !Ref VolumesIOPS
97+
DeleteOnTermination: true
98+
99+
UserData:
100+
Fn::Base64:
101+
!Sub |
102+
#!/bin/bash
103+
104+
set -ex
105+
106+
yum install -y \
107+
gcc \
108+
git \
109+
btrfs-progs-devel \
110+
libseccomp-devel
111+
112+
amazon-linux-extras install -y golang1.11
113+
114+
# Install container-storage-setup
115+
mkdir -p /tmp/container-storage-setup/unpacked/
116+
cd /tmp/container-storage-setup/
117+
curl -sL ${ContainerStorageSetup} -o archive.tar.gz
118+
tar -xzf archive.tar.gz -C unpacked --strip 1
119+
cd unpacked/
120+
make install-core
121+
rm -rf /tmp/container-storage-setup/
122+
123+
# Prepare EBS volumes
124+
mkdir -p /mnt/{disk1,disk2,disk3}
125+
126+
mkfs.ext4 /dev/sdb
127+
mount /dev/sdb /mnt/disk1/
128+
129+
mkfs.ext4 /dev/sdc
130+
mount /dev/sdc /mnt/disk2/
131+
132+
mkfs.ext4 /dev/sdd
133+
mount /dev/sdd /mnt/disk3
134+
135+
chgrp -R wheel /mnt/disk1/ /mnt/disk2/ /mnt/disk3/
136+
chmod -R 2775 /mnt/disk1/ /mnt/disk2/ /mnt/disk3/
137+
138+
# Prepare thin-pool device
139+
touch /etc/sysconfig/docker-storage-setup
140+
echo DEVS=/dev/sde >> /etc/sysconfig/docker-storage-setup
141+
echo VG=bench >> /etc/sysconfig/docker-storage-setup
142+
container-storage-setup
143+
144+
echo "Done"

‎contrib/aws/snapshotter_bench_readme.md‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
## Requirements
2+
3+
### Running
4+
Due to its dependency on `dmsetup`, executing the snapshotter process in an environment where a udev
5+
daemon is not accessible (such as a container) may result in unexpected behavior. In this case, try executing the
6+
snapshotter with the `DM_DISABLE_UDEV=1` environment variable, which tells `dmsetup` to ignore udev and manage devices
7+
itself. See [lvm(8)](http://man7.org/linux/man-pages/man8/lvm.8.html) and
8+
[dmsetup(8)](http://man7.org/linux/man-pages/man8/dmsetup.8.html) for more information.
9+
10+
## How to run snapshotters benchmark
11+
12+
- `containerd` project contains AWS CloudFormation template to run an EC2 instance suitable for benchmarking.
13+
It installs dependencies, prepares EBS volumes with same performance characteristics, and creates thin-pool device.
14+
You can make stack with the following command (note: there is a charge for using AWS resources):
15+
16+
```bash
17+
aws cloudformation create-stack \
18+
--stack-name benchmark-instance \
19+
--template-body file://benchmark_aws.yml \
20+
--parameters \
21+
ParameterKey=Key,ParameterValue=SSH_KEY \
22+
ParameterKey=SecurityGroups,ParameterValue=sg-XXXXXXXX \
23+
ParameterKey=VolumesSize,ParameterValue=20 \
24+
ParameterKey=VolumesIOPS,ParameterValue=1000
25+
```
26+
27+
- You can find an IP address of newly created EC2 instance in AWS Console or via AWS CLI:
28+
29+
```bash
30+
$ aws ec2 describe-instances \
31+
--instance-ids $(aws cloudformation describe-stack-resources --stack-name benchmark-instance --query 'StackResources[*].PhysicalResourceId' --output text) \
32+
--query 'Reservations[*].Instances[*].PublicIpAddress' \
33+
--output text
34+
```
35+
36+
- SSH to an instance and prepare `containerd`:
37+
38+
```bash
39+
ssh -i SSH_KEY ec2-user@IP
40+
mkdir /mnt/disk1/data /mnt/disk2/data /mnt/disk3/data
41+
go get github.com/containerd/containerd
42+
cd $(go env GOPATH)/src/github.com/containerd/containerd
43+
make
44+
```
45+
46+
- Now you're ready to run the benchmark:
47+
48+
```bash
49+
sudo su -
50+
cd snapshots/testsuite/
51+
go test -bench . \
52+
-dm.thinPoolDev=bench-docker--pool \
53+
-dm.rootPath=/mnt/disk1/data \
54+
-overlay.rootPath=/mnt/disk2/data \
55+
-native.rootPath=/mnt/disk3/data
56+
```
57+
58+
- The output will look like:
59+
60+
```bash
61+
goos: linux
62+
goarch: amd64
63+
pkg: github.com/containerd/containerd/snapshots/testsuite
64+
65+
BenchmarkOverlay/run-4 1 1019730210 ns/op 164.53 MB/s
66+
BenchmarkOverlay/prepare 1 26799447 ns/op
67+
BenchmarkOverlay/write 1 968200363 ns/op
68+
BenchmarkOverlay/commit 1 24582560 ns/op
69+
70+
BenchmarkDeviceMapper/run-4 1 3139232730 ns/op 53.44 MB/s
71+
BenchmarkDeviceMapper/prepare 1 1758640440 ns/op
72+
BenchmarkDeviceMapper/write 1 1356705388 ns/op
73+
BenchmarkDeviceMapper/commit 1 23720367 ns/op
74+
75+
PASS
76+
ok github.com/containerd/containerd/snapshots/testsuite 185.204s
77+
```
78+
79+
- Don't forget to tear down the stack so it does not continue to incur charges:
80+
81+
```bash
82+
aws cloudformation delete-stack --stack-name benchmark-instance
83+
```

0 commit comments

Comments
 (0)