You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm suggesting we start compressing the binaries in site/out/bin as a step in goreleaser. This can go two ways:
We serve the raw gzip or zip archives in coder server
We transparently decompress the files when serving them
Currently the coder binary is very big sitting at 287M. This can cause the coder binary to take a very long time to respond on first run (before ending up in the fs cache). Here's an example on my 2017 MBP:
wget https://github.com/coder/coder/releases/download/v0.6.4/coder_0.6.4_darwin_amd64.zip
unzip coder_0.6.4_darwin_amd64.zip
time ./coder --help
# ...
./coder 0.10s user 0.09s system 4% cpu 4.446 total
Simply running a gzip on these binaries brings the total binary size down from 287M to 128M and improves performance significantly.
time ./coder-gzipped --help
./coder-gzipped --help 0.10s user 0.02s system 7% cpu 1.665 total
There's small gains to be had by increasing compression level or using zstd, but not significant enough to warrant any added complexity vs just compressing with something.
Sounds like perhaps we eventually may want to have one distributable for completely standalone/airgapped deployments and one much smaller one that has access to the Internet.
I think this would be nice to get fixed for community release, I experimented a bit yesterday with a few options:
Rely on agent init script to decompress
Extract binaries to a folder (e.g. ${CODER_CACHE_DIRECTORY}/bin) and serve them from there
Implement a fs.FS wrapper around embed.FS that transparently decompresses gzip files
Option 1 would be the easiest, but adds a requirement for the system running the agent to have tools for decompressing (gzip, busybox gzip, etc).
Option 2 is quite easy/clean, but delays startup time during initial decompression.
For option 3, I have a POC for this where we'd utilize the gzip comment header to store file size and transparently rewrite .gz to the decompressed version. Optionally we could serve both allowing the agent init script to download a .gz when gzip is available. This would give us "best-of-both-worlds" where we don't pay a cost upfront (decompress), nor do we pay it at runtime if gzip is available.
Thoughts? I'm leaning towards 3 but 1 is also pretty nice if we think gzip is an OK dependency to add.
Actually, option 2 becomes quite attractive if wrap the binaries with tar and zstd. This is because zstd decompression is very fast (takes less than 1 second in all cases) and the --long compression can handle some deduplication. There's also a native Go library for zstd we could utilize to avoid external dependencies: https://github.com/klauspost/compress/tree/master/zstd#zstd
At the highest level, we get very attractive compression (44MiB vs ~90MiB individually gzipped) at the cost of compression time (takes 2min on my machine):
❯ time zstd -22 -T8 --ultra --long coder.tar
coder.tar : 17.77% ( 249 MiB => 44.2 MiB, coder.tar.zst)
zstd -22 -T8 --long --ultra coder.tar 131.18s user 0.57s system 99% cpu 2:11.83 total
Even at a lower compression rate, we get quite good compression at much greater speeds:
❯ time zstd -9 --long coder.tar
coder.tar : 25.78% ( 249 MiB => 64.1 MiB, coder.tar.zst)
zstd -9 --long coder.tar 6.42s user 0.30s system 105% cpu 6.377 total
Edit: My proposal for going forward with this method would be:
Compress all binaries inside a .tar.zst (tar + zstd), store in site/out/bin/coder-binaries.tar.zst
Generate a sha256sum for the archive and store in site/out/bin/coder-binaries.tar.zst.sha256sum
On coder server startup, check ${CODER_CACHE_DIRECOTRY}/bin/sha256sum, if the sum matches, skip, otherwise extract (and overwrite)
Serve using mux.Handle("/bin/", http.FileServer(http.Dir(coderBinPath)))
Activity
kylecarbs commentedon Jun 9, 2022
Ahh that's a significant improvement! We should certainly do this.
ketang commentedon Jun 9, 2022
Can someone tell me again why we have 287 MB binaries in the first place?
kylecarbs commentedon Jun 9, 2022
We put binaries inside of binaries so workspaces can pull em' down to start the agent.
See: https://dev.coder.com/bin/
ketang commentedon Jun 9, 2022
Oh goodness.
Sounds like perhaps we eventually may want to have one distributable for completely standalone/airgapped deployments and one much smaller one that has access to the Internet.
mafredri commentedon Jun 10, 2022
I think this would be nice to get fixed for community release, I experimented a bit yesterday with a few options:
${CODER_CACHE_DIRECTORY}/bin) and serve them from therefs.FSwrapper aroundembed.FSthat transparently decompresses gzip filesOption 1 would be the easiest, but adds a requirement for the system running the agent to have tools for decompressing (
gzip,busybox gzip, etc).Option 2 is quite easy/clean, but delays startup time during initial decompression.
For option 3, I have a POC for this where we'd utilize the
gzipcomment header to store file size and transparently rewrite.gzto the decompressed version. Optionally we could serve both allowing the agent init script to download a.gzwhengzipis available. This would give us "best-of-both-worlds" where we don't pay a cost upfront (decompress), nor do we pay it at runtime ifgzipis available.Thoughts? I'm leaning towards 3 but 1 is also pretty nice if we think
gzipis an OK dependency to add.mafredri commentedon Jun 10, 2022
Actually, option 2 becomes quite attractive if wrap the binaries with
tarandzstd. This is becausezstddecompression is very fast (takes less than 1 second in all cases) and the--longcompression can handle some deduplication. There's also a native Go library forzstdwe could utilize to avoid external dependencies: https://github.com/klauspost/compress/tree/master/zstd#zstdAt the highest level, we get very attractive compression (44MiB vs ~90MiB individually gzipped) at the cost of compression time (takes 2min on my machine):
Even at a lower compression rate, we get quite good compression at much greater speeds:
Edit: My proposal for going forward with this method would be:
.tar.zst(tar + zstd), store insite/out/bin/coder-binaries.tar.zstsha256sumfor the archive and store insite/out/bin/coder-binaries.tar.zst.sha256sumcoder serverstartup, check${CODER_CACHE_DIRECOTRY}/bin/sha256sum, if the sum matches, skip, otherwise extract (and overwrite)mux.Handle("/bin/", http.FileServer(http.Dir(coderBinPath)))feat: Compress and extract slim binaries with zstd (#2533)