Skip to content

Compress embedded binaries (site/out/bin) #2202

Closed

Description

@mafredri

I'm suggesting we start compressing the binaries in site/out/bin as a step in goreleaser. This can go two ways:

  1. We serve the raw gzip or zip archives in coder server
  2. We transparently decompress the files when serving them

Currently the coder binary is very big sitting at 287M. This can cause the coder binary to take a very long time to respond on first run (before ending up in the fs cache). Here's an example on my 2017 MBP:

wget https://github.com/coder/coder/releases/download/v0.6.4/coder_0.6.4_darwin_amd64.zip
unzip coder_0.6.4_darwin_amd64.zip
time ./coder --help
# ...
./coder  0.10s user 0.09s system 4% cpu 4.446 total

Simply running a gzip on these binaries brings the total binary size down from 287M to 128M and improves performance significantly.

time ./coder-gzipped --help
./coder-gzipped --help  0.10s user 0.02s system 7% cpu 1.665 total

There's small gains to be had by increasing compression level or using zstd, but not significant enough to warrant any added complexity vs just compressing with something.

Activity

  1. kylecarbs commented on Jun 9, 2022

    @kylecarbs
    Member

    Ahh that's a significant improvement! We should certainly do this.

  2. ketang commented on Jun 9, 2022

    @ketang
    Contributor

    Can someone tell me again why we have 287 MB binaries in the first place?

  3. kylecarbs commented on Jun 9, 2022

    @kylecarbs
    Member

    We put binaries inside of binaries so workspaces can pull em' down to start the agent.

    See: https://dev.coder.com/bin/

  4. ketang commented on Jun 9, 2022

    @ketang
    Contributor

    Oh goodness.

    Sounds like perhaps we eventually may want to have one distributable for completely standalone/airgapped deployments and one much smaller one that has access to the Internet.

  5. mafredri commented on Jun 10, 2022

    @mafredri
    MemberAuthor

    I think this would be nice to get fixed for community release, I experimented a bit yesterday with a few options:

    1. Rely on agent init script to decompress
    2. Extract binaries to a folder (e.g. ${CODER_CACHE_DIRECTORY}/bin) and serve them from there
    3. Implement a fs.FS wrapper around embed.FS that transparently decompresses gzip files

    Option 1 would be the easiest, but adds a requirement for the system running the agent to have tools for decompressing (gzip, busybox gzip, etc).

    Option 2 is quite easy/clean, but delays startup time during initial decompression.

    For option 3, I have a POC for this where we'd utilize the gzip comment header to store file size and transparently rewrite .gz to the decompressed version. Optionally we could serve both allowing the agent init script to download a .gz when gzip is available. This would give us "best-of-both-worlds" where we don't pay a cost upfront (decompress), nor do we pay it at runtime if gzip is available.

    Thoughts? I'm leaning towards 3 but 1 is also pretty nice if we think gzip is an OK dependency to add.

  6. mafredri commented on Jun 10, 2022

    @mafredri
    MemberAuthor

    Actually, option 2 becomes quite attractive if wrap the binaries with tar and zstd. This is because zstd decompression is very fast (takes less than 1 second in all cases) and the --long compression can handle some deduplication. There's also a native Go library for zstd we could utilize to avoid external dependencies: https://github.com/klauspost/compress/tree/master/zstd#zstd

    At the highest level, we get very attractive compression (44MiB vs ~90MiB individually gzipped) at the cost of compression time (takes 2min on my machine):

    ❯ time zstd -22 -T8 --ultra --long coder.tar
    coder.tar            : 17.77%   (   249 MiB =>   44.2 MiB, coder.tar.zst)
    zstd -22 -T8 --long --ultra coder.tar  131.18s user 0.57s system 99% cpu 2:11.83 total
    

    Even at a lower compression rate, we get quite good compression at much greater speeds:

    ❯ time zstd -9 --long coder.tar
    coder.tar            : 25.78%   (   249 MiB =>   64.1 MiB, coder.tar.zst)
    zstd -9 --long coder.tar  6.42s user 0.30s system 105% cpu 6.377 total
    

    Edit: My proposal for going forward with this method would be:

    1. Compress all binaries inside a .tar.zst (tar + zstd), store in site/out/bin/coder-binaries.tar.zst
    2. Generate a sha256sum for the archive and store in site/out/bin/coder-binaries.tar.zst.sha256sum
    3. On coder server startup, check ${CODER_CACHE_DIRECOTRY}/bin/sha256sum, if the sum matches, skip, otherwise extract (and overwrite)
    4. Serve using mux.Handle("/bin/", http.FileServer(http.Dir(coderBinPath)))
  7. added
    apiArea: HTTP API
    siteArea: frontend dashboard
    on Jun 14, 2022
  8. self-assigned this
    on Jun 20, 2022
  9. added a commit that references this issue on Jun 21, 2022
    e2785ad
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

apiArea: HTTP APIcliArea: CLIsiteArea: frontend dashboard

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions