Skip to content

Release management#36

Merged
mislav merged 6 commits intomasterfrom
release-mgmt
Nov 8, 2019
Merged

Release management#36
mislav merged 6 commits intomasterfrom
release-mgmt

Conversation

@probablycorey
Copy link
Copy Markdown
Contributor

@probablycorey probablycorey commented Oct 28, 2019

This PR sets us up to use https://goreleaser.com.

GoReleaser builds Go binaries for several platforms, creates a GitHub release and then pushes a Homebrew formula to a tap repository. All that wrapped in your favorite CI.

I used github actions to call goreleaser to build the binaries. It seems to be working well, but it doesn't create the release. I'm guessing the release isn't showing up because this is a PR and not on master, but I need to verify that. I've modified the action so it only will create a release when a new tag is pushed. So I think the pattern goes like this.

  • We work work work on master
  • When we are happy and want to create a new release we do this:

git tag -a v0.0.2 -m "v0.0.2" && git push origin v0.0.2

Next up I'm going to look at hub's install script, make a curl style install script, and then ping security to learn why this was such a bad idea.

@probablycorey
Copy link
Copy Markdown
Contributor Author

probablycorey commented Oct 28, 2019

Ignore all this, I solved it in the comment below.

I've hit a wall with this for today. Locally I can get it working with `goreleaser --snapshot --skip-publish --rm-dist`. But when it is run as an action this is the error I get:

> ##[error]   ⨯ release failed after 0.12s error=failed to build for linux_amd64: main.go:7:2: cannot > find package "github.com/github/gh-cli/command" in any of:
> 	/opt/hostedtoolcache/go/1.10.8/x64/src/github.com/github/gh-cli/command (from $GOROOT)
>	/home/runner/go/src/github.com/github/gh-cli/command (from $GOPATH)
>
> ##[error]The process '/home/runner/work/_temp/0d09158f-9529-4bd1-a5fc-2b2ec20c3b77/goreleaser' failed with exit code 1
> ##[error]Node run failed with exit code 1


My guess is that this has something to do with go modules, but I can't figure it out yet.

@probablycorey
Copy link
Copy Markdown
Contributor Author

Turns out it was go module related. I needed to make goreleaser use go v1.13. The action seems to work now, but I have no idea where the release it put!

@probablycorey probablycorey changed the title [WIP] Release management Release management Oct 29, 2019
Copy link
Copy Markdown
Contributor

@mislav mislav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is how hub does it: https://github.com/github/hub/blob/master/script/cross-compile

That defines the build matrix, then package creates the tarballs and github-release uses hub release create to attach assets to a new release.

If we wanted to use Actions and skip goreleaser (in case we don't manage to figure out how it works; in my mind it looks slightly over-engineered for our needs), we could potentially create a release and attach assets using Octokit https://octokit.github.io/rest.js/#octokit-routes-repos-upload-release-asset

on:
push:
branches:
- "!*"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this branches part necessary? We can only leave tags if we want to build only on v* tagged releases

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Turns out the tutorial I read was wrong and we DON'T need the branches filter!

@probablycorey probablycorey changed the title Release management [WIP] Release management Oct 29, 2019
@probablycorey
Copy link
Copy Markdown
Contributor Author

I'm still working on this PR.

(in case we don't manage to figure out how it works; in my mind it looks slightly over-engineered for our needs)

goreleaser is over-engineered for what we need, but it also leaves a pretty tiny footprint on our project (a github action and a .goreleaser.yml file). I'm ok with using it for now because it does a lot of stuff automatically for us and it will be easy to swap out if we decide not to use it in the future.

All that being said, I want to though goreleaser in off a bridge because it works perfectly except I can't get it to automatically update the brew formula. I'm still trying to figure out why that is, hopefully I will solve it in my dreams tonight.

@mislav
Copy link
Copy Markdown
Contributor

mislav commented Oct 30, 2019

@probablycorey You can use this to update the Homebrew formula, by yours truly https://github.com/marketplace/actions/bump-homebrew-formula 💖

@probablycorey
Copy link
Copy Markdown
Contributor Author

Automating our build process isn’t too difficult. But it turns out automating the install process IS difficult! Here are the options we have and why each of them stink in their own special way.

  • Host the gh binary as a release on a private repo
    • using brew or curl will fail because they don’t have the credentials to authenticate with GitHub. This means people would have to enter they username/password/2fa and then use duo to download the binary.
  • Use a tap and have brew build the repo (this is what https://github.com/github/homebrew-gh does)
    • You need to have git setup to use credentials, most developers will already have this.
    • You need to have the entire go build setup. This is annoying because it will take time, but brew handles this.
    • We don’t have as much control over the release unless we update the formula after each build.
    • It will only work for GitHubbers
  • Host the gh binary as a release on a public repo
    • This works, but anyone will be able to see the binary and download it.
    • If we want this automated, we’ll need to write a GitHub action to move the binary over to the public repo
  • Host the gh binary as a release on a private repo (but don’t use an install script)
    • This will work for GitHubbers, but it is annoying for them because they need to go to a website, download a tarball, untar it, and then install it in usr/local/bin.

Based on these I'm going to start working on the Host the gh binary as a release on a public repo option. I think this makes the most sense because it will work for githubbers and non-githubbers, and it is a pretty simple solution. I just need a github action to copy the release to another repo.

@probablycorey probablycorey changed the title [WIP] Release management Release management Nov 6, 2019
@probablycorey
Copy link
Copy Markdown
Contributor Author

@mislav I think this is finally ready to look at with you. If we like this, then we need to do three things.

  1. Pick a public repo to host the binaries and homebrew formula (it is using a temp repo I created called probablycorey/homebrew-gghh right now)
  2. Add a PAT to access the public repo
  3. Add your https://github.com/mislav/bump-homebrew-formula-action action on the public repo so the formula gets bumped.

Since this is installing a binary asset, I wasn't sure how to make your bump-homebrew-formula-action setup work, so I want to check in with you about that.

filters:
exclude:
- "^docs:"
- "^test:"
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may want to tell gorelaser to build this as production so it uses our production oauth app.

@probablycorey
Copy link
Copy Markdown
Contributor Author

This works, I tested it out by creating this tag https://github.com/github/gh-cli/releases/tag/v0.0.114 and a release was created, binaries built, binaries copied to https://github.com/probablycorey/homebrew-gghh/releases, and the brew formula was updated.

Some things I did after talking with @mislav
-[x] Instead of grabbing the latest release, I use the GITHUB_REF env var to grab the tag ref
-[ ] @mislav is concerned about race conditions around asset creation. But I looked into the gorelease code and what I saw was it synchronously builds and uploads the assets before the copy-release action is called. But there is a possibility I don't fully understand the race condition.
-[ ] Use inputs/outputs on actions to pass the download url to the "bump brew" action.
-[ ] Switched to typescript and https://github.com/zeit/ncc based on @mislavs suggestion. I don't love how I have to remember to run npm run build after every change, but I do love the type checking that typescript gives me.

The releases are still going to probablycorey/homebrew-gghh. I'm fine with moving to a more official repo now, or we can do that in another PR. I think reusing the github/homebrew-gh repo and making it public makes sense.

@vilmibm vilmibm added the master label Nov 6, 2019
@mislav
Copy link
Copy Markdown
Contributor

mislav commented Nov 7, 2019

Re: race conditions: the only way to create a Release with assets is to first create a Release (1 API call) and then attach assets to it one by one (N API calls). This process can be seen in goreleaser here.

So if you have an Action that is triggered on v* tag push or the "release create" event, that one executes as soon as the Release was created (first API call), but there's no guarantee that assets have finished uploading yet.

  1. Release created 👈 "release create" (and v* tag push) actions run triggered here
  2. Asset 1 finished uploading
  3. Asset 2 finished uploading

This problem is not very likely to bite us since assets typically upload faster than Actions spins up a job. I just wanted us to be aware of this in case Actions runs start being more efficient and we see some oddities down the line.

I think reusing the github/homebrew-gh repo and making it public makes sense.

That makes sense for me too!

Thanks for all the hard work on this 👍

@probablycorey
Copy link
Copy Markdown
Contributor Author

It works, it's on github/homebrew-gh and I am very very happy. Please review my code because I can't look at it with any objectivity anymore!

@mislav
Copy link
Copy Markdown
Contributor

mislav commented Nov 8, 2019

I've taken this and ran with it.

Goreleaser:

  • compiled binaries are now configured with correct Version and BuildDate
  • the "darwin" package is now called "macOS"
  • now builds a Windows binary too and packages it as zip
  • new package structure:
    $ tar tzf dist/gh_0.1.1_macOS_amd64.tar.gz
    gh_0.1.1_macOS_amd64/README.md
    gh_0.1.1_macOS_amd64/bin/gh
    

copy-release-to-another-repo:

  • added test-run.sh for easier testing
  • switched to import ... from "..." syntax for stricter type checking
  • eliminated any types
  • switched to context instead of reading GITHUB_REPOSITORY, GITHUB_REF
  • switched to getInput instead of reading INPUT_*
  • have the action work if target Release already exists
  • now both "macOS" and "linux" packages are copied

Copy link
Copy Markdown
Contributor

@mislav mislav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@mislav mislav merged commit f1d57cf into master Nov 8, 2019
@mislav mislav deleted the release-mgmt branch November 8, 2019 18:08
vilmibm pushed a commit that referenced this pull request Jun 29, 2021
Updating pr, Fixed output comparison
mislav pushed a commit that referenced this pull request Sep 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants