Skip to content

CI: the go.sum check compares the file with its own copy, and tee hides benchmark failures #920

Open

Description

@tomersein

I maintain zanadir, which reads a repository's CI and reports which kinds of checks it runs and whether each one can actually fail the build. I ran it on stackql and wanted to share what it found, with a suggestion to run it in CI.

There's a lot here already: golangci-lint with gosec and about fifty linters, actionlint, CodeQL, robot suites on every platform including WSL and Docker, and a benchmark threshold gate. A few checks can't fail the way they look like they should:

  1. The go.sum coherence check compares go.sum with a copy of itself. In build.yml (Linux Build, "Check go sum coherence") the step runs cp go.sum go.sum.bkp and then diff go.sum go.sum.bkp with nothing in between, so it always prints "go.sum check passed". lint.yml runs go mod tidy before golangci-lint but never checks whether that changed anything, so a PR with a stale go.mod or go.sum passes both. Replacing the lint step with go mod tidy -diff fails when the committed files aren't tidy and prints the diff:
          - name: Check go.mod and go.sum are tidy
            run: go mod tidy -diff
  2. A failing benchmark doesn't fail the Benchmark step. Both Benchmark steps (Linux and Linux arm64) run go test -bench ... ./... | tee ${outFile} and then test $?. Without a shell: key the step runs as bash -e {0} (see the run logs), so $? is tee's and the "Benchmarking run failed" branch can't be reached. Benchmarks only run here, so a driver benchmark that hits b.Fatalf just drops out of the benchstat comparison and the step passes. set -o pipefail at the top of both steps, or shell: bash, fixes it.
  3. The release integration test steps can never run. Five "Run robot integration tests" steps are gated on startsWith(env.STATE_SOURCE_TAG, 'build-release'). The "Git Ref Parse" steps write SOURCE_TAG to $GITHUB_STATE, which only feeds an action's own post step, so STATE_SOURCE_TAG is never set (no step environment in the logs has it). startsWith(github.ref_name, 'build-release'), which the traffic-light steps already use, would make them reachable. If releases now come only from main builds, as the release doc describes, they could go instead.
  4. The workflow-level GH_ACCESS_TOKEN reaches every step. build.yml sets it from ACTIONS_PRIVATE_PACKAGE_SECRET in the top-level env, so it is passed to tag-referenced third-party actions such as jiro4989/build-deb-action@v3 (its container starts with -e GH_ACCESS_TOKEN) and snok/install-poetry@v1, and the insteadOf rewrite leaves it in ~/.gitconfig for the test steps. Every stackql and stackql-labs module in go.mod is public, and lint.yml downloads them all without a token, so the secret, the rewrite and GOPRIVATE can probably go. Pinning third-party actions by SHA, kept current by a github-actions entry in dependabot.yml, would also help.
  5. Nothing fails on known vulnerabilities. Dependabot opens update PRs, but no job gates on advisories. govulncheck ./... in both Go modules reports the ones your code reaches, including in the standard library of the Go toolchain in use, the kind of problem [SECURITY] Published release v0.12.778 is built with go1.25.3 and still trusts the compromised eMudhra intermediates; cut a Go 1.26.10 release #857 tracked by hand.
  6. Smaller: rhysd/actionlint:1.6.23 is from early 2023, which is why .github/actionlint.yaml has to list ubuntu-24.04 and the arm runners as self-hosted; recent releases know them. The .NET SDK wrapper targets net8.0;net9.0, and both went out of support on 10 November 2026. Coverage (go test -coverprofile) and a secret scan such as Gitleaks on PRs are missing.

To run zanadir itself, add a job to lint.yml:

  zanadir:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: MustacheCase/zanadir-action@059d8ee8b790fb5a3d70716ca5cb79fda35c67cf # v1.41
        with:
          dir: .
          excluded-categories: Performance Testing,Supply Chain

It reports without failing the build; fail-on-weakened: true makes it gate. Today it scores 4 of 11 categories: linting (golangci-lint), unit tests, SAST (CodeQL) and workflow security (actionlint). Performance testing and supply chain are excluded because zanadir doesn't yet recognise the go test -bench and benchstat gate or npm provenance, which is a gap on our side.

Happy to send a PR for any of these, starting with the go.sum and benchmark checks.

Activity

  1. jeffreyaven commented on Nov 14, 2026

    @jeffreyaven
    Member

    Thanks @tomersein for the detailed report and the offer to help. We'd welcome a focused PR fixing the go.mod/go.sum consistency check and ensuring benchmark failures fail the build on both Linux amd64 and arm64.

    Please keep the existing tests passing and include verification that the updated checks catch the failures described. I'll review it promptly and make sure you're credited for the contribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions