The acceptance tests are blackbox* tests that are expected to interact with resources on a real GitHub instance. They are built on top of the go-internal/testscript package, which provides a framework for building tests for command line tools.

*Note: they aren't strictly blackbox because exec gh commands delegate to a binary set up by testscript that calls into ghcmd.Main. However, since our real func main is an extremely thin adapter over ghcmd.Main, this is reasonable. This tradeoff avoids us building the binary ourselves for the tests, and allows us to get code coverage metrics.

The acceptance tests have a build constraint of //go:build acceptance, this means that go test ./... will continue to work without any modifications. The acceptance tag must therefore be provided when running go test.

The following environment variables are required:

The GitHub host to target e.g. github.com

The organization in which the acceptance tests can manage resources in. Consider using gh-acceptance-testing on github.com.

The token to use for authenticating with the GH_ACCEPTANCE_HOST. This must already have the necessary scopes for each test, and must have permissions to act in the GH_ACCEPTANCE_ORG. See Effective Test Authoring for how tests must handle tokens without sufficient scopes.

It's recommended to create and use a Legacy PAT for this; Fine-Grained PATs do not offer all the necessary privileges required. You can use an OAuth token provided via gh auth login --web and can provide it to the acceptance tests via GH_ACCEPTANCE_TOKEN=$(gh auth token --hostname <host>) but this can be a bit confusing and annoying if you gh auth login again without -s and lose the required scopes.

The test harness infers whether a token authenticates a user from GitHub's documented token prefixes. OAuth (gho_), classic PAT (ghp_), fine-grained PAT (github_pat_), and GitHub App user (ghu_) tokens provide user capabilities. GitHub App installation (ghs_) tokens do not, so scripts marked requires-user-capability: true are omitted from unfiltered runs. Explicitly selecting an incompatible script with GH_ACCEPTANCE_SCRIPT fails with an error instead.

Users with write access can manually run the acceptance test workflow from a specified branch or tag in cli/cli. A run can target Linux, Windows, macOS, or all three, and can run either the complete suite or the tests for one command. The tests use a GitHub App installed for all repositories in the gh-acceptance-testing organization. The workflow uses the gh-acceptance-testing environment, where the App credentials are stored as the GH_ACCEPTANCE_TESTING_APP_CLIENT_ID and GH_ACCEPTANCE_TESTING_APP_PRIVATE_KEY secrets. Tests that require user authentication, including account key management and forks owned by a personal account, are excluded from this workflow based on each script's requires-user-capability declaration.

Managed fixture repositories reduce repository creation by sharing state where tests can safely coexist. Every job uses the same GitHub App installation rate-limit buckets. The dispatch helper warns when another acceptance workflow is already in flight and asks interactive users to confirm another run.

After the workflow exists on the default branch, use script/run-acceptance [REF] [COMMAND] [OS] to dispatch the version of the workflow on a branch or tag in cli/cli. The ref defaults to the current branch, while the command and operating system default to all. List the available command groups with:

script/run-acceptance groups

Acceptance test groups are discovered from the directories under testdata, so adding a group does not require updating the workflow or dispatch helper.

The GitHub App is owned by gh-acceptance-testing and restricted to installation on that account. The organization is dedicated to acceptance testing, so the App was intentionally granted broad installation permissions to let the suite exercise repository and organization administration without repeatedly changing the App registration.

1. In the gh-acceptance-testing organization settings, Developer settings, GitHub Apps, then New GitHub App were selected. See Registering a GitHub App.
2. The App was given a globally unique name and https://github.com/cli/cli as its homepage. No callback or redirect URI was configured, Expire user authorization tokens, Request user authorization (OAuth) during installation, and Enable Device Flow were disabled. No post-installation Setup URL was configured and Redirect on update was disabled. Webhooks were also disabled, and Only on this account was selected under Where can this GitHub App be installed?
3. Every Repository permission and Organization permission was set to the highest available access. Account permissions were not requested because an installation token does not authenticate a user, and the harness skips tests that require user capabilities.
4. After the App was created, its Client ID was recorded and Generate a private key was selected. The complete downloaded PEM file became the private-key secret; GitHub stores only the public half of the generated key. See Managing private keys for GitHub Apps.
5. From the App settings, Install App was selected, followed by gh-acceptance-testing and All repositories. See Installing your own GitHub App.
6. A gh-acceptance-testing environment was created in the cli/cli repository. Access requires approval from cli/code-reviewers, with self-review and administrator bypass disabled. It contains these environment secrets:
   • GH_ACCEPTANCE_TESTING_APP_CLIENT_ID: the App's Client ID.
   • GH_ACCEPTANCE_TESTING_APP_PRIVATE_KEY: the complete contents of the downloaded PEM file.

Each operating-system job mints its own installation token. The token is scoped to the gh-acceptance-testing installation, covers all repositories in that installation, and expires after one hour.

A full example invocation can be found below:

GH_ACCEPTANCE_HOST=<host> GH_ACCEPTANCE_ORG=<org> GH_ACCEPTANCE_TOKEN=<token> go test -tags=acceptance ./acceptance

While writing a new test, target the smallest live surface that can reproduce the behavior. Provide one or more comma-separated script names with GH_ACCEPTANCE_SCRIPT, use -run to select their group, and use -count=1 to bypass Go's test cache:

GH_ACCEPTANCE_SCRIPT=pr-view.txtar GH_ACCEPTANCE_HOST=<host> GH_ACCEPTANCE_ORG=<org> GH_ACCEPTANCE_TOKEN=<token> go test -tags=acceptance -count=1 -run '^TestAcceptance$/^pr$' ./acceptance

Start with one script for a deterministic failure. If concurrency is part of the failure, select only the scripts that exercise the contended resource and repeat that focused set before widening to the complete group or suite.

To get code coverage, go test can be invoked with coverpkg and coverprofile like so:

GH_ACCEPTANCE_HOST=<host> GH_ACCEPTANCE_ORG=<org> GH_ACCEPTANCE_TOKEN=<token> go test -tags=acceptance -coverprofile=coverage.out -coverpkg=./... ./acceptance

This section is to be expanded over time as we write more tests and learn more.

The following custom environment variables are made available to the scripts:

• GH_HOST: Set to value of the GH_ACCEPTANCE_ORG env var provided to go test
• ORG: Set to the value of the GH_ACCEPTANCE_ORG env var provided to go test
• GH_TOKEN: Set to the value of the GH_ACCEPTANCE_TOKEN env var provided to go test
• RANDOM_STRING: Set to a length 10 random string of letters to help isolate globally visible resources
• SCRIPT_NAME: Set to the name of the testscript currently running, without extension and replacing hyphens with underscores e.g. pr_view
• HOME: Set to the initial working directory. Required for git operations
• GH_CONFIG_DIR: Set to the initial working directory. Required for gh operations

Every script must begin with a structured header comment declaring whether it needs a token that authenticates a user:

# requires-user-capability: false

Every script must also declare exactly one repository fixture mode:

fixture-repo shared REPO
fixture-repo isolated REPO
fixture-repo none

shared reuses one initialized private repository across all opting-in scripts in the test process. Shared scripts must tolerate concurrent and accumulated state: use unique resource names, paginate and filter list operations, capture resource IDs instead of selecting the first or latest result, and avoid repository-global or default-branch mutations. When scripts create the same tree from the same parent, include the script's $RANDOM_STRING in the commit contents or message; branch names do not affect commit IDs.

isolated creates an initialized private repository exclusively for the script. Use it when clean state, repository-global mutation, or multiple coordinated Git ref updates are required. Consolidate related operations into one isolated script when they can share that repository sequentially.

none creates no managed repository. Use it when no repository is needed or when a test needs multiple repositories, public visibility, special creation options, or direct coverage of repository lifecycle commands. In that mode, the script owns creation and cleanup.

Scripts share token-wide API rate limits. Count requests across the whole test process and combine compatible live assertions. Keep coverage for a narrowly limited endpoint in one script so concurrent scripts cannot burst the limit. For Code Search's 10 requests/minute bucket, use at most five HTTP requests in the entire acceptance process even when they run sequentially. This leaves room for pagination, retries, and other token activity. Keep representative live coverage and use unit tests for remaining variants.

Tests that cancel workflow runs should use a self-contained, deliberately long-running job so it cannot finish before the cancellation request, plus a short job timeout to bound a failed cancellation. Wait for the run to become in_progress before canceling, but do not wait for GitHub to finish processing an accepted cancellation request.

After pushing a new workflow file, use wait-for-workflow instead of a fixed sleep before invoking or inspecting it. Use wait-for-run to allow up to one minute for a triggered run to appear. After gh workflow run, the helper uses the run URL returned by GitHub.com or a compatible GitHub Enterprise Server and only polls when no URL is available. If that deadline expires, the helper logs the run filters, local and remote refs, workflow files, recent runs, commit check suites, and an Actions API request ID before the repository is cleaned up.

The following custom commands are defined within acceptance_test.go to help with writing tests:

• fixture-repo: select the script's repository fixture mode. For shared and isolated, the final argument names the environment variable that receives the repository's bare name.

  fixture-repo shared REPO
  exec gh issue create --repo $ORG/$REPO --title $SCRIPT_NAME-$RANDOM_STRING --body Body
  

• cleanup-repo: idempotently delete an unmanaged repository during deferred cleanup. Use this when a lifecycle test may have already deleted or renamed the repository.

  defer cleanup-repo $SCRIPT_NAME-$RANDOM_STRING
  

• wait-for-workflow: poll until GitHub registers a pushed workflow definition.

  wait-for-workflow 'Test Workflow Name'
  exec gh workflow run 'Test Workflow Name'
  

• wait-for-repository-ready: poll until an initialized repository's default branch commit is available. Use it before repository-global operations that can conflict with asynchronous repository initialization. Repository rename is a narrow exception: GitHub can retain its creation-operation lock after the commit becomes readable, so keep the gh repo rename command inline and allow a 10-second stabilization delay after this check.

  wait-for-repository-ready $ORG/$REPO
  

• wait-for-run-status: poll a registered workflow run until it reaches the requested status. Use this before operations such as cancellation that can race with run startup.

  wait-for-run RUN_ID
  wait-for-run-status $RUN_ID in_progress
  exec gh run cancel $RUN_ID
  

• defer: register a command to run after the testscript completes

  # Defer repo cleanup
  defer gh repo delete --yes $ORG/$SCRIPT_NAME-$RANDOM_STRING
  

• env2upper: set environment variable to the uppercase version of another environment variable

  # Prepare organization secret, GitHub Actions uppercases secret names
  env2upper ORG_SECRET_NAME=$RANDOM_STRING
  

• replace: replace placeholders in file with interpolated content provided

  env2upper SECRET_NAME=$SCRIPT_NAME_$RANDOM_STRING
  
  # Modify workflow file to use generated organization secret name
  mv ../workflow.yml .github/workflows/workflow.yml
  replace .github/workflows/workflow.yml SECRET_NAME=$SECRET_NAME
  
  -- workflow.yml --
  on:
    workflow_dispatch:
  env:
    ORG_SECRET: ${{ secrets.$SECRET_NAME }}
  

• stdout2env: set environment variable containing standard output from previous command

  # Create the PR
  exec gh pr create --title 'Feature Title' --body 'Feature Body' --assignee '@me' --label 'bug'
  stdout2env PR_URL
  

• wait-for-run: poll for a workflow run until it registers, then set an environment variable to its database ID. Pass gh run list filter flags after the variable name.

  wait-for-run RUN_ID --branch $WORKFLOW_BRANCH --event push
  

• jq-assert: evaluate a jq expression on a JSON environment variable and assert the result matches a regexp

  jq-assert ISSUE_JSON '.title' 'Expected Title'
  jq-assert DISCUSSION_JSON '.comments | length' '^2$'
  

• jq2env: evaluate a jq expression on a JSON environment variable and store the result in another environment variable

  jq2env ISSUE_JSON '.title' ISSUE_TITLE
  

Due to the //go:build acceptance build constraint, some functionality is limited because gopls isn't being informed about the tag. To resolve this, set the following in your settings.json:

  "gopls": {
    "buildFlags": [
        "-tags=acceptance"
    ]
  },

You can install the txtar or vscode-testscript extensions to get syntax highlighting.

When tests fail they fail like this:

➜ go test -tags=acceptance ./acceptance
--- FAIL: TestAcceptance (0.00s)
    --- FAIL: TestAcceptance/pr (0.00s)
        --- FAIL: TestAcceptance/pr/pr-merge (11.07s)
        testscript.go:584: WORK=/private/var/folders/45/sdnm1hp10nj1s9q57dp3bc5h0000gn/T/go-test-script2778137936/script-pr-merge
            # Use gh as a credential helper (0.693s)
            # Create a repository with a file so it has a default branch (1.155s)
            # Defer repo cleanup (0.000s)
            # Clone the repo (1.551s)
            # Prepare a branch to PR with a single file (1.168s)
            # Create the PR (1.903s)
            # Check that the file doesn't exist on the main branch (0.059s)
            # Merge the PR (2.426s)
            # Check that the state of the PR is now merged (0.571s)
            # Pull and check the file exists on the main branch (1.074s)
            # And check we had a merge commit (0.462s)
            > exec git show HEAD
            [stdout]
            commit 85d32c1a83ace270f6754c61f3f7e14956be0a47
            Author: William Martin <williammartin@william-github-laptop.kpn>
            Date:   Fri Oct 11 15:23:56 2024 +0200

                Add file.txt

            diff --git a/file.txt b/file.txt
            new file mode 100644
            index 0000000..7449899
            --- /dev/null
            +++ b/file.txt
            @@ -0,0 +1 @@
            +Unimportant contents
            > stdout 'Merge pull request #1'
            FAIL: testdata/pr/pr-merge.txtar:42: no match for `Merge pull request #1` found in stdout

This is generally enough information to understand why a test has failed. However, we can get more information by providing the -v flag to go test, which turns on verbose mode and shows each command and any associated stdio.

By default testscript removes the directory in which it was running the script, and if you've been a conscientious engineer, you should be cleaning up resources using the defer statement. However, this can be an impediment to debugging. As such you can set GH_ACCEPTANCE_PRESERVE_WORK_DIR=true and GH_ACCEPTANCE_SKIP_DEFER=true to skip these cleanup steps.

This section is to be expanded over time as we write more tests and learn more.

The testscript library creates a somewhat isolated environment for each script. Each script gets a directory with limited environment variables by default. As far as reasonable, we should look to write scripts that depend on nothing more than themselves, the GitHub resources they manage, and limited additional environmental injection from our own testscript setup.

Here are some guidelines around test isolation:

• Favour duplication in test setup over abstracting a new testscript command
• Favour a testscript owning an entire resource lifecycle over shared resource until we see a performance or rate limiting issue
• Use the RANDOM_STRING env var for globally visible resources to avoid conflicts

Since these scripts are creating resources on a GitHub instance, we should try our best to cleanup after them. Use the defer keyword to ensure a command runs at the end of a test even in the case of failure.

TODO: I believe tests should early exit if the correct scopes aren't in place to execute the entire lifecycle. It's extremely annoying if a defer fails to clean up resources because there's no delete_repo scope for example. However, I'm not sure yet whether this scope checking should be in the Go tests or in the scripts themselves. It seems very cool to understand required scopes for a script just by looking at the script itself.

https://bitfieldconsulting.com/posts/test-scripts

https://atlasgo.io/blog/2024/09/09/how-go-tests-go-test

https://encore.dev/blog/testscript-hidden-testing-gem