Repository navigation
ci: Smoke test on a gcloud VM with a 1 hour TTL, a retry, Jaeger, and the executor - #1477
Closed
marcleblanc2 wants to merge 26 commits into
Closed
marcleblanc2 wants to merge 26 commits into
marcleblanc2 wants to merge 26 commits into
Conversation
Skip waiting on the VM deletion, poll container health instead of sleeping 90 seconds, pull images in parallel, install fewer Vagrant plugins, and skip the smoke test on docs-only builds. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c731-7931-7091-9b5e-664cc2379c79 Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1c731-7931-7091-9b5e-664cc2379c79 Co-authored-by: Amp <amp@ampcode.com>
Measured on build 16813: image pulls took 90s, against 93s on pd-standard in build 16811, so the disk is not what limits them. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c731-7931-7091-9b5e-664cc2379c79 Co-authored-by: Amp <amp@ampcode.com>
if_changed compares a build against origin/main. For a commit which is already on main, and is not the newest, that comparison is empty, so the smoke test was skipped. Only apply if_changed on other branches. The health check wait is shorter than the sleep it replaced, so a crash-looping container was more likely to be seen as running. Fail the test if any container has restarted. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c731-7931-7091-9b5e-664cc2379c79 Co-authored-by: Amp <amp@ampcode.com>
The Buildkite pipeline's default branch is still "master", so pipeline.default_branch never matched a build on main. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c731-7931-7091-9b5e-664cc2379c79 Co-authored-by: Amp <amp@ampcode.com>
…ale VMs A build cancelled while GCP was still creating the VM leaked it, because Vagrant only records the VM once the creation has finished, so the cleanup found nothing to delete. The VM now carries its Buildkite job ID as a label, and the cleanup deletes by that label. As a backstop for jobs which die before their cleanup runs, every job deletes labelled VMs older than an hour. The step times out after 30 minutes, so no running job owns a VM that old. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c731-7931-7091-9b5e-664cc2379c79 Co-authored-by: Amp <amp@ampcode.com>
Vagrant creates the boot disk before the VM, and the disk carries no labels. A build cancelled between the two left the disk behind, and the label lookup from the previous commit could not find it. sourcegraph-ci holds three such disks, from September. The VM and its disk are now named docker-test-job-<Buildkite job ID>. The cleanup and the stale sweep delete VMs by that name, and disks by that name which no VM uses. This replaces the label. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c731-7931-7091-9b5e-664cc2379c79 Co-authored-by: Amp <amp@ampcode.com>
- db-only-migrate.docker-compose.yaml now extends the pgsql and codeintel-db services in docker-compose.yaml, instead of copying them - CI runs `docker compose config` on every Compose file, combined the way its docs say to run it - executor.docker-compose.yaml declares the sourcegraph network, so the standalone command in its README works on Compose v2 - dev/run.sh uses `docker compose`, like the README Amp-Thread-ID: https://ampcode.com/threads/T-01a1c724-4acc-77b0-a43d-b94f22933b8a Co-authored-by: Amp <amp@ampcode.com>
- The smoke test compares running containers against the services in docker-compose.yaml, so adding or removing a service needs no test edit - Remove the loop which claimed to watch containers for 5 minutes, but checked them twice, one second apart - Run test/executor-smoke-test.sh after the smoke test Amp-Thread-ID: https://ampcode.com/threads/T-01a1c724-4acc-77b0-a43d-b94f22933b8a Co-authored-by: Amp <amp@ampcode.com>
… VM profile - Each job fetches the builder key into its own temporary file, and removes it when the job ends - The Docker Hub credentials reach the smoke test as provisioner environment variables, instead of being appended to /root/.profile Amp-Thread-ID: https://ampcode.com/threads/T-01a1c724-4acc-77b0-a43d-b94f22933b8a Co-authored-by: Amp <amp@ampcode.com>
Same block as release.yaml: a retried finalize finds its own tag, and only a tag on a different commit is an error. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c724-4acc-77b0-a43d-b94f22933b8a Co-authored-by: Amp <amp@ampcode.com>
Amp-Thread-ID: https://ampcode.com/threads/T-01a1c724-4acc-77b0-a43d-b94f22933b8a Co-authored-by: Amp <amp@ampcode.com>
The VM image lacks the signing key for its Chrome apt repository, so a full `apt-get update` fails. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c724-4acc-77b0-a43d-b94f22933b8a Co-authored-by: Amp <amp@ampcode.com>
Vagrant came in 2020 so one Vagrantfile could run the smoke test in a local VirtualBox VM or on GCE. Only the GCE provider is left, so Vagrant now only adds Ruby, the vagrant-google plugin, a pinned Vagrant on the agents, and an rsync of the repo. smoke-test-vm.sh replaces vagrant-run.sh, the Vagrantfile, and servers.yaml. It creates the VM with the same settings, authorizes a key made for the job, copies the repo over ssh, and runs the steps servers.yaml listed as root, sending the Docker Hub password over stdin. The boot disk grows from vagrant-google's default 10 GB to 20 GB, which the Ubuntu 26.04 e2e image will need for Docker 29's containerd image store. Co-authored-by: Amp <amp@ampcode.com> Amp-Thread-ID: https://ampcode.com/threads/T-01a1c770-8d7e-76ad-b47f-2a2ff80873cc
The smoke test started only docker-compose.yaml, then started a second, separate executor with `docker compose run`. Jaeger never ran. Start all three Compose files with one `up`, so the container count, health check wait, and restart check cover every service. The executor exits without a password and a queue name, so a test override file sets them, and executor-smoke-test.sh reads the password from the container. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c731-7931-7091-9b5e-664cc2379c79 Co-authored-by: Amp <amp@ampcode.com>
…er an hour A VM which stops answering SSH did not fail the tests, so the script deletes it and runs the tests once more on a new VM. A VM which still answers ran the tests, and they failed, so the script copies its logs and does not retry. Each VM is created with --max-run-duration=1h and --instance-termination-action=DELETE, so GCP deletes the VM and its boot disk even if no job is left to do it. The hourly sweep stays for disks whose VM was never created. The script works with both the current Ubuntu 20.04 image and the Ubuntu 26.04 image from sourcegraph/infrastructure#7627, which ships jq. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c7a5-6b65-723b-980c-b85e1e9008b5 Co-authored-by: Amp <amp@ampcode.com>
…e Compose plugin if the VM has it The old trap only saved the logs of running containers whose status was not Up, so it saved nothing for an exited or unhealthy container. The Ubuntu 26.04 VM image has the Compose plugin instead of the standalone docker-compose 1.29.2 binary. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c7a5-6b65-723b-980c-b85e1e9008b5 Co-authored-by: Amp <amp@ampcode.com>
The file described the repository as of June. Since then the Go tooling, Checkov, shellcheck, TESTING.md, tools/, and nginx/ were removed. - Drop statements about files and tools that no longer exist - Prettier skips Markdown and YAML; say so - The smoke test reads CI secrets and creates a GCP VM; point agents at the pull request's Buildkite build instead of running it - Add the rules an agent cannot infer: keep db-only-migrate in sync, update expect_containers, leave image tags to the release process, backport labels, and the branches the release tooling owns
Customers who clone this repository check out the release tag; only customers who fork it merge the tag into their own branch.
The db-only file now extends the base file, CI validates every compose file, runs shellcheck and the executor test, counts the services itself, and runs the smoke test VM with gcloud instead of Vagrant. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c7a5-6b65-723b-980c-b85e1e9008b5 Co-authored-by: Amp <amp@ampcode.com>
… fallback The Ubuntu 20.04 VM image has Docker Compose v2.18.1 as a plugin (build 16848), so the fallback to the standalone docker-compose never ran. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c7a5-6b65-723b-980c-b85e1e9008b5 Co-authored-by: Amp <amp@ampcode.com>
Replaces the polling loop. The Ubuntu 20.04 VM image's Compose v2.18.1 has --wait and --wait-timeout, so this does not need the new image. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c7a5-6b65-723b-980c-b85e1e9008b5 Co-authored-by: Amp <amp@ampcode.com>
This reverts commit 869abd2. Build 16853: `up --wait` failed with "container searcher-0 is unhealthy" while `docker ps` showed it healthy in the same second. searcher has no health check start period, so it reports unhealthy until the frontend is up, and --wait fails on the first unhealthy report. The polling loop tolerates that. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c7a5-6b65-723b-980c-b85e1e9008b5 Co-authored-by: Amp <amp@ampcode.com>
Build 16854 failed when Docker Hub's token endpoint timed out during the pull. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c7a5-6b65-723b-980c-b85e1e9008b5 Co-authored-by: Amp <amp@ampcode.com>
Resolves the .tool-versions conflict with #1467: main's versions, plus shellcheck. Amp-Thread-ID: https://ampcode.com/threads/T-01a1c7a5-6b65-723b-980c-b85e1e9008b5 Co-authored-by: Amp <amp@ampcode.com>
Contributor
Author
|
Closing in favor of the stack at https://github.com/sourcegraph/deploy-sourcegraph-docker/stack/1491, which splits this into seven PRs: #1484, #1485, #1486, #1487, #1488, #1489, #1490. Build 16857 on this branch showed the pieces work together. One thing differs: this branch retried the smoke test on a new VM, and #1488 only retries the SSH connection on the same VM. The branch stays. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
One pull request for the CI work spread across #1468, #1471, #1472, #1474, #1475, and the
AGENTS.mdfrom #1396. It replaces all six. Nothing here changes what customers deploy, except the two Compose file fixes listed under "Compose files".Smoke test VM
.buildkite/smoke-test-vm.shcreates the VM withgcloud, and copies the repository and runs the tests overssh.vagrant-run.sh,test/Vagrantfile, andtest/servers.yamlare gone.--max-run-duration=1h --instance-termination-action=DELETE, so GCP deletes the VM and its boot disk even if the job dies. GCP has no TTL for a disk whose VM was never created, so each job also deletesdocker-test-job-*VMs and unattached disks older than an hour. The step times out after 30 minutes.docker composeplugin; the script installs jq only where it is missing. Switching is a one-line change toIMAGEonce that image is built.Smoke test
jaeger/,executors/, andtest/docker-compose.override.yaml, checks that Jaeger responds, then runstest/executor-smoke-test.sh, which checks that the executor registers.mainalways runs it.Other checks
validate-docker-compose.shvalidates every Compose file, in each combination the docs say to run..tool-versions).batch-change/release.yaml: the tag step is safe to retry.Compose files
db-only-migrate.docker-compose.yamlnowextendspgsqlandcodeintel-dbfrom the base file, so the two can no longer drift.sg ops update-imagesskips services without animage:, so the release tooling is unaffected.executors/executor.docker-compose.yamldeclares thesourcegraphnetwork, so it is valid on its own.Decisions
retry:rule, so a real test failure is never rerun.gcloud beta compute instances create, because the agents' gcloud 417 only has--max-run-durationin beta (GA in 483).docker compose up --waitdoes not replace the polling loop: in build 16853 it failed with "container searcher-0 is unhealthy" on a stack which was healthy seconds later.searcherhas no health check start period, so it reports unhealthy until the frontend is up.vagrant, and the agents keep Vagrant: release branches up to8.1.xstill use it.Not in this pull request
master. It is a pipeline setting, not in any repository, so it has to be changed in Buildkite.pipeline.yamlnamesmainand works with either value.Checklist
Test plan
maxRunDuration3600 seconds,instanceTerminationAction: DELETE, and a 20 GB boot disk.gcloudandssh: tests pass, tests fail (no retry, logs copied), VM dies during the tests (retry passes), VM creation fails (retry passes), both VMs die (fails), VM dies then tests fail (fails, logs copied).docker.