Skip to content

ci: Make the smoke test wait for health checks, and check more - #1487

Open
marcleblanc2 wants to merge 2 commits into
marc/batch-change-release-tag-retryfrom
marc/smoke-test-wait-for-health
Open

marcleblanc2 wants to merge 2 commits into
marc/batch-change-release-tag-retryfrom
marc/smoke-test-wait-for-health

Conversation

@marcleblanc2

@marcleblanc2 marcleblanc2 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Makes the smoke test wait for health checks instead of sleeping, and check more. One change reaches customers: three services get a longer health check start period.

Smoke test

  • Runs docker compose up --wait --wait-timeout 300 instead of sleep 90. It returns once every container is running and healthy, and fails if one reports unhealthy.
  • Pulls images in parallel first, with up to three attempts, because Docker Hub times out now and then.
  • Counts the expected containers from the Compose file instead of a hard-coded 25.
  • Fails if any container restarted.
  • Saves the logs of every container on failure, including the ones that exited.
  • Uses docker compose (v2) instead of docker-compose (v1).
  • Adds these to the smoke test section of AGENTS.md.

What customers will see

searcher-0, precise-code-intel-worker, and syntactic-code-intel-worker get start_period: 300s on their health checks, the same as sourcegraph-frontend-internal. searcher-0 had none, and the two workers had 60 seconds.

None of the three is ready until sourcegraph-frontend-internal is, which waits for the migrator. Until then their health checks fail, so they could show unhealthy during a start, and --wait gives up on the first unhealthy container. With the start period they show starting until their first check passes.

Nothing else changes: Docker does not restart an unhealthy container, and no service waits on the health of these three.

Risk: low. One health check setting on three services; the rest is the test.

Checklist

  • All images have a valid tag and SHA256 sum (no image changes)

Test plan

  • Build 16954 passed, the smoke test job took 404 seconds
  • Without the start period, --wait fails: build 16853 with container searcher-0 is unhealthy, and build 16905 with container precise-code-intel-worker is unhealthy
  • shellcheck test/smoke-test.sh is clean

Closes FE-586

@marcleblanc2
marcleblanc2 added this pull request to stack #1491 November 11, 2026 10:45
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

@marcleblanc2
marcleblanc2 force-pushed the marc/smoke-test-wait-for-health branch from 7298ea5 to d198de1 Compare November 11, 2026 22:23
@marcleblanc2
marcleblanc2 force-pushed the marc/smoke-test-wait-for-health branch from d198de1 to 3290bd2 Compare November 11, 2026 23:46
@marcleblanc2
marcleblanc2 removed this pull request from stack #1491 November 12, 2026 22:27
@marcleblanc2
marcleblanc2 force-pushed the marc/smoke-test-wait-for-health branch from 3e0f620 to 0041077 Compare November 12, 2026 22:27
@marcleblanc2
marcleblanc2 added this pull request to stack #1498 November 12, 2026 22:27
@marcleblanc2
marcleblanc2 force-pushed the marc/smoke-test-wait-for-health branch from 0041077 to e9e20ec Compare November 12, 2026 22:56
@marcleblanc2
marcleblanc2 removed this pull request from stack #1498 November 13, 2026 04:58
@marcleblanc2
marcleblanc2 changed the base branch from marc/batch-change-release-tag-retry to main November 13, 2026 04:58
@marcleblanc2
marcleblanc2 force-pushed the marc/smoke-test-wait-for-health branch from e9e20ec to b37e6b1 Compare November 13, 2026 04:58
@marcleblanc2
marcleblanc2 changed the base branch from main to marc/batch-change-release-tag-retry November 13, 2026 04:58
@marcleblanc2
marcleblanc2 added this pull request to stack #1504 November 13, 2026 04:58
@marcleblanc2
marcleblanc2 marked this pull request as ready for review November 13, 2026 05:36
@marcleblanc2
marcleblanc2 requested a review from a team November 13, 2026 05:36
@marcleblanc2
marcleblanc2 enabled stacked auto-merge (squash) November 13, 2026 05:40
stacked auto-merge was automatically disabled November 13, 2026 07:29

Auto-merge was disabled for another pull request in the stack

@marcleblanc2
marcleblanc2 force-pushed the marc/smoke-test-wait-for-health branch from b37e6b1 to 3b5b662 Compare November 13, 2026 07:40
marcleblanc2 and others added 2 commits November 13, 2026 01:50
Replaces the 90 second sleep with docker compose up --wait, pulls images in parallel with retries, counts the expected containers from the Compose file, fails if a container restarted, and saves every container's logs on failure.

searcher-0 gets a health check start period, like sourcegraph-frontend-internal. Without it, it reports unhealthy while it waits for the frontend, which fails --wait.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1c7a5-6b65-723b-980c-b85e1e9008b5
Co-authored-by: Amp <amp@ampcode.com>
precise-code-intel-worker and syntactic-code-intel-worker are not ready
until sourcegraph-frontend-internal serves them the site configuration,
and that container waits for the migrator. With a 60 second start
period, a slow start marks them unhealthy, and `docker compose up
--wait` fails. Build 16902 failed that way: precise-code-intel-worker
was unhealthy 71 seconds after it started.

searcher-0 got the same start period for the same reason.

Amp-Thread-ID: https://ampcode.com/threads/T-01a1c7a5-6b65-723b-980c-b85e1e9008b5
Co-authored-by: Amp <amp@ampcode.com>
@marcleblanc2
marcleblanc2 force-pushed the marc/smoke-test-wait-for-health branch from 3b5b662 to 3f6e2ae Compare November 13, 2026 07:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant