Skip to content

fix(coderd/telemetry): give the reporter its own HTTP transport - #32160

Open
ThomasK33 wants to merge 8 commits into
mainfrom
tk/cli-telemetry-transport
Open

ThomasK33 wants to merge 8 commits into
mainfrom
tk/cli-telemetry-transport

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Why

A CI run on an Automations PR (#32140) found a flake on main. TestServer_TelemetryDisabled_FinalReport failed once because the server's first telemetry POST returned net/http: HTTP/1.x transport connection broken: http: CloseIdleConnections called. The deployment report errored, no snapshot followed, and the test waited for submitted snapshot until its deadline.

The telemetry reporter used http.DefaultTransport, which every client without its own transport shares. net/http returns the connection of a response without a body, like the telemetry server's 202, to the idle pool before it hands the response to the caller. A CloseIdleConnections call on the same transport inside that gap fails a request that already succeeded. cli/server.go makes that call when a server stops, and httptest.Server.Close makes it too. Many parallel cli tests use both.

What changed

The reporter builds its client on a clone of the default transport, so it has its own connection pool, and Close closes that pool's idle connections. The defer http.DefaultClient.CloseIdleConnections() in cli/server.go stays, because it cleans up the embedded-postgres idle connection that goleak catches (#3246).

Scope

This covers the telemetry reporter, the one client with evidence of the failure. It does not touch other clients of the default transport or the cli/server.go defer.

Tradeoffs

Narrowing the cli/server.go defer to clients the server owns would not remove the cause. httptest.Server.Close and tests that close their codersdk clients (default transport) still close the shared pool's idle connections.

The gap is too narrow to hit by looping: 100 runs of the cli test pass on main and on this branch. A forced reproduction (a PutIdleConn trace hook that closes the default transport's idle connections and holds the response) fails the old reporter with the CI error on every run. The new test TestTelemetry_ReporterOwnsConnectionPool fails on the old code, on a shared client, on a missing clone, on a Close that keeps the pool, and on a Close that releases the pool before its final report.

If a report is still in flight over HTTP/2 when Close runs, its connection stays open until the transport's 90 s idle timeout. Both production callers of Close run at process exit, so this is one socket in an exiting process. Turning off keep-alives would close it at once and give up connection reuse within a report.

Live check: none. The change is limited to the reporter's HTTP client, and the cli and enterprise/cli server tests run it.


Generated with xum • Model: anthropic:claude-sonnet-5-5 • Thinking: xhigh

… transport

The reporter shares http.DefaultTransport. A CloseIdleConnections call on
it while a bodyless 202 response is between the idle pool and the caller
fails the request, so the deployment POST errors and no snapshot follows.
The hook forces that window. Both tests fail until the reporter owns its
transport.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
The reporter used http.DefaultTransport. Any CloseIdleConnections call on
it in the same process, for example cli/server.go on shutdown or
httptest.Server.Close in a parallel test, could fail a telemetry request
whose bodyless 202 response was just returning its connection to the pool.

The reporter now clones the default transport and closes its own idle
connections in Close, which the shared defer in cli/server.go used to do
for it.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
Replaces the first two tests and the isolation test of the previous
commit with one test. It closes the idle connections of the default
transport between two reports, then checks that the reporter reused its
connection, and that Close releases it. It fails for each of: the old
shared client, a clone step that is dropped, and a Close that does not
close the pool.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
@ThomasK33
ThomasK33 temporarily deployed to main November 13, 2026 02:51 — with GitHub Actions Inactive
@ThomasK33
ThomasK33 temporarily deployed to main November 13, 2026 02:51 — with GitHub Actions Inactive
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-13T16:34:51.888251Z 0c28b95 Manual request
🔒 Security Review ✅ Completed 2026-11-13T16:35:54.863148Z 0c28b95 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 396529174b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 396529174b

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…ction

The reporter test only waited for the first connection to close, so a Close
that released the pool before its final report passed. Count the server's
new and closed connections and require them to match after Close.
@ThomasK33
ThomasK33 temporarily deployed to main November 13, 2026 04:47 — with GitHub Actions Inactive
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@ThomasK33
ThomasK33 temporarily deployed to main November 13, 2026 04:47 — with GitHub Actions Inactive
@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: a3d124fe13

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a3d124fe13

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/telemetry/telemetry_test.go Resolved
Comment thread coderd/telemetry/telemetry.go Resolved
Comment thread coderd/telemetry/telemetry.go Resolved
The test clones a database, sends three reports and polls for every
connection to close. Repository testing guidance asks for
testutil.WaitLong.

_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
@ThomasK33
ThomasK33 temporarily deployed to main November 13, 2026 05:17 — with GitHub Actions Inactive
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 089c9dc981

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 089c9dc981

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ThomasK33
ThomasK33 temporarily deployed to main November 13, 2026 06:54 — with GitHub Actions Inactive
@ThomasK33
ThomasK33 temporarily deployed to main November 13, 2026 07:12 — with GitHub Actions Inactive

@ethanndickson ethanndickson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we drop the new test and inline the clone instead of adding another helper? Details inline.

Generated by an agent on behalf of the user

Comment on lines +117 to +123
func newHTTPClient() *http.Client {
transport := &http.Transport{Proxy: http.ProxyFromEnvironment}
if defaultTransport, ok := http.DefaultTransport.(*http.Transport); ok {
transport = defaultTransport.Clone()
}
return &http.Client{Transport: transport}
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have this same clone in a bunch of places (coderdtest.NewIsolatedHTTPClient, newHTTPTransport in cli/root.go, both mcpHTTPClients). Could we skip another helper and just inline it in New? Something like:

if transport, ok := http.DefaultTransport.(*http.Transport); ok {
	reporter.client.Transport = transport.Clone()
}

Generated by an agent on behalf of the user

// deployment report then errors and no snapshot follows. The reporter must
// keep its connection through such a close, and Close must release every
// connection the reporter opened, the one for its final report included.
func TestTelemetry_ReporterOwnsConnectionPool(t *testing.T) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we delete this test? It doesn't reproduce the race: on the old client both reports still succeed and only the connection count differs, so it's really just checking which pool the client uses. The earlier fixes for this flake (#19462, #23494) didn't add one either.

Generated by an agent on behalf of the user

- TestTelemetry_ReporterOwnsConnectionPool: replace the local messageSink and
  waitForMessage helpers with testutil.FakeSink and testutil.Eventually. The
  test keeps the same client-side completion signal, the "submitted snapshot"
  log line, and the same two guards: reuse of the reporter's connection after
  http.DefaultClient.CloseIdleConnections, and release of every connection on
  Close.
- M3, TestTelemetry_ReporterOwnsConnectionPool: wrap the waitSubmitted wait in require.True, so a missed wait fails the test at once and does not run on with an expired context.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
@ThomasK33
ThomasK33 temporarily deployed to main November 13, 2026 16:32 — with GitHub Actions Inactive
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 0c28b953cf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 0c28b953cf

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

This branch was previously deployed

1 inactive deployment
main — 0c28b953 Deployed Nov 13, 2026 by ThomasK33 via cla #256713
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants