Conversation
Test databases are cloned from a template, are tiny, and have no planner statistics, so cost estimates for complex queries are wildly inflated and cross jit_above_cost. PostgreSQL then LLVM-compiles the query: GetUserStatusCounts takes 318ms with JIT and 0.3ms without on a test database. Disable JIT in every place the test PostgreSQL is configured (Makefile container, embedded PostgreSQL on macOS/Windows, dbtestutil's fallback container). Request STRATEGY = FILE_COPY when cloning the template on PostgreSQL 15 or newer; the WAL_LOG default roughly doubles per-database CPU for the thousands of short-lived, fsync=off databases the suite creates. Run TestWorkspaceTagsTerraform subtests in parallel. Each subtest uses its own coderd, provisioner daemon, and plugin cache, and only the read-only provider mirror is shared. The test dominated the enterprise/coderd package, which is the last package to finish.
…allelize tests TestWorkspaceTagsTerraform and TestWorkspaceTemplateParamsChange used t.Setenv(TF_CLI_CONFIG_FILE), which forced both to run serially and made them the ~80s tail of the enterprise/coderd package. Thread the path through terraform.ServeOptions.CliConfigPath via a new coderdenttest option so both tests and their subtests can run in parallel.
… first testing.T removes finished cleanups by reslicing, which leaves each closure in the backing array, and top-level tests stay reachable until the binary exits. Helper cleanups that captured a coderd.API, its router, NATS server, provisioner daemon, agent, or aibridged daemon therefore kept every test server alive for the whole package run. Add testutil.Cleanup, which drops its reference to the closure once it has run, and use it for those helpers. In the coderd package this cuts APIs still reachable at the end of the run from 469/1607 to 9/1607, peak RSS from ~7.0GB to ~1.2GB, and end-of-run live heap from ~4GB to well under 1GB, which shrinks every GC cycle. Also close the tailnet resume signing keycache created by coderd.New in API.Close so its refresh timer does not outlive the API, and list the slowest packages first in make test so enterprise/coderd no longer starts near the end of the run and finishes last.
kylecarbs
changed the title
Locally on the coderd package GOGC=200 cuts Go user CPU 14% (90s to 77s) for a peak RSS of 2.1GB instead of 1.2GB; GOGC=400 saves 18% but needs 5.7GB. Limited to Linux since the macOS runners are memory constrained.
…erge)" This reverts commit eee336e.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Profiling of
test-go-pgfound that the suite is CPU-bound on the 8-core runner, split roughly 50/50 between the Go test binaries and PostgreSQL, that a large share of the PostgreSQL half is spent LLVM JIT-compiling queries on tiny, statistics-less test databases, and that thecoderdtest binary retains every test server it ever started until the process exits. This PR disables JIT for the test PostgreSQL, usesFILE_COPYwhen cloning template databases on PostgreSQL 15+, releases the cleanup closures that retained finished test servers, runs the slowest packages first, and parallelizes the two Terraform tests that formed theenterprise/coderdtail.Draft, to measure the effect in CI. No tests are removed or skipped.
CI measurements
Runner variance is large, so the numbers below are the
DONE ... in Xsline from gotestsum on uncached (-count=1) Linux runs.Both PR samples are below every main sample. Per-package effects that are visible regardless of noise:
coderd/database172-328s -> 47-77s, and the post-enterprise/coderdtail is gone (last packages to finish are now small ones).Problem
PostgreSQL JIT
pg_stat_statementson thecoderd/databasepackage attributed 354s of 469s total statement time toGetUserStatusCounts(756 calls, ~470ms each) on nearly empty databases.EXPLAIN ANALYZEshows why: the planner estimates 5.7M rows where 61 exist, the cost crossesjit_above_cost/jit_optimize_above_cost, and PostgreSQL runs LLVM inlining and optimization for every execution:317.7mswithjit=onvs0.31mswithjit=off. Any complex query on a fresh test database is exposed to this.Locally, with
go testand PostgreSQL 13 pinned to 8 CPUs,jit=offmoved the full suite from 317s to 286s wall and cut PostgreSQL CPU from 1155s to 902s (-22%);coderd/databasewent from 152s to 40s. In CI thecoderd/databasepackage went from 172-328s on recentmainruns to 77s (PG13) and 43s (PG17).On PostgreSQL 15+, the default
CREATE DATABASEstrategy isWAL_LOG, which roughly doubles per-clone CPU compared toFILE_COPYfor the ~9100 short-lived databases the suite creates (52.7ms vs 27.8ms per create+connect+drop withshared_buffers=2GB). This is part of whytest-go-pg-17was slower than the PG13 job.Retained test servers
testing.Tremoves finished cleanups by reslicingc.cleanups, which leaves each closure in the backing array, and every top-level test stays reachable from the test binary's root until it exits. Helper cleanups incoderdtest,coderdenttest,agenttest, andaibridgedtestcaptured thecoderd.API(directly or via a dial closure), its chi router, NATS server, provisioner daemon, or agent. Weak-pointer tracking incoderdtest.NewWithAPIshowed 469 of 1607 APIs still reachable at the end of thecoderdpackage run, with live heap climbing from 17MB to ~4GB and GC at ~20% of Go CPU.enterprise/coderdtailenterprise/coderdran alone for the final ~75s becauseTestWorkspaceTagsTerraform(67s) andTestWorkspaceTemplateParamsChange(14s) usedt.Setenv(TF_CLI_CONFIG_FILE), which forces them to run serially before the package's parallel tests start.go test ./...also runs packages alphabetically, soenterprise/coderdonly started ~330s into the run.Fix
-c jit=offinmake test-postgres-docker,ALTER SYSTEM SET jit = 'off'inscripts/embedded-pg/main.go, and-c jit=offindbtestutil.openContainer.dbtestutil.createDatabaseSQLappendsSTRATEGY = FILE_COPYwhenserver_version_num >= 150000.testutil.Cleanup(t, f)registers a holder that drops its reference tofonce it has run, with a test that fails when swapped for plaint.Cleanup. Used for the heavyweight helper cleanups.coderdpackage: APIs reachable at end of run 469/1607 -> 9/1607, peak RSS ~7.0GB -> ~1.2GB.coderd.Newstores the tailnet resume signing keycache onAPIand closes it inAPI.Closeso its refresh timer does not outlive the API.coderdenttest.WithTerraformCLIConfigPathpassesterraform.ServeOptions.CliConfigPathexplicitly; both Terraform tests dropt.Setenvand run in parallel, including their subtests.make testlists the slowest packages first whenTEST_PACKAGESis./...(go testpreserves argument order and deduplicates). CustomTEST_PACKAGESvalues are unchanged.test-go-pgjobs run withGOGC=200. With retention fixed this is safe: on thecoderdpackage it cuts Go user CPU 14% (90s -> 77s) for a peak RSS of 2.1GB instead of 1.2GB.GOGC=400saves 18% but needs 5.7GB, so it was not chosen.Profiling notes and follow-ups
Method
mainruns. Wall time on the shared runners varies a lot between runs (PG13DONEranged 370-534s across five consecutivemainruns), so per-package times and CPU accounting are the reliable signals.go testand the PostgreSQL container pinned to 8 CPUs (taskset,--cpuset-cpus,GOMAXPROCS=8)./usr/bin/time), PostgreSQL CPU (container cgroupcpu.stat), per-test timing (go test -json), Go CPU and heap profiles, GC trace, and PostgreSQL statement time (pg_stat_statementswith a 500k entry table, since every test database gets its own entries).CREATE DATABASE ... WITH TEMPLATE/DROP DATABASEon PG13 and PG17 with differentshared_buffersand clone strategies.viewcoreon a core dump (runtime.main -> testing.tRunner -> []*testing.T -> cleanup backing array -> closure -> API), then tracked every API created bycoderdtest.NewWithAPIwithweak.Pointerto attribute the remaining retainers to specific helpers.Where the time goes
-count=1):DONE 31255 tests in 435s, 283 packages, per-package sum 1419s. Longest:coderd227s,coderd/database176s,cli122s,coderd/database/migrations119s,enterprise/coderd105s.GOCACHE(a PR run showed 165 packages(cached)). A fully cold compile of all test binaries on 8 cores costs 3m26s / 26 CPU-minutes.-parallelis not the limit.init()is not a factor: thecoderd/databasebinary used 2.25s CPU over a 33s run.CREATE/DROP DATABASEpairs per run; on PG13 both force an immediate checkpoint that scans all ofshared_buffers(17k requested checkpoints). About 11% of PG CPU. Smallershared_bufferssaves ~25% of that.coderdpackage alone (~9 per test database, ~2-3ms PG CPU each).NewDBopens a dedicated connection just forALTER DATABASE ... SET TIMEZONE.coderd: OPA/rego evaluation ~22% of CPU, GC ~20%,coderdtest.NewWithAPI13.5% (includingaibridge/prices.Seed, which parses a 189KB JSON and upserts ~1000 rows per test server),agentrsa.GenerateDeterministicKey4%.t.Cleanupclosures that captureapi(for examplesetupDynamicParamsTest), which is negligible.Follow-ups
aibridge/pricesJSON once per process instead of percoderd.New.ALTER DATABASEindbtestutil.NewDB.test-go-pgacross runners or a larger runner scales wall time almost linearly at similar total cost.TestGetUserStatusCountscreates ~900 databases; one per timezone would remove most of them.Generated by Coder Agents on behalf of @kylecarbs.