Skip to content

CI: run codegen checks and E2E on macOS and Windows - #50

Merged
devin-ai-integration[bot] merged 10 commits into
mainfrom
devin/1793743129-cross-platform-ci
Nov 3, 2026
Merged

devin-ai-integration[bot] merged 10 commits into
mainfrom
devin/1793743129-cross-platform-ci

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

CI now runs the codegen checks and every E2E suite on macOS and Windows as well as Linux. A new reusable workflow, .github/workflows/cross-platform.yml, is called from build.yml after build-wasm. It downloads the plugin.wasm built on Linux (the artifact a release publishes) and runs that exact file through sqlc on each OS.

Matrix (OS × Ruby × DB)

OS Ruby PostgreSQL (pg) MySQL (mysql2) SQLite (sqlite3)
Linux (tests.yml, unchanged) 3.1, 3.2, 3.3, 3.4, 4.0 (+ module-name / no-prepared variants) 16, service container 8.3, service container ✓
macOS (macos-latest) 3.1, 4.0 16 (ankane/setup-postgres) 8.4 (ankane/setup-mysql) ✓
Windows (windows-latest) 3.1, 4.0 17 (ankane/setup-postgres; the only version on windows-2025) 8.4, preinstalled on the image ✓

Every macOS/Windows × Ruby cell runs two jobs in parallel:

  • Codegen: unit + golden tests on that OS's Ruby, then sqlc diff with the Linux-built WASM and again with the process plugin (exe\plugin.cmd on Windows), plus the WASM codegen check for that Ruby version.
  • End-to-End: generates the examples with the WASM and runs the full SQLite, PostgreSQL and MySQL suites against native servers using the native pg / mysql2 / sqlite3 gems, then the generator-options E2E.

Why only Ruby 3.1 and 4.0 on macOS/Windows: they are the oldest and newest supported versions. 3.1 is the Struct code path and has the old Bundler/default gems; 4.0 is the Data path with the newest stdlib/bundled-gem split. Ruby-version-specific behaviour is already covered on every version on Linux. What macOS/Windows add is OS behaviour (paths, line endings, native gems, client libraries, the fiber scheduler), and the two ends of the range cover that.

The setup actions are pinned by SHA (ankane/setup-postgres@b1a598d…, ankane/setup-mysql@4aa1b08…, sqlc-dev/setup-sqlc@bac53b7…). MySQL gets SET GLOBAL local_infile = 1 for :copyfrom.

Hard limits

Every job in build.yml, tests.yml, cross-platform.yml, docs.yml and release.yml now has timeout-minutes: 10. The slowest job is currently 3m49s.

CI duration

wall clock (first job start → last job end) slowest jobs
main (60a6d90) ~1m45s Linux MySQL E2E Ruby 3.2: 72s
this PR, first version (one job per OS × Ruby) 7m31s Windows Ruby 4.0: 6m18s, Windows Ruby 3.1: 5m31s
this PR, now (gem cache hit) 4m19s E2E Windows Ruby 4.0: 3m49s, E2E Windows Ruby 3.1: 3m09s, E2E macOS 3.1/4.0: ~2m, Codegen Windows: ~1m55s

These changes cut the first version's time:

  • Codegen and E2E run as separate parallel jobs. The unit/golden tests and sqlc diff (~1 min on Windows) are off the E2E critical path, and the Codegen jobs need no database setup.
  • The E2E gems are cached (actions/cache, keyed on runner image + Ruby + Gemfile.e2e). On Windows, prism, mysql2 and date compile from source, which took ~2 min per job. The key includes the image version because the native extensions link against the image's libraries.

The remaining Windows time goes mostly to the MySQL suite (~1 min: mysql2 queries don't overlap there, see below), PostgreSQL/MySQL/Ruby setup (~1.5 min) and the generator-options E2E. Linux tests.yml is unchanged (~1.5 min) and runs in parallel.

OS-specific bugs found

  1. Fixed: mysql2 :copyfrom corrupted rows containing Ctrl-Z (0x1A) on Windows. The client library reads the LOAD DATA LOCAL INFILE temp file in text mode on Windows, where a raw Ctrl-Z ends the file. The generated csv_line now writes it as \Z, which MySQL decodes back to 0x1A on every OS. examples/mysql2 and the golden files are regenerated, and test_seeded_random_copyfrom_round_trip_and_duplicate_is_atomic got a Ctrl-Z row. Before the fix, the seeded random binary payloads failed on Windows.
  2. Fixed: the process plugin could not run on Windows. exe/plugin.rb cannot be executed directly there. The new exe/plugin.cmd (@ruby "%~dp0plugin.rb" %*, CRLF) wraps it. It must be referenced as exe\plugin.cmd because sqlc runs it through cmd.exe, which parses exe/plugin.cmd as a switch.
  3. Documented, not fixable in the plugin: mysql2 queries don't run concurrently under Sync on Windows. The pooled queries ran serially there (1.58s concurrent vs 1.55s serial). The mysql2 C extension blocks the thread on Windows instead of yielding to the fiber scheduler. Results are still asserted on Windows; only the overlap assertion runs off Windows. The README's async note now says so.
  4. Test-suite fixes (no generated-code change):
    • The SQLite E2E sometimes deadlocked on Ruby 3.1, on Linux too; this was the long-running job. A database left to the GC could be finalized on another thread while a timed-out SQL sleep held its mutex. The test now closes it in teardown.
    • tests/rbs_types.rb hit File.read → EBADF under the fiber scheduler on Windows Ruby 3.1. It now reads the .rbs files before queries run.
    • Gemfile.e2e needs logger and bigdecimal on Ruby ≥ 3.4/4.0, where they are no longer default gems.
  5. CI environment issues:
    • ankane/setup-mysql fails on windows-2025 (EPERM renaming over the preinstalled 8.4), so the job starts the image's MySQL instead.
    • Ruby is set up after the databases. Otherwise PostgreSQL's bundled zlib/iconv DLLs shadow the MSYS2 toolchain's and gcc fails silently.
    • On Windows, mysql2 is built against MSYS2's libmariadbclient with --with-mysql-config, because its extconf can't find mariadb_config.exe through PATH.
    • On macOS, mysql2 gets --with-opt-dir for Homebrew's openssl/zstd.
    • .gitattributes keeps LF everywhere, so sqlc diff and the golden tests stay byte-for-byte equal on Windows checkouts.
    • scripts/wasm/update_sha.sh falls back to shasum -a 256.

docs/Testing.md and CLAUDE.md describe the new jobs. The README changed only for the mysql2/Windows async note; it makes no other platform claims.

Checklist

  • examples/ is regenerated (rake sqlc_generate_process) and committed
  • Tests added or updated (rake test, and rake end2end_* when the generated code changes)
  • The generated code stays RuboCop-clean (rake rubocop) and Steep-clean (rake steep)
  • Labelled (bug, enhancement, documentation, or breaking) so the release notes are grouped correctly

…ilt WASM

Co-Authored-By: Doron Eli Rachman <doron050@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration devin-ai-integration Bot added the enhancement New feature or request label Nov 3, 2026
devin-ai-integration Bot and others added 9 commits November 3, 2026 22:15
…gger in Gemfile.e2e for Ruby 4.0

Co-Authored-By: Doron Eli Rachman <doron050@gmail.com>
Co-Authored-By: Doron Eli Rachman <doron050@gmail.com>
…eadlock); Windows plugin.cmd path with backslashes

Co-Authored-By: Doron Eli Rachman <doron050@gmail.com>
…n wins on PATH; document the new coverage

Co-Authored-By: Doron Eli Rachman <doron050@gmail.com>
…one on Windows); E2E: read .rbs files before fibers run, no fiber overlap expected from mysql2 on Windows; Windows mysql2 build points at mariadb_config

Co-Authored-By: Doron Eli Rachman <doron050@gmail.com>
…e E2E gems per runner image

Co-Authored-By: Doron Eli Rachman <doron050@gmail.com>
…nested bundle install on Windows Ruby 3.1)

Co-Authored-By: Doron Eli Rachman <doron050@gmail.com>
@devin-ai-integration
devin-ai-integration Bot merged commit 8e87ac5 into main Nov 3, 2026
57 checks passed
@devin-ai-integration
devin-ai-integration Bot deleted the devin/1793743129-cross-platform-ci branch November 3, 2026 23:24
devin-ai-integration Bot added a commit that referenced this pull request Nov 4, 2026
…ransactions and late PostgreSQL enum arrays (#53)

## Summary
This fixes the five bugs found by the latest independent pre-release
check. Each bug has a new E2E test in `tests/end2end_*.rb`, and each of
those tests fails on `main` until its fix is applied.

| Bug | Test | Fix |
|---|---|---|
| With a table `exceptions`, a raise inside `transaction` committed
instead of rolling back, because the generated `Exception` model
shadowed `::Exception` (all engines) |
`test_models_named_like_core_constants_do_not_shadow_them` (pg, mysql2,
sqlite3) | Generated Ruby and RBS now refer to core and driver constants
with a leading `::` (`::Exception`, `::Thread`, `::Data`, `::Struct`,
`::Set`, `::Monitor`, `::PG`, `::ConnectionPool`, ...), in the driver
templates and in `TypeResolver#driver_type` |
| Other table names (`data`, `structs`, `threads`, `processes`,
`standard_errors`, `sets`, `strings`, `integers`, `times`, `encodings`,
`monitors`, `mutexes`, `connection_pools`) broke loading, transactions
or the logger | same test. The example schemas now include 14 tables
with these names | same fix |
| SQLite `:execresult` with `RETURNING` never ran the statement |
`test_execresult_with_returning_runs_the_statement` (sqlite3), new
`CreateTagResult` query | `Sqlite3#exec_result_lines` now runs the
statement to the end (`.to_a`) when it returns columns |
| A `tx` used from an `Enumerator`, `Fiber`, `Thread` or `Async` task
ran its queries on another pooled connection, outside the transaction
(pg, mysql2). SQLite raised `ThreadError` or hung |
`test_transaction_query_object_works_from_other_fibers_and_threads` (pg,
mysql2, sqlite3) | `transaction` now yields a copy of the query object
bound to a `TransactionConnection`. It runs queries from any fiber or
thread one at a time (mutex), unlocks while a nested savepoint block
runs, and raises `TransactionEnded` once the transaction is over. The
README documents that other query objects used from other fibers still
run outside the transaction |
| PostgreSQL enum arrays came back as a raw `"{a,b}"` string when the
enum was created after the connection was configured |
`test_enum_arrays_of_a_type_created_after_connecting_are_decoded` (pg,
`late` schema) | Unknown result oids now fall back to a `NewTypes`
(`PG::TypeMapInRuby`) map. It looks each oid up in `pg_type` once per
connection, so arrays of the type become `Array[String]` |

Behavior change: the `tx` yielded by `transaction` is no longer `self`.
Code that keeps `tx` after the block now gets `TransactionEnded` instead
of silently running outside the transaction.

Also in this PR:
- `tests/rbs_types.rb` strips the leading `::` from RBS types before
checking them.
- Examples, golden files, `test/fixtures/*.bin` and
`steep_expectations.yml` are regenerated.
- `README.md`, `docs/Coverage.md` and `docs/Testing.md` are updated.
- `main` is merged in, including the cross-platform CI from #50.

## Checklist
- [x] `examples/` is regenerated (`rake sqlc_generate_process`) and
committed
- [x] Tests added or updated (`rake test`, and `rake end2end_*` when the
generated code changes)
- [x] The generated code stays RuboCop-clean (`rake rubocop`) and
Steep-clean (`rake steep`)
- [x] Labelled (`bug`, `enhancement`, `documentation`, or `breaking`) so
the release notes are grouped correctly

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Doron Eli Rachman <doron050@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant