Skip to content

build benchmark binaries once for all matrix targets#8010

Open
myrrc wants to merge 1 commit into
developfrom
myrrc/ci-sql-bench-single-build
Open

build benchmark binaries once for all matrix targets#8010
myrrc wants to merge 1 commit into
developfrom
myrrc/ci-sql-bench-single-build

Conversation

@myrrc
Copy link
Copy Markdown
Contributor

@myrrc myrrc commented May 18, 2026

No description provided.

@myrrc myrrc force-pushed the myrrc/ci-sql-bench-single-build branch from 00fb59b to cd78f34 Compare May 18, 2026 16:15
@myrrc myrrc requested a review from joseph-isaacs May 18, 2026 16:15
@myrrc myrrc added action/benchmark Trigger full benchmarks to run on this PR action/benchmark-sql Trigger SQL benchmarks to run on this PR changelog/chore A trivial change and removed changelog/feature A new feature labels May 18, 2026
@github-actions github-actions Bot removed action/benchmark Trigger full benchmarks to run on this PR action/benchmark-sql Trigger SQL benchmarks to run on this PR labels May 18, 2026
@myrrc myrrc added action/benchmark Trigger full benchmarks to run on this PR action/benchmark-sql Trigger SQL benchmarks to run on this PR labels May 18, 2026
@github-actions github-actions Bot removed action/benchmark-sql Trigger SQL benchmarks to run on this PR action/benchmark Trigger full benchmarks to run on this PR labels May 18, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done ff8e9f3 1 Explore Profiling Data
Previous Runs (1)
Status Commit Job Attempt Link
🟢 Done cd78f34 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark FineWeb NVMe failed! Check the workflow run for details.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

Benchmarks: PolarSignals Profiling

Vortex (geomean): 0.963x ➖


datafusion / vortex-file-compressed (0.963x ➖, 1↑ 0↓)
name PR ff8e9f3 (ns) base 52e26d1 (ns) ratio (PR/base)
polarsignals_q00/datafusion:vortex-file-compressed 124666047 131981674 0.94
polarsignals_q01/datafusion:vortex-file-compressed 🚀 277735178 313809466 0.89
polarsignals_q02/datafusion:vortex-file-compressed 23269414 24531810 0.95
polarsignals_q03/datafusion:vortex-file-compressed 277452158 291223103 0.95
polarsignals_q04/datafusion:vortex-file-compressed 11606229 12211292 0.95
polarsignals_q05/datafusion:vortex-file-compressed 15674854 15638717 1.00
polarsignals_q06/datafusion:vortex-file-compressed 18690834 18208202 1.03
polarsignals_q07/datafusion:vortex-file-compressed 13898570 13701477 1.01
polarsignals_q08/datafusion:vortex-file-compressed 411863980 423287466 0.97
polarsignals_q09/datafusion:vortex-file-compressed 10960348 11620293 0.94

@github-actions
Copy link
Copy Markdown
Contributor

File Sizes: PolarSignals Profiling

No file size changes detected.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-H SF=1 on NVME failed! Check the workflow run for details.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-DS SF=1 on NVME failed! Check the workflow run for details.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark FineWeb S3 failed! Check the workflow run for details.

Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
@myrrc myrrc force-pushed the myrrc/ci-sql-bench-single-build branch from cd78f34 to ff8e9f3 Compare May 18, 2026 16:32
@myrrc myrrc added action/benchmark Trigger full benchmarks to run on this PR action/benchmark-sql Trigger SQL benchmarks to run on this PR labels May 18, 2026
@github-actions github-actions Bot removed action/benchmark-sql Trigger SQL benchmarks to run on this PR action/benchmark Trigger full benchmarks to run on this PR labels May 18, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 18, 2026

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-H SF=1 on S3 failed! Check the workflow run for details.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented May 18, 2026

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚡ 1 improved benchmark
❌ 1 regressed benchmark
✅ 1219 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation chunked_varbinview_canonical_into[(100, 100)] 273.3 µs 308 µs -11.27%
Simulation chunked_varbinview_opt_canonical_into[(1000, 10)] 224.8 µs 187.5 µs +19.92%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing myrrc/ci-sql-bench-single-build (ff8e9f3) with develop (faf7e42)

Open in CodSpeed

@github-actions
Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark Statistical and Population Genetics failed! Check the workflow run for details.

@github-actions
Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark Clickbench on NVME failed! Check the workflow run for details.

@github-actions
Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-H SF=10 on NVME failed! Check the workflow run for details.

@github-actions
Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark TPC-H SF=10 on S3 failed! Check the workflow run for details.

@github-actions
Copy link
Copy Markdown
Contributor

Benchmarks: Compression

Vortex (geomean): 1.008x ➖
Parquet (geomean): 1.018x ➖


unknown / unknown (1.024x ➖, 3↑ 10↓)
name PR ff8e9f3 (ns) base faf7e42 (ns) ratio (PR/base)
compress time/Arade 1211605421 1153623134 1.05
compress time/Bimbo 7248948663 6993420633 1.04
compress time/CMSprovider 2907745050 2864467777 1.02
compress time/Euro2016 456450715 418464316 1.09
compress time/Food 421516174 386054689 1.09
compress time/HashTags 866767622 813422295 1.07
compress time/TPC-H l_comment canonical 1351450994 1305012343 1.04
compress time/TPC-H l_comment chunked 1349597120 1298584375 1.04
compress time/taxi 738712569 725160458 1.02
compress time/wide table cols=100 chunks=1 rows=1000 12777535 11624887 1.10
compress time/wide table cols=100 chunks=50 rows=1000 🚀 12033188 13969097 0.86
compress time/wide table cols=1000 chunks=1 rows=1000 134239536 125418539 1.07
compress time/wide table cols=1000 chunks=50 rows=1000 142193250 141337159 1.01
compress time/wide table cols=10000 chunks=1 rows=1000 1540405438 1451520654 1.06
compress time/wide table cols=10000 chunks=50 rows=1000 1527421737 1533517146 1.00
decompress time/Arade 30472321 27897118 1.09
decompress time/Bimbo 84739337 82300198 1.03
decompress time/CMSprovider 🚨 91113788 75799253 1.20
decompress time/Euro2016 🚨 19077493 16949359 1.13
decompress time/Food 🚨 8868444 7497258 1.18
decompress time/HashTags 76447266 71119989 1.07
decompress time/TPC-H l_comment canonical 42870457 39829046 1.08
decompress time/TPC-H l_comment chunked 42585772 40070134 1.06
decompress time/taxi 🚨 17336943 14351538 1.21
decompress time/wide table cols=100 chunks=1 rows=1000 2732568 2542667 1.07
decompress time/wide table cols=100 chunks=50 rows=1000 🚀 2660621 2978869 0.89
decompress time/wide table cols=1000 chunks=1 rows=1000 🚨 26792822 22242829 1.20
decompress time/wide table cols=1000 chunks=50 rows=1000 25680205 25643525 1.00
decompress time/wide table cols=10000 chunks=1 rows=1000 270536178 256133908 1.06
decompress time/wide table cols=10000 chunks=50 rows=1000 270974403 295411231 0.92
parquet size/Arade 258014282 258014282 1.00
parquet size/Bimbo 384517292 384517292 1.00
parquet size/CMSprovider 376885545 376885545 1.00
parquet size/Euro2016 122975499 122975499 1.00
parquet size/Food 35699500 35699500 1.00
parquet size/HashTags 133510943 133510943 1.00
parquet size/TPC-H l_comment canonical 158358238 158358238 1.00
parquet size/TPC-H l_comment chunked 158358238 158358238 1.00
parquet size/taxi 55283635 55283635 1.00
parquet size/wide table cols=100 chunks=1 rows=1000 932404 932404 1.00
parquet size/wide table cols=100 chunks=50 rows=1000 932404 932404 1.00
parquet size/wide table cols=1000 chunks=1 rows=1000 9324004 9324004 1.00
parquet size/wide table cols=1000 chunks=50 rows=1000 9324004 9324004 1.00
parquet size/wide table cols=10000 chunks=1 rows=1000 93240004 93240004 1.00
parquet size/wide table cols=10000 chunks=50 rows=1000 93240004 93240004 1.00
parquet_rs-zstd compress time/Arade 2724742278 2617390548 1.04
parquet_rs-zstd compress time/Bimbo 13037233113 12564329238 1.04
parquet_rs-zstd compress time/CMSprovider 7340056309 6849080929 1.07
parquet_rs-zstd compress time/Euro2016 1367175476 1285188552 1.06
parquet_rs-zstd compress time/Food 801985343 786010387 1.02
parquet_rs-zstd compress time/HashTags 2303712668 2122058875 1.09
parquet_rs-zstd compress time/TPC-H l_comment canonical 3351994624 3167838017 1.06
parquet_rs-zstd compress time/TPC-H l_comment chunked 3353416030 3146067218 1.07
parquet_rs-zstd compress time/taxi 1255976061 1192186355 1.05
parquet_rs-zstd compress time/wide table cols=100 chunks=1 rows=1000 🚨 7523660 6313901 1.19
parquet_rs-zstd compress time/wide table cols=100 chunks=50 rows=1000 7432964 8186762 0.91
parquet_rs-zstd compress time/wide table cols=1000 chunks=1 rows=1000 82262869 75214317 1.09
parquet_rs-zstd compress time/wide table cols=1000 chunks=50 rows=1000 82412014 84265226 0.98
parquet_rs-zstd compress time/wide table cols=10000 chunks=1 rows=1000 828277251 774560739 1.07
parquet_rs-zstd compress time/wide table cols=10000 chunks=50 rows=1000 818913879 859481477 0.95
parquet_rs-zstd decompress time/Arade 639037678 611388256 1.05
parquet_rs-zstd decompress time/Bimbo 1709976992 1695736253 1.01
parquet_rs-zstd decompress time/CMSprovider 1792057361 1709538523 1.05
parquet_rs-zstd decompress time/Euro2016 395436218 372720766 1.06
parquet_rs-zstd decompress time/Food 210372056 197642355 1.06
parquet_rs-zstd decompress time/HashTags 🚨 737728485 622248764 1.19
parquet_rs-zstd decompress time/TPC-H l_comment canonical 611216645 585776565 1.04
parquet_rs-zstd decompress time/TPC-H l_comment chunked 610521468 590528552 1.03
parquet_rs-zstd decompress time/taxi 256231819 242087767 1.06
parquet_rs-zstd decompress time/wide table cols=100 chunks=1 rows=1000 3016936 2852928 1.06
parquet_rs-zstd decompress time/wide table cols=100 chunks=50 rows=1000 🚀 3033123 3465264 0.88
parquet_rs-zstd decompress time/wide table cols=1000 chunks=1 rows=1000 35811046 32657325 1.10
parquet_rs-zstd decompress time/wide table cols=1000 chunks=50 rows=1000 36575290 37788078 0.97
parquet_rs-zstd decompress time/wide table cols=10000 chunks=1 rows=1000 359084131 346059142 1.04
parquet_rs-zstd decompress time/wide table cols=10000 chunks=50 rows=1000 355552747 369038599 0.96
vortex-file-compressed size/Arade 145363796 145363796 1.00
vortex-file-compressed size/Bimbo 468763332 468763332 1.00
vortex-file-compressed size/CMSprovider 417907812 417907812 1.00
vortex-file-compressed size/Euro2016 163757340 163497796 1.00
vortex-file-compressed size/Food 41926936 41926936 1.00
vortex-file-compressed size/HashTags 195647828 195647828 1.00
vortex-file-compressed size/TPC-H l_comment canonical 179087360 179087360 1.00
vortex-file-compressed size/TPC-H l_comment chunked 179087360 179087360 1.00
vortex-file-compressed size/taxi 52363948 52363948 1.00
vortex-file-compressed size/wide table cols=100 chunks=1 rows=1000 930848 930848 1.00
vortex-file-compressed size/wide table cols=100 chunks=50 rows=1000 930848 930848 1.00
vortex-file-compressed size/wide table cols=1000 chunks=1 rows=1000 9293648 9293648 1.00
vortex-file-compressed size/wide table cols=1000 chunks=50 rows=1000 9293648 9293648 1.00
vortex-file-compressed size/wide table cols=10000 chunks=1 rows=1000 92957648 92957648 1.00
vortex-file-compressed size/wide table cols=10000 chunks=50 rows=1000 92957648 92957648 1.00
vortex:parquet-zstd ratio compress time/Arade 0 0 1.01
vortex:parquet-zstd ratio compress time/Bimbo 0 0 1.00
vortex:parquet-zstd ratio compress time/CMSprovider 0 0 0.95
vortex:parquet-zstd ratio compress time/Euro2016 0 0 1.03
vortex:parquet-zstd ratio compress time/Food 0 0 1.07
vortex:parquet-zstd ratio compress time/HashTags 0 0 0.98
vortex:parquet-zstd ratio compress time/TPC-H l_comment canonical 0 0 0.98
vortex:parquet-zstd ratio compress time/TPC-H l_comment chunked 0 0 0.98
vortex:parquet-zstd ratio compress time/taxi 0 0 0.97
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=1 rows=1000 1 1 0.92
vortex:parquet-zstd ratio compress time/wide table cols=100 chunks=50 rows=1000 1 1 0.95
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=1 rows=1000 1 1 0.98
vortex:parquet-zstd ratio compress time/wide table cols=1000 chunks=50 rows=1000 1 1 1.03
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=1 rows=1000 1 1 0.99
vortex:parquet-zstd ratio compress time/wide table cols=10000 chunks=50 rows=1000 1 1 1.05
vortex:parquet-zstd ratio decompress time/Arade 0 0 1.05
vortex:parquet-zstd ratio decompress time/Bimbo 0 0 1.02
vortex:parquet-zstd ratio decompress time/CMSprovider 🚨 0 0 1.15
vortex:parquet-zstd ratio decompress time/Euro2016 0 0 1.06
vortex:parquet-zstd ratio decompress time/Food 🚨 0 0 1.11
vortex:parquet-zstd ratio decompress time/HashTags 0 0 0.91
vortex:parquet-zstd ratio decompress time/TPC-H l_comment canonical 0 0 1.03
vortex:parquet-zstd ratio decompress time/TPC-H l_comment chunked 0 0 1.03
vortex:parquet-zstd ratio decompress time/taxi 🚨 0 0 1.14
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=1 rows=1000 0 0 1.02
vortex:parquet-zstd ratio decompress time/wide table cols=100 chunks=50 rows=1000 0 0 1.02
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=1 rows=1000 0 0 1.10
vortex:parquet-zstd ratio decompress time/wide table cols=1000 chunks=50 rows=1000 0 0 1.03
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=1 rows=1000 0 0 1.02
vortex:parquet-zstd ratio decompress time/wide table cols=10000 chunks=50 rows=1000 0 0 0.95
vortex:parquet-zstd size/Arade 0 0 1.00
vortex:parquet-zstd size/Bimbo 1 1 1.00
vortex:parquet-zstd size/CMSprovider 1 1 1.00
vortex:parquet-zstd size/Euro2016 1 1 1.00
vortex:parquet-zstd size/Food 1 1 1.00
vortex:parquet-zstd size/HashTags 1 1 1.00
vortex:parquet-zstd size/TPC-H l_comment canonical 1 1 1.00
vortex:parquet-zstd size/TPC-H l_comment chunked 1 1 1.00
vortex:parquet-zstd size/taxi 0 0 1.00
vortex:parquet-zstd size/wide table cols=100 chunks=1 rows=1000 0 0 1.00
vortex:parquet-zstd size/wide table cols=100 chunks=50 rows=1000 0 0 1.00
vortex:parquet-zstd size/wide table cols=1000 chunks=1 rows=1000 0 0 1.00
vortex:parquet-zstd size/wide table cols=1000 chunks=50 rows=1000 0 0 1.00
vortex:parquet-zstd size/wide table cols=10000 chunks=1 rows=1000 0 0 1.00
vortex:parquet-zstd size/wide table cols=10000 chunks=50 rows=1000 0 0 1.00

@github-actions
Copy link
Copy Markdown
Contributor

Benchmarks: Random Access

Vortex (geomean): 0.991x ➖
Parquet (geomean): 1.005x ➖


unknown / unknown (1.103x ❌, 2↑ 6↓)
name PR ff8e9f3 (ns) base faf7e42 (ns) ratio (PR/base)
random-access/feature-vectors/correlated/lance-tokio-local-disk 🚨 2042578 391721 5.21
random-access/feature-vectors/correlated/lance-tokio-local-disk-footer 🚨 1508526 1038299 1.45
random-access/feature-vectors/correlated/parquet-tokio-local-disk 8222873129 8241471074 1.00
random-access/feature-vectors/correlated/parquet-tokio-local-disk-footer 8242749009 8380374208 0.98
random-access/feature-vectors/correlated/vortex-tokio-local-disk 🚨 9211689 7919674 1.16
random-access/feature-vectors/correlated/vortex-tokio-local-disk-footer 7278332 7359555 0.99
random-access/feature-vectors/uniform/lance-tokio-local-disk 🚨 5747841 1188808 4.83
random-access/feature-vectors/uniform/lance-tokio-local-disk-footer 🚨 6510712 1840017 3.54
random-access/feature-vectors/uniform/parquet-tokio-local-disk 8230379180 8441581141 0.97
random-access/feature-vectors/uniform/parquet-tokio-local-disk-footer 8233944514 8138240117 1.01
random-access/feature-vectors/uniform/vortex-tokio-local-disk 13247323 13215284 1.00
random-access/feature-vectors/uniform/vortex-tokio-local-disk-footer 12938538 13138881 0.98
random-access/lance-tokio-local-disk 🚨 760097 648673 1.17
random-access/lance-tokio-local-disk-footer 1322812 1311017 1.01
random-access/nested-lists/correlated/lance-tokio-local-disk 250097 241123 1.04
random-access/nested-lists/correlated/lance-tokio-local-disk-footer 636936 581738 1.09
random-access/nested-lists/correlated/parquet-tokio-local-disk 127385396 126742562 1.01
random-access/nested-lists/correlated/parquet-tokio-local-disk-footer 128469586 126758392 1.01
random-access/nested-lists/correlated/vortex-tokio-local-disk 585510 612238 0.96
random-access/nested-lists/correlated/vortex-tokio-local-disk-footer 560329 615965 0.91
random-access/nested-lists/uniform/lance-tokio-local-disk 1068963 1010059 1.06
random-access/nested-lists/uniform/lance-tokio-local-disk-footer 1475791 1374527 1.07
random-access/nested-lists/uniform/parquet-tokio-local-disk 128297285 126655926 1.01
random-access/nested-lists/uniform/parquet-tokio-local-disk-footer 128026291 126563842 1.01
random-access/nested-lists/uniform/vortex-tokio-local-disk 2087635 2089489 1.00
random-access/nested-lists/uniform/vortex-tokio-local-disk-footer 2069350 2067511 1.00
random-access/nested-structs/correlated/lance-tokio-local-disk 391585 369077 1.06
random-access/nested-structs/correlated/lance-tokio-local-disk-footer 597156 548772 1.09
random-access/nested-structs/correlated/parquet-tokio-local-disk 21348246 21529893 0.99
random-access/nested-structs/correlated/parquet-tokio-local-disk-footer 21415147 21502122 1.00
random-access/nested-structs/correlated/vortex-tokio-local-disk 738518 740018 1.00
random-access/nested-structs/correlated/vortex-tokio-local-disk-footer 728884 745243 0.98
random-access/nested-structs/uniform/lance-tokio-local-disk 2576913 2528590 1.02
random-access/nested-structs/uniform/lance-tokio-local-disk-footer 2798278 2720466 1.03
random-access/nested-structs/uniform/parquet-tokio-local-disk 21321305 21296154 1.00
random-access/nested-structs/uniform/parquet-tokio-local-disk-footer 21541801 20955370 1.03
random-access/nested-structs/uniform/vortex-tokio-local-disk 1661295 1544787 1.08
random-access/nested-structs/uniform/vortex-tokio-local-disk-footer 1622181 1617360 1.00
random-access/parquet-tokio-local-disk 166836213 165547024 1.01
random-access/parquet-tokio-local-disk-footer 167922447 165617757 1.01
random-access/taxi/correlated/lance-tokio-local-disk 944286 943014 1.00
random-access/taxi/correlated/lance-tokio-local-disk-footer 🚀 1604962 1822389 0.88
random-access/taxi/correlated/parquet-tokio-local-disk 252987171 248691996 1.02
random-access/taxi/correlated/parquet-tokio-local-disk-footer 251316169 249023474 1.01
random-access/taxi/correlated/vortex-tokio-local-disk 🚀 1572412 1760730 0.89
random-access/taxi/correlated/vortex-tokio-local-disk-footer 1747122 1772169 0.99
random-access/taxi/uniform/lance-tokio-local-disk 9584615 9493317 1.01
random-access/taxi/uniform/lance-tokio-local-disk-footer 10201276 10036949 1.02
random-access/taxi/uniform/parquet-tokio-local-disk 264169362 263043477 1.00
random-access/taxi/uniform/parquet-tokio-local-disk-footer 265205333 262432364 1.01
random-access/taxi/uniform/vortex-tokio-local-disk 4517497 4510635 1.00
random-access/taxi/uniform/vortex-tokio-local-disk-footer 4487818 4524292 0.99
random-access/vortex-tokio-local-disk 1196150 1229042 0.97
random-access/vortex-tokio-local-disk-footer 1215992 1257349 0.97

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant