Skip to content

feat: Support test sharding for CI parallelization with test_optimizer #1538

@TonyDowney

Description

@TonyDowney

Problem

very_good test --optimization consolidates all tests into a single .test_optimizer.dart file, which is great for reducing startup overhead. However, this single-file approach prevents CI-level sharding — splitting the test suite across multiple parallel CI runners.

For large Flutter projects, the test phase is often the CI bottleneck. In our case, PR checks take 13-23 minutes with the test phase accounting for 8-15 minutes, even with -j 4 concurrency within a single runner. Our NFR target is under 10 minutes for the full PR pipeline.

Proposal

Add --shard and --total-shards parameters (or similar) to very_good test --optimization that would:

  1. Partition the consolidated test imports across N shards deterministically (e.g., by hash or alphabetical split)
  2. Generate shard-specific optimizer files (e.g., .test_optimizer_1_of_3.dart) each containing a subset of test imports
  3. Allow CI workflows to use a strategy.matrix to run shards in parallel:
strategy:
  matrix:
    shard: [1, 2, 3]
steps:
  - run: very_good test --optimization --shard ${{ matrix.shard }} --total-shards 3

Alternatives Considered

  • -j N concurrency: Already in use, but limited to single-runner parallelism. Diminishing returns beyond core count.
  • --exclude-tags: Helps reduce scope but doesn't address the fundamental single-runner bottleneck.
  • Manual test file splitting: Defeats the purpose of test_optimizer's consolidation.
  • flutter test --shard-index: Flutter's native sharding works on individual test files, but conflicts with test_optimizer's single-file approach.

Context

This would complement the existing --optimization flag by extending its benefits to multi-runner CI setups. The test_optimizer already knows the full list of test files — partitioning them into N balanced groups seems like a natural extension.

Would also pair well with very_good_workflows if the reusable workflow added a shards input parameter.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Projects

Status

Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions