Skip to content

Support parallel step insertion and parallelism breakup in Bitbucket pipelines#1051

Merged
nathanjmcdougall merged 19 commits intomainfrom
copilot/support-parallel-steps-bitbucket
Oct 27, 2025
Merged

Support parallel step insertion and parallelism breakup in Bitbucket pipelines#1051
nathanjmcdougall merged 19 commits intomainfrom
copilot/support-parallel-steps-bitbucket

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Oct 26, 2025 •
edited
Loading

Support adding steps in parallel in Bitbucket pipelines

Summary:
This PR implements support for adding steps in parallel and breaking up existing parallelism in Bitbucket pipelines, completing the integration with the pipeweld solver.

Features Implemented:

  1. InsertParallel instruction handling

    • Converts single steps to parallel blocks
    • Adds to existing parallel blocks (both simple and expanded YAML formats)
    • Handles StepItem, ParallelItem, and StageItem cases

  2. Breaking up existing parallelism

    • Extracts and moves steps from parallel blocks to satisfy dependencies
    • Automatically simplifies parallel blocks when reduced to one step
    • Supports full pipeweld instruction sequences for rearrangement

Changes:

  • Modified apply_pipeweld_instruction_via_doc to handle both new step insertion and existing step movement
  • Added _insert_parallel_step singledispatch for parallel block creation/extension
  • Added _extract_step_from_items and _extract_step_from_item for step extraction
  • Added comprehensive test coverage (6 test cases total)

Recent updates:

  • Restored new_step.name in import pipeline error message for better error reporting
  • Removed redundant test_extract_step_from_parallel test
  • Clarified comment: instructions apply to existing steps (not "move" instructions)
  • Changed to raise AssertionError when step not found in pipeline
  • Added detailed comments explaining when is_new_step is true vs false with real-world examples
  • Removed docstrings from all .register functions (keeping only on @singledispatch functions)
  • Refactored to use MissingStepError exception with try-except instead of None checking for more Pythonic and efficient code
  • Extracted instruction application logic into _apply_instruction_to_items helper function to remove noqa comment
  • Renamed new_step parameter to step_to_insert throughout the module for consistency
  • Added test for ParallelExpanded format (line 360)
  • Added test for extracting last step from parallel block, leaving it empty (line 287-289)
Original prompt

This section details on the original issue you should resolve

<issue_title>Support adding steps in parallel in Bitbucket pipelines</issue_title>
<issue_description>Motivation
Based on the current implementation I am working on for #54, I am making the decision to start by only supporting adding new steps in series, not in parallel. Pipeweld, the solver that underlies this logic, already supports adding steps in parallel, we mostly just need to write the integration for this aspect. There's also a similar integration to handle an edge case where we need to move an existing step to break up a parallelism.

Supporting parallelism makes pipelines run more efficiently by taking into account (the lack of) dependency relations between steps. Supporting the ability to break up an existing parallelism allows us to satisfy otherwise-impossible dependency relations.

Summary of desired enhancement
Handle adding a new step to a Bitbucket pipeline in parallel to an existing step, and breaking up an existing parallelism.
</issue_description>

Comments on the Issue (you are @copilot in this section)

@nathanjmcdougall This is where to look: https://github.com/nathanjmcdougall/usethis-python/blob/0c6fb508d668a38e14c872b708bba281c0d6a24f/src/usethis/_integrations/bitbucket/pipeweld.py#L149

https://github.com/nathanjmcdougall/usethis-python/blob/0c6fb508d668a38e14c872b708bba281c0d6a24f/src/usethis/_integrations/bitbucket/pipeweld.py#L124

We would start by creating test cases.</comment_new>

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 5 commits October 27, 2025 00:05
@nathanjmcdougall
Add InsertParallel instruction handling and test cases
7f29321 
Co-authored-by: nathanjmcdougall <18602289+nathanjmcdougall@users.noreply.github.com>
@nathanjmcdougall
Fix singledispatch signature for _insert_parallel_step
42dacbf 
Co-authored-by: nathanjmcdougall <18602289+nathanjmcdougall@users.noreply.github.com>
@nathanjmcdougall
Add support for breaking up parallelism by moving existing steps
81bb3af 
Co-authored-by: nathanjmcdougall <18602289+nathanjmcdougall@users.noreply.github.com>
@nathanjmcdougall
Fix test to use correct instruction order for A, C, B sequence
6500216 
Co-authored-by: nathanjmcdougall <18602289+nathanjmcdougall@users.noreply.github.com>
@nathanjmcdougall
Add clarifying comments to test for instruction/new_step relationship
147b856 
Co-authored-by: nathanjmcdougall <18602289+nathanjmcdougall@users.noreply.github.com>
Copilot AI changed the title [WIP] Support adding steps in parallel in Bitbucket pipelines Support parallel step insertion and parallelism breakup in Bitbucket pipelines Oct 27, 2025
nathanjmcdougall
nathanjmcdougall requested changes Oct 27, 2025
View reviewed changes
@nathanjmcdougall
Restore new_step.name in import pipeline error message
e35688e 
Co-authored-by: nathanjmcdougall <18602289+nathanjmcdougall@users.noreply.github.com>
nathanjmcdougall
nathanjmcdougall requested changes Oct 27, 2025
View reviewed changes
@nathanjmcdougall
Remove test_extract_step_from_parallel test
6eda12d 
Co-authored-by: nathanjmcdougall <18602289+nathanjmcdougall@users.noreply.github.com>
@codecov
Copy link
Copy Markdown

codecov bot commented Oct 27, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Oct 27, 2025 •
edited
Loading

CodSpeed Performance Report

Merging #1051 will not alter performance

Comparing copilot/support-parallel-steps-bitbucket (fa35be1) with main (aff3339)

Summary

✅ 2 untouched

nathanjmcdougall
nathanjmcdougall requested changes Oct 27, 2025
View reviewed changes
@nathanjmcdougall
Clarify comment and raise AssertionError when step not found
7410b36 
Co-authored-by: nathanjmcdougall <18602289+nathanjmcdougall@users.noreply.github.com>
nathanjmcdougall
nathanjmcdougall requested changes Oct 27, 2025
View reviewed changes
@nathanjmcdougall
Add detailed comments explaining new vs existing step logic and remov…
6758f8c 
…e docstrings from register functions

Co-authored-by: nathanjmcdougall <18602289+nathanjmcdougall@users.noreply.github.com>
nathanjmcdougall
nathanjmcdougall requested changes Oct 27, 2025
View reviewed changes
@nathanjmcdougall
Refactor to use MissingStepError exception instead of None checking
25f4c99 
Co-authored-by: nathanjmcdougall <18602289+nathanjmcdougall@users.noreply.github.com>
nathanjmcdougall
nathanjmcdougall requested changes Oct 27, 2025
View reviewed changes
@nathanjmcdougall
Extract instruction application logic and rename new_step to step_to_…
095e3f8 
…insert

Co-authored-by: nathanjmcdougall <18602289+nathanjmcdougall@users.noreply.github.com>
nathanjmcdougall
nathanjmcdougall requested changes Oct 27, 2025
View reviewed changes
@nathanjmcdougall
Add tests for ParallelExpanded format and empty parallel block extrac…
7a28d39 
…tion

Co-authored-by: nathanjmcdougall <18602289+nathanjmcdougall@users.noreply.github.com>
@nathanjmcdougall nathanjmcdougall marked this pull request as ready for review October 27, 2025 03:39
Copilot AI reviewed Oct 27, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements support for adding steps in parallel and breaking up existing parallelism in Bitbucket pipelines to complete integration with the pipeweld solver. The changes enable the system to handle complex dependency scenarios by converting single steps to parallel blocks, extending existing parallel blocks, and extracting steps from parallel blocks when needed to satisfy dependencies.

  • Adds handling for InsertParallel instructions to create or extend parallel step blocks
  • Implements step extraction from parallel blocks to break up existing parallelism when dependencies require it
  • Refactors instruction application to support both new step insertion and existing step rearrangement

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
src/usethis/_integrations/ci/bitbucket/pipeweld.py Core implementation of parallel step insertion and extraction logic with new helper functions and exception handling
src/usethis/_integrations/ci/bitbucket/steps.py Updated parameter name from new_step to step_to_insert for consistency
src/usethis/_integrations/ci/bitbucket/errors.py Added MissingStepError exception for handling missing steps in pipelines
tests/usethis/_integrations/ci/bitbucket/test_pipeweld.py Comprehensive test coverage for parallel insertion and parallelism breakup scenarios

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@nathanjmcdougall
Fix outdated variable name in comment
fa35be1 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@nathanjmcdougall nathanjmcdougall merged commit d58c87d into main Oct 27, 2025
20 checks passed
@nathanjmcdougall nathanjmcdougall deleted the copilot/support-parallel-steps-bitbucket branch October 27, 2025 04:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

+1 more reviewer

@nathanjmcdougall nathanjmcdougall nathanjmcdougall approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@nathanjmcdougall nathanjmcdougall

Copilot code review Copilot

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support adding steps in parallel in Bitbucket pipelines

3 participants

@nathanjmcdougall