-
Notifications
You must be signed in to change notification settings - Fork 174
ROX-27677: Eliminate Python dependency from operator build #18947
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
159 commits
Select commit
Hold shift + click to select a range
f2bcbaf
chore(operator): wrapper for bundle helpers
porridge 5e1b473
chore(operator): Go implemetation of fix-spec-descriptor-order
porridge 61b39a4
fixes
porridge 3fa1620
merge
porridge bfe4711
tidy
porridge 5068886
move the tidy
porridge 6766236
go mod tidy
porridge 11d68b9
fix deps, checkout
porridge 849d3c6
cleanup e2e
porridge e52c177
fix normalizer path
porridge e33466c
fix comparison
porridge 10f6db8
improve comparison, fix summary
porridge 4352773
keep linter happy
porridge c39b056
lint
porridge 45788b4
lint
porridge 36bb72c
try multiple modes
porridge 0028766
pass related images mode
porridge 3286a78
do not fail fast, set related image vars
porridge 05d7951
matrix artifacts
porridge 1a0e641
set -e and git diff at end
porridge abde0a2
branding
porridge 92ce394
chore(operator): complete Go bundle helpers implementation
GrimmiMeloni 4995d3f
docs(operator): add bundle helpers README
GrimmiMeloni 5f58e1f
fix(operator): address Sourcery review comments on bundle helpers
GrimmiMeloni 89c5123
fix(operator): use errors.New instead of fmt.Errorf without args
GrimmiMeloni acdb6e3
chore: Trigger Konflux pipeline re-run
GrimmiMeloni 985c4fd
chore(ci): add gomod prefetch to operator bundle build
GrimmiMeloni 570c1da
adjust konflux docker file for go build
GrimmiMeloni 564c018
switch microdnf to dnf
GrimmiMeloni dd0ab37
switch default version of python3 in the builder to installed 3.12
GrimmiMeloni 7ba2145
align python versions across docker build and test GHA
GrimmiMeloni 68c8437
review changes
GrimmiMeloni 3c8b871
refactor(operator): deduplicate skip version processing logic
GrimmiMeloni 830f737
remove obsoletes
GrimmiMeloni 3236639
simply skip version detection
GrimmiMeloni e1142d3
simplify more
GrimmiMeloni 510b04f
more shared common logic
GrimmiMeloni 3826c1f
simplify
GrimmiMeloni 889296f
adjust behavior of echoReplaceVersion
GrimmiMeloni c6ab251
improve bundle helper comparison script isolation
GrimmiMeloni 788ce75
add error handling for ENV vars
GrimmiMeloni 09a2147
adjust comment
GrimmiMeloni af1c133
modernize sorter
GrimmiMeloni 6ad98e9
further simplify sorter
GrimmiMeloni 5a21f41
deduplicate code around yaml encoding
GrimmiMeloni fdb7a2d
address code review from sourcery.ai
GrimmiMeloni 2ce59ea
adjust pyyaml version
GrimmiMeloni d256219
fix(operator): upgrade pip and setuptools for Python 3.12 compatibility
GrimmiMeloni 4407c07
fix(operator): use two-stage pip upgrade for old CI environments
GrimmiMeloni 7927523
fix style
GrimmiMeloni 2f32121
fix(operator): simplify to single-stage upgrade with pip 21.3.1
GrimmiMeloni bfe8edb
fix(operator): rely on venv's default setuptools for compatibility
GrimmiMeloni d7ecc1a
fix(operator): remove pip upgrade to avoid Python 3.12 incompatibility
GrimmiMeloni fa05049
Update .github/workflows/test-bundle-helpers.yaml
GrimmiMeloni ee1a728
Update .github/workflows/test-bundle-helpers.yaml
GrimmiMeloni 67d6db7
Update .github/workflows/test-bundle-helpers.yaml
GrimmiMeloni 6a5687a
Update .github/workflows/test-bundle-helpers.yaml
GrimmiMeloni a11fa5e
Update .github/workflows/test-bundle-helpers.yaml
GrimmiMeloni 2dc18bd
Update .github/workflows/test-bundle-helpers.yaml
GrimmiMeloni 0dd3e02
Update .github/workflows/test-bundle-helpers.yaml
GrimmiMeloni dc76560
Update .github/workflows/test-bundle-helpers.yaml
GrimmiMeloni 96ada65
incorporate review comments
GrimmiMeloni 2c17dce
Update operator/bundle_helpers/yaml-normalizer.py
GrimmiMeloni c9e3687
Update operator/bundle_helpers/yaml-normalizer.py
GrimmiMeloni 160fa22
Update operator/bundle_helpers/README.md
GrimmiMeloni b374852
Update operator/bundle_helpers/README.md
GrimmiMeloni 1425c1b
Update operator/bundle_helpers/README.md
GrimmiMeloni a436939
address review comments
GrimmiMeloni 1013d3a
address review comments
GrimmiMeloni 4aa2253
Update operator/bundle_helpers/pkg/rewrite/rewriter_test.go
GrimmiMeloni 6a07478
refactor replacedVersion from pointer to value type
GrimmiMeloni 60d2d62
Update operator/bundle_helpers/pkg/rewrite/rewriter.go
GrimmiMeloni c7c062a
Update operator/bundle_helpers/pkg/rewrite/rewriter.go
GrimmiMeloni 3a72beb
Update operator/bundle_helpers/pkg/descriptor/sorter.go
GrimmiMeloni e924bf1
Update operator/bundle_helpers/pkg/csv/version.go
GrimmiMeloni bd311eb
test: update TestCalculateReplacedVersion to expect nil for no replac…
GrimmiMeloni af6c1d1
refactor: change CalculateReplacedVersion to return *XyzVersion
GrimmiMeloni e87114d
refactor: update CalculateReplacedVersionForCSV to return *XyzVersion
GrimmiMeloni 4246452
fix: only set spec["replaces"] when replacedVersion is not nil
GrimmiMeloni ed15741
fix: only print replacedVersion when not nil in echo mode
GrimmiMeloni de48ded
refactor: parse version strings early in CalculateReplacedVersion
GrimmiMeloni b1cbd6b
refactor: parse version strings early in CalculateReplacedVersionForCSV
GrimmiMeloni 92e3f13
fix integration tests: export RELATED_IMAGE_* env vars
GrimmiMeloni 578c6fe
test: add initial TestFixCSVDescriptorsMap cases
GrimmiMeloni 75c2ed7
test: add remaining TestFixCSVDescriptorsMap cases
GrimmiMeloni 710d772
test: add TestAllowRelativeFieldDependenciesMap
GrimmiMeloni eb39f8f
feat(bundle-helpers): add GetString helper for chartutil.Values
GrimmiMeloni b3869ff
chore(bundle-helpers): fix code style in values package
GrimmiMeloni d884111
feat(bundle-helpers): add GetMap helper
GrimmiMeloni aa4a8e4
fix(bundle-helpers): fix GetMap to use PathValue pattern
GrimmiMeloni 9763cb2
feat(bundle-helpers): add GetArray helper
GrimmiMeloni d49040f
feat(bundle-helpers): add GetValue and PathExists helpers
GrimmiMeloni 7465bea
feat(bundle-helpers): add SetValue helper
GrimmiMeloni a8fdac9
chore(bundle-helpers): fix code style in values package
GrimmiMeloni 19d80db
refactor(bundle-helpers): use chartutil.ReadValues in patch_csv
GrimmiMeloni b29a618
fix(bundle-helpers): handle top-level paths and chartutil.Values in v…
GrimmiMeloni 0dd6004
fix(bundle-helpers): handle chartutil.Values in rewrite.Strings
GrimmiMeloni 2d4d1b7
refactor(bundle-helpers): use chartutil.Values and helpers in patcher
GrimmiMeloni e28b360
refactor(bundle-helpers): eliminate duplicated branch in injectRelate…
GrimmiMeloni 5f67097
fix(bundle-helpers): fix GetMap and handle dotted annotation/label keys
GrimmiMeloni 90f5688
refactor(bundle-helpers): use chartutil.Values in fix-spec-descriptor…
GrimmiMeloni 5e97de1
refactor(bundle-helpers): remove dead chartutil.Values branches in ad…
GrimmiMeloni 6f22895
refactor(bundle-helpers): simplify metadata.labels access in PatchCSV
GrimmiMeloni 0534e09
fix(bundle-helpers): always ensure metadata.labels map exists in Patc…
GrimmiMeloni 865cbee
refactor(bundle-helpers): remove pathValue wrapper, use PathValue dir…
GrimmiMeloni f977d42
refactor(bundle-helpers): remove dead code GetValue and PathExists
GrimmiMeloni e476b08
refactor(bundle-helpers): remove GetMap wrapper, use vals.Table directly
GrimmiMeloni fd8be01
refactor(bundle-helpers): simplify CRD owned lookup in addSecurityPol…
GrimmiMeloni cf6dc26
fix comment
GrimmiMeloni 972bac3
fix(bundle-helpers): propagate strconv.Atoi errors in ParseXyzVersion
GrimmiMeloni 62316e1
stronger typing, prefer XyzVersion over string
GrimmiMeloni b699bc5
rename symbol
GrimmiMeloni 8da2d38
improve comment in advancePastSkips
GrimmiMeloni 3fd7c68
add semantic comparison methods to XyzVersion
GrimmiMeloni 67b657c
refactor(bundle-helpers): simplify test inputs with inline YAML
GrimmiMeloni 0bdc43a
refactor(bundle-helpers): use t.Setenv instead of os.Setenv in tests
GrimmiMeloni bf9f935
refactor(bundle-helpers): move shared YAML output helpers to cmd/outp…
GrimmiMeloni 1ee80f0
style fix
GrimmiMeloni e455b97
fix(bundle-helpers): restore RELATED_IMAGE_* cleanup in NoEnvVars test
GrimmiMeloni 42698d9
fix(bundle-helpers): fix compare script cp destination for build-bundle
GrimmiMeloni 60a0a8b
fix(bundle-helpers): restore direct map mutation in addSecurityPolicyCRD
GrimmiMeloni 5b3e6f3
fix(bundle-helpers): delete sed backup files before diff in compare s…
GrimmiMeloni 74402de
bump operator golang builder to 1.25.7
GrimmiMeloni 3178b54
Update operator/bundle_helpers/cmd/patch_csv.go
GrimmiMeloni 6e46d7a
Update operator/bundle_helpers/cmd/patch_csv.go
GrimmiMeloni 63f44cf
Update operator/bundle_helpers/pkg/csv/patcher.go
GrimmiMeloni ed5d880
Update operator/bundle_helpers/compare-implementations.sh
GrimmiMeloni ce906e8
Update operator/bundle_helpers/compare-implementations.sh
GrimmiMeloni 7637039
Update operator/bundle_helpers/compare-implementations.sh
GrimmiMeloni 4def08d
Update operator/bundle_helpers/compare-implementations.sh
GrimmiMeloni 994d4ae
Update operator/bundle_helpers/pkg/csv/patcher.go
GrimmiMeloni 38c7b57
Update operator/bundle_helpers/pkg/csv/patcher.go
GrimmiMeloni d4f0ed1
Update operator/bundle_helpers/pkg/csv/patcher.go
GrimmiMeloni 994f68e
Update operator/bundle_helpers/pkg/csv/patcher.go
GrimmiMeloni 8379fd9
Update operator/bundle_helpers/pkg/csv/patcher.go
GrimmiMeloni 8b87315
Update operator/bundle_helpers/pkg/csv/patcher_test.go
GrimmiMeloni 801cc3a
Update operator/bundle_helpers/pkg/descriptor/sorter.go
GrimmiMeloni 6676a85
Update operator/bundle_helpers/pkg/rewrite/rewriter.go
GrimmiMeloni 7f7b13f
address review
GrimmiMeloni 2cb85db
remove unnecessary closure
GrimmiMeloni 89b389a
rename symbol
GrimmiMeloni fd27d76
rename symbol
GrimmiMeloni 45479b2
split testcase
GrimmiMeloni 333bd78
remove superflous test
GrimmiMeloni f43e662
add logging
GrimmiMeloni aedf211
use PathValue in sorter instead of map traversal
GrimmiMeloni 7358bdd
rename and simplify
GrimmiMeloni 0f3e97d
fix regression
GrimmiMeloni 7d55d83
bugfix field dep path split
GrimmiMeloni 35b1bf7
add missing type assertion check, missed the first time
GrimmiMeloni a66d9b4
downgrade PyYAML to 6.0.1 for Konflux compatibility
GrimmiMeloni 47166bd
add support for FACT to test harness
GrimmiMeloni 4c349bb
remove CGO from konflux bundle builds
GrimmiMeloni d9c63cf
rename as agreed in review
GrimmiMeloni 692bef1
switch konflux bundle builder to rhel_9 for consistency
GrimmiMeloni 18279ce
switch konflux bundle builder back to ubi9-minimal with golang
GrimmiMeloni e1df128
disable Go module caching in test-bundle-helpers workflow
GrimmiMeloni b25d966
use shared cache-go-dependencies action in test-bundle-helpers
GrimmiMeloni 6b43bf6
switch konflux bundle builder back to openshift-golang-builder
GrimmiMeloni File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,250 @@ | ||
| # This workflow makes sure that the Go and Python outputs of operator bundle helpers are byte for byte equivalent. | ||
| # It will be removed together with the Python implementation in a follow-up PR. | ||
| name: Test Bundle Helpers | ||
|
|
||
| on: | ||
| push: | ||
| tags: | ||
| - '*' | ||
| branches: | ||
| - master | ||
| - release-* | ||
| pull_request: | ||
| types: | ||
| - opened | ||
| - reopened | ||
| - synchronize | ||
|
|
||
| env: | ||
| ROX_PRODUCT_BRANDING: RHACS_BRANDING | ||
| RELATED_IMAGE_MAIN: foo | ||
| RELATED_IMAGE_SCANNER: foo | ||
| RELATED_IMAGE_SCANNER_SLIM: foo | ||
| RELATED_IMAGE_SCANNER_DB: foo | ||
| RELATED_IMAGE_SCANNER_DB_SLIM: foo | ||
| RELATED_IMAGE_COLLECTOR: foo | ||
| RELATED_IMAGE_FACT: foo | ||
| RELATED_IMAGE_ROXCTL: foo | ||
| RELATED_IMAGE_CENTRAL_DB: foo | ||
| RELATED_IMAGE_SCANNER_V4_DB: foo | ||
| RELATED_IMAGE_SCANNER_V4: foo | ||
|
|
||
GrimmiMeloni marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| jobs: | ||
| test-python-implementation: | ||
| strategy: | ||
| matrix: | ||
| related_img_mode: | ||
| - konflux | ||
| - omit | ||
| - downstream | ||
| name: Test Python Implementation | ||
| runs-on: ubuntu-latest | ||
| defaults: | ||
| run: | ||
| working-directory: operator | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
| ref: ${{ github.event.pull_request.head.sha }} | ||
|
|
||
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: '3.12' | ||
|
|
||
| - name: Install Python dependencies | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| pip install -r bundle_helpers/requirements.txt | ||
|
|
||
| - name: Run Python unit tests | ||
| run: make test-bundle-helpers | ||
|
|
||
| - name: Generate bundle with Python | ||
| env: | ||
| USE_GO_BUNDLE_HELPER: false | ||
| RELATED_IMAGES_MODE: ${{ matrix.related_img_mode }} | ||
| run: | | ||
| set -euo pipefail | ||
| go mod tidy | ||
| make bundle bundle-post-process | ||
| mkdir -p /tmp/artifacts | ||
| cp -r bundle /tmp/artifacts/bundle-python | ||
| cp -r build/bundle /tmp/artifacts/build-bundle-python | ||
| # Make sure the build is clean. | ||
| git diff --exit-code HEAD | ||
|
|
||
| - name: Upload Python bundle artifacts | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: bundle-python-${{ matrix.related_img_mode }} | ||
| path: /tmp/artifacts/ | ||
| retention-days: 1 | ||
|
|
||
| test-go-implementation: | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| related_img_mode: | ||
| - konflux | ||
| - omit | ||
| - downstream | ||
| name: Test Go Implementation | ||
| runs-on: ubuntu-latest | ||
| defaults: | ||
| run: | ||
| working-directory: operator | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
| ref: ${{ github.event.pull_request.head.sha }} | ||
|
|
||
| - name: Set up Go | ||
| uses: actions/setup-go@v5 | ||
| with: | ||
| go-version-file: go.mod | ||
GrimmiMeloni marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| cache: false | ||
|
|
||
| - name: Cache Go dependencies | ||
| uses: ./.github/actions/cache-go-dependencies | ||
|
|
||
| - name: Run Go unit tests | ||
| run: | | ||
| cd bundle_helpers | ||
| go test ./... | ||
|
|
||
| # Needed since the Go implementation uses yaml-normalizer.py | ||
GrimmiMeloni marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - name: Set up Python | ||
| uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: '3.12' | ||
|
|
||
| # Needed since the Go implementation uses yaml-normalizer.py | ||
| - name: Install Python dependencies | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| pip install -r bundle_helpers/requirements.txt | ||
|
|
||
| - name: Generate bundle with Go | ||
| env: | ||
| USE_GO_BUNDLE_HELPER: true | ||
| RELATED_IMAGES_MODE: ${{ matrix.related_img_mode }} | ||
| run: | | ||
| set -euo pipefail | ||
| go mod tidy | ||
| make bundle bundle-post-process | ||
| mkdir -p /tmp/artifacts | ||
| cp -r bundle /tmp/artifacts/bundle-go | ||
| cp -r build/bundle /tmp/artifacts/build-bundle-go | ||
| # Make sure the build is clean. | ||
| git diff --exit-code HEAD | ||
|
|
||
| - name: Upload Go bundle artifacts | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: bundle-go-${{ matrix.related_img_mode }} | ||
| path: /tmp/artifacts/ | ||
| retention-days: 1 | ||
|
|
||
| compare-implementations: | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| related_img_mode: | ||
| - konflux | ||
| - omit | ||
| - downstream | ||
| name: Compare Python and Go Outputs | ||
| runs-on: ubuntu-latest | ||
| needs: [test-python-implementation, test-go-implementation] | ||
|
|
||
| steps: | ||
| - name: Download Python bundle | ||
| uses: actions/download-artifact@v4 | ||
| with: | ||
| name: bundle-python-${{ matrix.related_img_mode }} | ||
| path: /tmp/python | ||
|
|
||
| - name: Download Go bundle | ||
| uses: actions/download-artifact@v4 | ||
| with: | ||
| name: bundle-go-${{ matrix.related_img_mode }} | ||
| path: /tmp/go | ||
|
|
||
| - name: Install comparison tools | ||
| run: | | ||
| sudo apt-get update | ||
| sudo apt-get install -y diffutils | ||
|
|
||
| - name: Compare bundle outputs | ||
| run: | | ||
| echo "=== Comparing Python and Go bundle outputs ===" | ||
|
|
||
| if diff -ruN /tmp/python/bundle-python /tmp/go/bundle-go; then | ||
| echo "✓ SUCCESS: Bundle outputs are identical" | ||
| else | ||
| echo "✗ FAILURE: Bundle outputs differ" | ||
| exit 1 | ||
| fi | ||
| echo "" | ||
| echo "=== Pruning createdAt lines..." | ||
| sed -i '/^ createdAt:/d' /tmp/*/build-bundle-*/manifests/rhacs-operator.clusterserviceversion.yaml | ||
|
|
||
| echo "" | ||
| echo "=== Comparing build/bundle outputs ===" | ||
| if diff -ruN /tmp/python/build-bundle-python /tmp/go/build-bundle-go; then | ||
| echo "✓ SUCCESS: Build bundle outputs are identical" | ||
| else | ||
| echo "✗ FAILURE: Build bundle outputs differ" | ||
| exit 1 | ||
| fi | ||
| echo "" | ||
| echo "=== Listing contents" | ||
| find /tmp/python /tmp/go -ls | ||
| find /tmp/python /tmp/go -type f -print0 | xargs -0 md5sum | sort | ||
|
|
||
| status-check: | ||
| strategy: | ||
| matrix: | ||
| related_img_mode: | ||
| - konflux | ||
| - omit | ||
| - downstream | ||
| name: Bundle Helper Tests Status | ||
| runs-on: ubuntu-latest | ||
| needs: [test-python-implementation, test-go-implementation, compare-implementations] | ||
| if: always() | ||
|
|
||
| steps: | ||
| - name: Check test results | ||
| run: | | ||
| python_result="${{ needs.test-python-implementation.result }}" | ||
| go_result="${{ needs.test-go-implementation.result }}" | ||
| compare_result="${{ needs.compare-implementations.result }}" | ||
|
|
||
| echo "Python tests: $python_result" | ||
| echo "Go tests: $go_result" | ||
| echo "Comparison: $compare_result" | ||
|
|
||
| if [ "$python_result" != "success" ]; then | ||
| echo "✗ Python tests failed" | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [ "$go_result" != "success" ]; then | ||
| echo "✗ Go tests failed" | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [ "$compare_result" != "success" ]; then | ||
| echo "✗ Go/Python comparison failed" | ||
| exit 1 | ||
| fi | ||
|
|
||
| echo "✓ All required tests passed" | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.