Skip to content

Ci fast fail on job failure #5759

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jyotshnayaparla-00 wants to merge 2 commits into from
Closed

Ci fast fail on job failure #5759

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
728ba82
ci: skip build phases on re-enqueue if artifacts already exist
jyotshnayaparla-00 Mar 5, 2026
0b0244e
ci: address review comments and fix docker image check
jyotshnayaparla-00 Mar 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions .github/workflows/build-docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,17 @@ name: Build and Publish Docker Image

on:
workflow_call:
inputs:
artifacts_run_id:
description: "Run ID to download build artifacts from (empty = current run)"
type: string
required: false
default: ""

permissions:
contents: read
packages: write
actions: read

env:
REGISTRY: ghcr.io
Expand Down Expand Up @@ -55,18 +62,24 @@ jobs:
with:
name: fda-${{ matrix.rust_target }}
path: build
run-id: ${{ inputs.artifacts_run_id || github.run_id }}
github-token: ${{ secrets.GITHUB_TOKEN }}

- name: Download pipeline-manager
uses: actions/download-artifact@v4
with:
name: pipeline-manager-${{ matrix.rust_target }}
path: build
run-id: ${{ inputs.artifacts_run_id || github.run_id }}
github-token: ${{ secrets.GITHUB_TOKEN }}

- name: Download Compiler Binaries
uses: actions/download-artifact@v4
with:
name: feldera-sql-compiler
path: build
run-id: ${{ inputs.artifacts_run_id || github.run_id }}
github-token: ${{ secrets.GITHUB_TOKEN }}

# Remove if https://github.com/actions/upload-artifact/issues/38 ever gets fixed
- name: Make binaries executable
Expand Down Expand Up @@ -188,3 +201,15 @@ jobs:
"${{ vars.FELDERA_IMAGE_NAME }}" \
"${{ steps.meta.outputs.version }}" \
"amd64,arm64"

# Upload after the image is verified so check-prior-build knows the
# sha-{sha} tag actually exists in GHCR (not just that digests were pushed).
- name: Signal docker image ready
run: echo "${{ steps.meta.outputs.version }}" > /tmp/docker-image-ready.txt

- name: Upload docker-image-ready artifact
uses: actions/upload-artifact@v4
with:
name: docker-image-ready
path: /tmp/docker-image-ready.txt
retention-days: 7
153 changes: 146 additions & 7 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,116 @@ on:
jobs:
# NOTE: For every job added here, add a matching cancel-if-<name>-failed
# sentinel job in the cancel-if-* section below.

# Checks whether all build artifacts already exist in a prior run for this
# commit. If so, downstream build jobs are skipped and tests download
# artifacts from that prior run instead of the current one.
# NOTE: No cancel sentinel is needed for this job — it is designed to always
# succeed. Individual steps use continue-on-error and the final 'result'
# step runs unconditionally, so the job never leaves the workflow in a
# failed state.
check-prior-build:
name: Check for Prior Build Artifacts
runs-on: ubuntu-latest-amd64
permissions:
actions: read
# Outputs are set by the final 'result' step so this job always succeeds
# and never blocks downstream jobs even if individual checks fail.
outputs:
artifacts_run_id: ${{ steps.result.outputs.artifacts_run_id }}
skip_docker: ${{ steps.result.outputs.skip_docker }}
steps:
- name: Find prior run with all build artifacts for this commit
id: find
continue-on-error: true
env:
GH_TOKEN: ${{ github.token }}
run: |
set +e
rust_java_required=(
"fda-x86_64-unknown-linux-gnu"
"fda-aarch64-unknown-linux-gnu"
"pipeline-manager-x86_64-unknown-linux-gnu"
"pipeline-manager-aarch64-unknown-linux-gnu"
"feldera-test-binaries-x86_64-unknown-linux-gnu"
"feldera-test-binaries-aarch64-unknown-linux-gnu"
"feldera-sql-compiler"
)
# This artifact is uploaded by merge-manifests only after the image is
# verified in GHCR, so its presence guarantees the sha-{sha} tag exists.
docker_required=(
"docker-image-ready"
)

run_ids=$(curl -fsSL \
-H "Authorization: Bearer $GH_TOKEN" \
-H "Accept: application/vnd.github+json" \
"https://api.github.com/repos/${{ github.repository }}/actions/workflows/ci.yml/runs?head_sha=${{ github.sha }}&per_page=20" \
| jq -r --argjson cur '${{ github.run_id }}' \
'.workflow_runs[] | select(.id != $cur) | .id') || true

for run_id in $run_ids; do
names=$(curl -fsSL \
-H "Authorization: Bearer $GH_TOKEN" \
-H "Accept: application/vnd.github+json" \
"https://api.github.com/repos/${{ github.repository }}/actions/runs/${run_id}/artifacts?per_page=200" \
| jq -r '.artifacts[] | select(.expired == false) | .name') || continue

all_rust_java=true
for artifact in "${rust_java_required[@]}"; do
if ! printf '%s\n' $names | grep -qx "$artifact"; then
all_rust_java=false
break
fi
done

if $all_rust_java; then
echo "Found all Rust/Java build artifacts in prior run $run_id"
echo "run_id=$run_id" >> "$GITHUB_OUTPUT"

all_docker=true
for artifact in "${docker_required[@]}"; do
if ! printf '%s\n' $names | grep -qx "$artifact"; then
all_docker=false
break
fi
done

if $all_docker; then
echo "Docker digest artifacts also found, Docker build will be skipped"
echo "skip_docker=true" >> "$GITHUB_OUTPUT"
else
echo "Docker digest artifacts not found, Docker build will run"
echo "skip_docker=false" >> "$GITHUB_OUTPUT"
fi
exit 0
fi
done

echo "No prior run found with all build artifacts, builds will run"
echo "run_id=" >> "$GITHUB_OUTPUT"
echo "skip_docker=false" >> "$GITHUB_OUTPUT"

# Consolidates outputs so both are always set even if the find step failed.
# Defaults: artifacts_run_id='' (run builds), skip_docker=false (run Docker build).
- name: Consolidate outputs
id: result
if: always()
run: |
echo "artifacts_run_id=${{ steps.find.outputs.run_id }}" >> "$GITHUB_OUTPUT"
echo "skip_docker=${{ steps.find.outputs.skip_docker || 'false' }}" >> "$GITHUB_OUTPUT"

invoke-build-rust:
name: Build Rust
needs: [check-prior-build]
if: needs.check-prior-build.outputs.artifacts_run_id == ''
uses: ./.github/workflows/build-rust.yml
secrets: inherit

invoke-build-java:
name: Build Java
needs: [check-prior-build]
if: needs.check-prior-build.outputs.artifacts_run_id == ''
uses: ./.github/workflows/build-java.yml
secrets: inherit

Expand All @@ -24,49 +127,84 @@ jobs:

invoke-tests-unit:
name: Unit Tests
needs: [invoke-build-rust, invoke-build-java]
needs: [check-prior-build, invoke-build-rust, invoke-build-java]
if: |
always() &&
Copy link
Copy Markdown
Contributor

@gz gz Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there some special semantic associated with always() in gha? otherwise this is just if true && a && b and can be rewritten as if a && b

Copy link
Copy Markdown
Contributor Author

@jyotshnayaparla-00 jyotshnayaparla-00 Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

always() is needed here, without it, GHA silently skips dependent jobs when a needs: job is skipped, so the result == 'skipped' checks would never be reached and test jobs would be silently skipped whenever the build phase is skipped.

always() is not true, it prevents GHA from skipping the job before evaluating the condition.

(needs.invoke-build-rust.result == 'success' || needs.invoke-build-rust.result == 'skipped') &&
(needs.invoke-build-java.result == 'success' || needs.invoke-build-java.result == 'skipped')
uses: ./.github/workflows/test-unit.yml
with:
artifacts_run_id: ${{ needs.check-prior-build.outputs.artifacts_run_id }}
secrets: inherit

invoke-tests-adapter:
name: Adapter Tests
needs: [invoke-build-rust]
needs: [check-prior-build, invoke-build-rust]
if: |
always() &&
Copy link
Copy Markdown
Contributor

@gz gz Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same q as above

(needs.invoke-build-rust.result == 'success' || needs.invoke-build-rust.result == 'skipped')
uses: ./.github/workflows/test-adapters.yml
with:
artifacts_run_id: ${{ needs.check-prior-build.outputs.artifacts_run_id }}
secrets: inherit

invoke-build-docker:
name: Build Docker
needs: [invoke-build-rust, invoke-build-java]
needs: [check-prior-build, invoke-build-rust, invoke-build-java]
if: |
always() &&
Copy link
Copy Markdown
Contributor

@gz gz Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

needs.check-prior-build.outputs.skip_docker != 'true' &&
(needs.invoke-build-rust.result == 'success' || needs.invoke-build-rust.result == 'skipped') &&
(needs.invoke-build-java.result == 'success' || needs.invoke-build-java.result == 'skipped')
uses: ./.github/workflows/build-docker.yml
with:
artifacts_run_id: ${{ needs.check-prior-build.outputs.artifacts_run_id }}
secrets: inherit

invoke-generate-sbom:
name: Generate SBOMs
needs: [invoke-build-docker]
if: |
Copy link
Copy Markdown
Contributor

@gz gz Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

always() &&
(needs.invoke-build-docker.result == 'success' || needs.invoke-build-docker.result == 'skipped')
uses: ./.github/workflows/generate-sbom.yml
secrets: inherit

invoke-tests-integration-platform:
name: Integration Tests
needs: [invoke-build-docker]
needs: [check-prior-build, invoke-build-docker]
if: |
Copy link
Copy Markdown
Contributor

@gz gz Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

always() &&
(needs.invoke-build-docker.result == 'success' || needs.invoke-build-docker.result == 'skipped')
uses: ./.github/workflows/test-integration-platform.yml
with:
artifacts_run_id: ${{ needs.check-prior-build.outputs.artifacts_run_id }}
secrets: inherit

invoke-tests-integration-runtime:
name: Integration Tests
needs: [invoke-build-java]
needs: [check-prior-build, invoke-build-java]
if: |
Copy link
Copy Markdown
Contributor

@gz gz Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

always() &&
(needs.invoke-build-java.result == 'success' || needs.invoke-build-java.result == 'skipped')
uses: ./.github/workflows/test-integration-runtime.yml
secrets: inherit

invoke-tests-java:
name: Java Tests
needs: [invoke-build-java]
needs: [check-prior-build, invoke-build-java]
if: |
Copy link
Copy Markdown
Contributor

@gz gz Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

always() &&
(needs.invoke-build-java.result == 'success' || needs.invoke-build-java.result == 'skipped')
uses: ./.github/workflows/test-java.yml
secrets: inherit

invoke-publish-crates-dry-run:
name: Publish Crates (Dry Run)
needs: [invoke-build-rust]
needs: [check-prior-build, invoke-build-rust]
if: |
Copy link
Copy Markdown
Contributor

@gz gz Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same q

always() &&
(needs.invoke-build-rust.result == 'success' || needs.invoke-build-rust.result == 'skipped')
uses: ./.github/workflows/publish-crates.yml
with:
environment: ci
Expand Down Expand Up @@ -234,6 +372,7 @@ jobs:
if: always()
runs-on: ubuntu-latest-amd64
needs:
- check-prior-build
- invoke-build-rust
- invoke-build-java
- invoke-build-docs
Expand Down
14 changes: 12 additions & 2 deletions .github/workflows/test-adapters.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,22 @@ name: Adapter Tests

on:
workflow_call:
inputs:
artifacts_run_id:
description: "Run ID to download build artifacts from (empty = current run)"
type: string
required: false
default: ""
workflow_dispatch:
inputs:
run_id:
description: "ID of the workflow run that uploaded the artifact"
required: true

permissions:
actions: read
contents: read

jobs:
adapter-tests:
if: ${{ !contains(vars.CI_SKIP_JOBS, 'adapter-tests') }}
Expand Down Expand Up @@ -60,7 +70,7 @@ jobs:
with:
name: feldera-test-binaries-${{ matrix.target }}
path: build
run-id: ${{ github.event.inputs.run_id || github.run_id }}
run-id: ${{ inputs.artifacts_run_id || github.event.inputs.run_id || github.run_id }}
github-token: ${{ secrets.GITHUB_TOKEN }}

# Remove if https://github.com/actions/upload-artifact/issues/38 ever gets fixed
Expand Down Expand Up @@ -139,7 +149,7 @@ jobs:
with:
name: feldera-test-binaries-x86_64-unknown-linux-gnu
path: build
run-id: ${{ github.event.inputs.run_id || github.run_id }}
run-id: ${{ inputs.artifacts_run_id || github.event.inputs.run_id || github.run_id }}
github-token: ${{ secrets.GITHUB_TOKEN }}

# Remove if https://github.com/actions/upload-artifact/issues/38 ever gets fixed
Expand Down
13 changes: 12 additions & 1 deletion .github/workflows/test-integration-platform.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,12 @@ name: Platform Integration Tests

on:
workflow_call:
inputs:
artifacts_run_id:
description: "Run ID to download build artifacts from (empty = current run)"
type: string
required: false
default: ""
workflow_dispatch:
inputs:
run_id:
Expand Down Expand Up @@ -147,6 +153,10 @@ jobs:
oss-platform-tests:
if: ${{ !contains(vars.CI_SKIP_JOBS, 'oss-platform-tests') }}
name: Platform Integration Tests (OSS Docker Image)
permissions:
actions: read
contents: read
packages: read
strategy:
matrix:
include:
Expand Down Expand Up @@ -188,7 +198,6 @@ jobs:
--health-interval 10s
--health-timeout 5s
--health-retries 5
--cap-add=PERFMON

steps:
- name: Show Kubernetes node
Expand Down Expand Up @@ -230,6 +239,8 @@ jobs:
with:
name: fda-${{ matrix.target }}
path: build
run-id: ${{ inputs.artifacts_run_id || github.event.inputs.run_id || github.run_id }}
github-token: ${{ secrets.GITHUB_TOKEN }}

# Remove if https://github.com/actions/upload-artifact/issues/38 ever gets fixed
- name: Make fda binary executable
Expand Down
14 changes: 14 additions & 0 deletions .github/workflows/test-unit.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,21 @@ name: Unit Tests

on:
workflow_call:
inputs:
artifacts_run_id:
description: "Run ID to download build artifacts from (empty = current run)"
type: string
required: false
default: ""
workflow_dispatch:
Copy link
Copy Markdown
Contributor

@gz gz Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this also need a run id similar to test-integration-platform.yaml?

Copy link
Copy Markdown
Contributor Author

@jyotshnayaparla-00 jyotshnayaparla-00 Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, added in the latest commit.


env:
RUST_BACKTRACE: 1

permissions:
actions: read
contents: read

jobs:
rust-unit-tests:
name: Rust Unit Tests
Expand All @@ -33,12 +43,16 @@ jobs:
with:
name: feldera-test-binaries-${{ matrix.target }}
path: build
run-id: ${{ inputs.artifacts_run_id || github.run_id }}
github-token: ${{ secrets.GITHUB_TOKEN }}

- name: Download Compiler Binaries
uses: actions/download-artifact@v4
with:
name: feldera-sql-compiler
path: sql-build
run-id: ${{ inputs.artifacts_run_id || github.run_id }}
github-token: ${{ secrets.GITHUB_TOKEN }}

# Remove if https://github.com/actions/upload-artifact/issues/38 ever gets fixed
- name: Make binaries executable
Expand Down
Loading