feat(cli): Add `image build` command to `airbyte-cdk` CLI by devin-ai-integration[bot] · Pull Request #489 · airbytehq/airbyte-python-cdk

devin-ai-integration · 2025-04-16T22:17:42Z

Notes from AJ (@aaronsteers):

This PR allows you to quickly build a docker image for your connector. This is a follow-on to the previous PR that added connector test and secrets fetch capabilities. It uses most of the same practices as the official airbyte-ci pipeline, but it is faster and has some modest improvements.

See the base PR for more context on the CLI in general:

feat: add airbyte-cdk CLI with support for: connector test, secrets fetch and --version #493

Other notes:

For now, this always does a multi-arch build, so (assuming connector is sources-s3 you'll have images:
- airbyte/source-s3:dev-linux-amd64
- airbyte/source-s3:dev-linux-arm64
- airbyte/source-s3:dev (same as the ARM64 one, assuming for now that local dev is happening on an M-series Mac).
This uses uv instead of pip to do the install. This speeds up the builds significantly.
As of now, this only supports Python connectors.
- Manifest-only connectors will be added to this PR before merging.
- Java/Kotlin connectors may be added, but are not currently the priority.
As with the other airbyte-cdk CLI commands, you can run this statelessly using uvx airbyce-cdk image build. This doesn't require pre-installing the CDK, and will use the latest CDK version by default.
There's no current plan to replace/remove airbyte-ci for build+publish. We can discuss this, but currently, this functionality is only for dev/test workflows.

https://www.loom.com/share/5ca23374a5e24af1a4163597e3048154?sid=47e4cfa3-3a03-4eea-beb3-9a9dca3cb0ea

Notes From Devin:

Adds a new airbyte-cdk image build command to the Python CDK that allows building connector Docker images directly using Docker, without requiring the Dagger pipeline used in airbyte-ci build.

This PR:

Adds a new CLI entrypoint for airbyte-cdk image build
Implements Docker image building using the host's Docker daemon
Supports different connector types based on metadata and file structure
Verifies built images by running the spec command
Includes documentation for the new command

Requested by: Aaron ("AJ") Steers (aj@airbyte.io)
Link to Devin run: https://app.devin.ai/sessions/6a63c18a1da44463a87b2099bcd1c092

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

devin-ai-integration · 2025-04-16T22:17:49Z

Original prompt from Aaron:

@Devin - Can you analyze the airbyte-ci build command in the main airbyte repo? I'd like to migrate this logic into the Python CDK, under a new cli entrypoint and command called "airbyte-cdk build". Exclude the process of building base images.

devin-ai-integration · 2025-04-16T22:17:50Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment and CI monitoring

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

pyproject.toml

docs/cli/index.md

airbyte_cdk/cli/build/_run.py

…/docker for Dockerfile Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

docs/cli/build.md

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

airbyte_cdk/cli/build/_run.py

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

…s to models directory Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

…p of temporary files Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

…ailable Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

pyproject.toml

… fix entry point Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

pyproject.toml

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

pyproject.toml

…dd-build-command

aaronsteers · 2025-04-24T17:13:17Z

/autofix

Auto-Fix Job Info

This job attempts to auto-fix any linting or formating issues. If any fixes are made,
those changes will be automatically committed and pushed back to the PR.

Note: This job can only be run by maintainers. On PRs from forks, this command requires
that the PR author has enabled the Allow edits from maintainers option.

PR auto-fix job started... Check job output.

🟦 Job completed successfully (no changes).

aaronsteers · 2025-04-24T17:21:34Z

Replaced by:

feat: new CLI: airbyte-cdk image build #504

feat: Add build command to CDK CLI

c65d832

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

github-actions bot added the enhancement New feature or request label Apr 16, 2025

fix: Update formatting for build command files

bd7a765

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

devin-ai-integration bot changed the title ~~feat: Add build command to CDK CLI (do not merge)~~ feat(cli): Add build command to CDK CLI (do not merge) Apr 16, 2025

aaronsteers marked this pull request as draft April 16, 2025 22:23

devin-ai-integration bot changed the title ~~feat(cli): Add build command to CDK CLI (do not merge)~~ feat(cli): Add build command to CDK CLI Apr 16, 2025

devin-ai-integration bot marked this pull request as ready for review April 16, 2025 22:29

devin-ai-integration bot and others added 5 commits April 17, 2025 18:33

feat: Add Click-based CLI interface

2a32e27

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

chore: Update poetry.lock

653bda1

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

fix: Add type annotations to CLI functions

9510f2d

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

feat: Always build for both AMD64 and ARM64 architectures

84d649b

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

fix: Format code with ruff

f3e1743

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

aaronsteers reviewed Apr 17, 2025

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

aaronsteers reviewed Apr 17, 2025

View reviewed changes

docs/cli/index.md Outdated Show resolved Hide resolved

airbyte_cdk/cli/build/_run.py Outdated Show resolved Hide resolved

airbyte_cdk/cli/build/_run.py Outdated Show resolved Hide resolved

devin-ai-integration bot and others added 3 commits April 17, 2025 19:11

fix: Address PR comments - remove infer_connector_language, use build…

2bc8848

…/docker for Dockerfile Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

fix: Update Dockerignore to use * pattern and fix formatting

ca1bbbe

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

fix: Format all files with ruff

373f263

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

aaronsteers reviewed Apr 17, 2025

View reviewed changes

docs/cli/build.md Outdated Show resolved Hide resolved

devin-ai-integration bot and others added 2 commits April 17, 2025 19:28

docs: Move build.md content to __init__.py docstring

430dd8d

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

docs: Update __init__.py docstring with build command documentation

fe946ac

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

aaronsteers reviewed Apr 17, 2025

View reviewed changes

airbyte_cdk/cli/build/_run.py Outdated Show resolved Hide resolved

aaronsteers and others added 6 commits April 17, 2025 12:51

Delete airbyte_cdk/cli/entrypoint/run.py

af3e56e

fix: Clean up duplicate code in _run.py and update imports

7b6dc5d

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

refactor: Move build functionality to utils/docker and metadata model…

2b1db11

…s to models directory Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

fix: Remove unsupported --ignorefile flag and implement proper cleanu…

4d45bab

…p of temporary files Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

fix: Add fallback to single platform build when multi-platform not av…

7e68925

…ailable Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

fix: Update dockerignore to include source_* directories

df4b315

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

coderabbitai bot approved these changes Apr 18, 2025

View reviewed changes

aaronsteers reviewed Apr 18, 2025

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

fix: Address PR feedback - fix typo in __all__, update docstring, and…

63cbc9b

… fix entry point Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

aaronsteers reviewed Apr 18, 2025

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

fix: Correct CLI entry point in pyproject.toml

c8f91cf

Co-Authored-By: Aaron <AJ> Steers <aj@airbyte.io>

aaronsteers reviewed Apr 18, 2025

View reviewed changes

pyproject.toml Outdated Show resolved Hide resolved

Apply suggestions from code review

71652ef

aaronsteers force-pushed the devin/1744841809-add-build-command branch from d2c9fc9 to 71652ef Compare April 18, 2025 04:34

aaronsteers added 2 commits April 17, 2025 21:34

add-back the help text

78b173c

refactor, tidy up

598d454

aaronsteers mentioned this pull request Apr 18, 2025

feat: add airbyte-cdk CLI with support for: connector test, secrets fetch and --version #493

Merged

aaronsteers added 12 commits April 18, 2025 18:31

Merge branch 'aj/feat/add-standard-tests-cli' into devin/1744841809-a…

bcf5d9f

…dd-build-command

refactor docker implementation

871cb71

toggle default arch, fix return value for verify step

42af39d

better err handling

964a215

clean up

d7f9c7b

tidy docstring

3e9183b

ruff fix

6ea0623

poe lock

ea5fd85

fix mypy

0229421

fix base image

297a98c

use uv instead of pip for installs

f7b19e1

use rich click

7a87b62

aaronsteers changed the title ~~feat(cli): Add build command to CDK CLI~~ feat(cli): Add image build command to airbyte-cdk CLI Apr 21, 2025

refactor, use build args

f7e4ddf

aaronsteers added 2 commits April 24, 2025 10:16

improve language detection

f323849

remove java image option

51301d4

aaronsteers mentioned this pull request Apr 24, 2025

feat: new CLI: airbyte-cdk image build #504

Merged

aaronsteers closed this Apr 24, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cli): Add `image build` command to `airbyte-cdk` CLI#489

feat(cli): Add `image build` command to `airbyte-cdk` CLI#489
devin-ai-integration[bot] wants to merge 53 commits intoaj/feat/add-standard-tests-clifrom
devin/1744841809-add-build-command

devin-ai-integration bot commented Apr 16, 2025 •

edited by aaronsteers

Loading

Uh oh!

devin-ai-integration bot commented Apr 16, 2025

Uh oh!

devin-ai-integration bot commented Apr 16, 2025 •

edited by aaronsteers

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aaronsteers commented Apr 24, 2025 •

edited by github-actions bot

Loading

Uh oh!

aaronsteers commented Apr 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

devin-ai-integration bot commented Apr 16, 2025 • edited by aaronsteers Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes from AJ (@aaronsteers):

Notes From Devin:

Uh oh!

devin-ai-integration bot commented Apr 16, 2025

Uh oh!

devin-ai-integration bot commented Apr 16, 2025 • edited by aaronsteers Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 Devin AI Engineer

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aaronsteers commented Apr 24, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aaronsteers commented Apr 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

devin-ai-integration bot commented Apr 16, 2025 •

edited by aaronsteers

Loading

devin-ai-integration bot commented Apr 16, 2025 •

edited by aaronsteers

Loading

aaronsteers commented Apr 24, 2025 •

edited by github-actions bot

Loading