Skip to content

Commit a444692

Browse files
YassinNouh21YassinNouh21
authored
feat: Add integration tests for dbt import (#5899)
* Add comprehensive integration tests for dbt import feature - Add 82 integration tests covering dbt manifest parsing, mapper, codegen, and CLI - Add real dbt project with DuckDB for testing - Add GitHub Actions workflow for dbt integration tests - Fix entity column handling: codegen now includes entity columns in schema to match mapper behavior (FeatureView.__init__ expects them) - Fix type mapping: Int32→ValueType.INT32, Float32→ValueType.FLOAT (instead of widening to INT64/DOUBLE) - Add multi-entity column support tests - Add comprehensive test coverage for edge cases, validation errors, special characters, and CLI commands Signed-off-by: yassinnouh21 <yassinnouh21@gmail.com> * Add ok-to-test label check to dbt integration workflow - Add PR trigger types (opened, synchronize, labeled) - Add label check condition requiring ok-to-test/approved/lgtm - Prevents resource spin-up for every PR Signed-off-by: yassinnouh21 <yassinnouh21@gmail.com> --------- Signed-off-by: yassinnouh21 <yassinnouh21@gmail.com>
1 parent 19f9bb8 commit a444692

File tree

18 files changed

+1645
-3
lines changed

18 files changed

+1645
-3
lines changed

‎.github/workflows/dbt-integration-tests.yml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
name: dbt-integration-tests
2+
3+
# Run dbt integration tests on PRs
4+
on:
5+
pull_request:
6+
types:
7+
- opened
8+
- synchronize
9+
- labeled
10+
paths:
11+
- 'sdk/python/feast/dbt/**'
12+
- 'sdk/python/tests/integration/dbt/**'
13+
- 'sdk/python/tests/unit/dbt/**'
14+
- '.github/workflows/dbt-integration-tests.yml'
15+
16+
jobs:
17+
dbt-integration-test:
18+
if:
19+
((github.event.action == 'labeled' && (github.event.label.name == 'approved' || github.event.label.name == 'lgtm' || github.event.label.name == 'ok-to-test')) ||
20+
(github.event.action != 'labeled' && (contains(github.event.pull_request.labels.*.name, 'ok-to-test') || contains(github.event.pull_request.labels.*.name, 'approved') || contains(github.event.pull_request.labels.*.name, 'lgtm')))) &&
21+
github.event.pull_request.base.repo.full_name == 'feast-dev/feast'
22+
runs-on: ubuntu-latest
23+
strategy:
24+
matrix:
25+
python-version: ["3.11", "3.12"]
26+
env:
27+
PYTHON: ${{ matrix.python-version }}
28+
steps:
29+
- uses: actions/checkout@v4
30+
31+
- name: Setup Python
32+
uses: actions/setup-python@v5
33+
with:
34+
python-version: ${{ matrix.python-version }}
35+
architecture: x64
36+
37+
- name: Install the latest version of uv
38+
uses: astral-sh/setup-uv@v5
39+
with:
40+
enable-cache: true
41+
42+
- name: Install dependencies
43+
run: make install-python-dependencies-ci
44+
45+
- name: Install dbt and dbt-duckdb
46+
run: |
47+
uv pip install --system dbt-core dbt-duckdb
48+
49+
- name: Run dbt integration tests
50+
run: make test-python-integration-dbt
51+
52+
- name: Minimize uv cache
53+
run: uv cache prune --ci

‎Makefile‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -187,6 +187,28 @@ test-python-integration-container: ## Run Python integration tests using Docker
187187
python -m pytest -n 8 --integration sdk/python/tests \
188188
) || echo "This script uses Docker, and it isn't running - please start the Docker Daemon and try again!";
189189

190+
test-python-integration-dbt: ## Run dbt integration tests
191+
@echo "Running dbt integration tests..."
192+
@cd sdk/python/tests/integration/dbt/test_dbt_project && \
193+
echo "Installing dbt dependencies..." && \
194+
dbt deps && \
195+
echo "Building dbt models..." && \
196+
dbt build
197+
@cd sdk/python/tests/integration/dbt && \
198+
echo "Setting up Feast project..." && \
199+
mkdir -p feast_repo/data && \
200+
echo "project: feast_dbt_test\nregistry: data/registry.db\nprovider: local\nonline_store:\n type: sqlite\n path: data/online_store.db" > feast_repo/feature_store.yaml
201+
@cd sdk/python/tests/integration/dbt/feast_repo && \
202+
echo "Testing feast dbt import..." && \
203+
feast dbt import -m ../test_dbt_project/target/manifest.json -e driver_id -d file --tag feast && \
204+
echo "Verifying Feast objects..." && \
205+
feast feature-views list && \
206+
feast entities list
207+
@cd sdk/python && \
208+
echo "Running pytest integration tests..." && \
209+
python -m pytest tests/integration/dbt/test_dbt_integration.py -v --tb=short
210+
@echo "✓ dbt integration tests completed successfully!"
211+
190212
test-python-universal-spark: ## Run Python Spark integration tests
191213
PYTHONPATH='.' \
192214
FULL_REPO_CONFIGS_MODULE=sdk.python.feast.infra.offline_stores.contrib.spark_repo_configuration \

‎sdk/python/feast/dbt/codegen.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -248,7 +248,9 @@ def generate(
248248
if not entity_cols:
249249
raise ValueError("At least one entity column must be specified")
250250

251-
excluded = set(entity_cols) | {self.timestamp_field}
251+
# Note: entity columns should NOT be excluded - FeatureView.__init__
252+
# expects entity columns to be in the schema and will extract them
253+
excluded = {self.timestamp_field}
252254
if exclude_columns:
253255
excluded.update(exclude_columns)
254256

‎sdk/python/feast/dbt/mapper.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,9 @@
2929
# Mapping from FeastType to ValueType for entity value inference
3030
FEAST_TYPE_TO_VALUE_TYPE: Dict[FeastType, ValueType] = {
3131
String: ValueType.STRING,
32-
Int32: ValueType.INT64,
32+
Int32: ValueType.INT32,
3333
Int64: ValueType.INT64,
34-
Float32: ValueType.DOUBLE,
34+
Float32: ValueType.FLOAT,
3535
Float64: ValueType.DOUBLE,
3636
Bool: ValueType.BOOL,
3737
Bytes: ValueType.BYTES,

‎sdk/python/tests/integration/dbt/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Integration tests for dbt import functionality

‎sdk/python/tests/integration/dbt/conftest.py‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
"""
2+
Conftest for dbt integration tests.
3+
4+
This is a standalone conftest that doesn't depend on the main Feast test infrastructure.
5+
"""
6+
7+
from pathlib import Path
8+
9+
import pytest
10+
11+
# This conftest is minimal and doesn't import the main feast conftest
12+
# to avoid complex dependency chains for dbt-specific tests
13+
14+
# Path to the test dbt project manifest
15+
TEST_DBT_PROJECT_DIR = Path(__file__).parent / "test_dbt_project"
16+
TEST_MANIFEST_PATH = TEST_DBT_PROJECT_DIR / "target" / "manifest.json"
17+
18+
19+
def pytest_collection_modifyitems(config, items): # noqa: ARG001
20+
"""
21+
Skip dbt integration tests if manifest.json doesn't exist.
22+
23+
These tests require running 'dbt build' first to generate the manifest.
24+
The dbt-integration-test workflow handles this, but regular unit test
25+
runs don't, so we skip them to avoid failures.
26+
"""
27+
if not TEST_MANIFEST_PATH.exists():
28+
skip_marker = pytest.mark.skip(
29+
reason="dbt manifest.json not found - run 'dbt build' first or use dbt-integration-test workflow"
30+
)
31+
for item in items:
32+
item.add_marker(skip_marker)

‎sdk/python/tests/integration/dbt/pytest.ini‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
[pytest]
2+
# Prevent loading parent conftest.py files which may import heavy dependencies
3+
# like ray that are not needed for dbt integration tests
4+
norecursedirs = ..
5+
asyncio_mode = auto
6+
7+
# Test markers
8+
markers =
9+
dbt: dbt integration tests

0 commit comments

Comments
 (0)