A cheap, fast way to validate the marts/reports by reading the same Iceberg
tables Athena serves — straight from S3, in-process, free (no Athena scan cost).
It's an ad-hoc mirror of common/validation.py and the dbt tests.
content_engagement_daily.sessions_count== a fresh recompute fromfact_view_sessions(0 mismatches).completion_ratewithin [0, 1].genre_mix_daily.pct_of_daysums to ~100 per day.- No duplicate grain in the content mart.
- Summary: top titles by watch time.
brew install duckdb # one-time
duckdb < analytics/duckdb/validate.sqlCreds come from your ~/.aws (the credential_chain secret). Proven against the
real warehouse — all four checks pass.
A read-only DuckDB box (gated, apply-on-demand):
cd infra/terraform
tofu apply -var='enable_duckdb_ec2=true'
tofu output duckdb_ec2_instance # => i-0...
aws ssm start-session --target i-0... --region us-east-1
# on the box:
duckdb < /opt/streaming/analytics/duckdb/validate.sql
# done:
aws ec2 stop-instances --instance-ids i-0... --region us-east-1 # ~$0 stopped- Cross-region: the warehouse is us-west-2; the
CREATE SECRET ... REGION 'us-west-2'handles S3. The GlueATTACHpath is not used — the Glue catalog is us-east-1 but the data is us-west-2, and a single secret region can't serve both (DuckDB issue #265). So tables are read by their S3 root instead. - Snapshot resolution:
SET unsafe_enable_version_guessing = trueletsiceberg_scanfind the latest metadata from the table root (Glue-written tables have no version-hint). - Read-only by design: the EC2 instance role grants only S3 + Glue reads, so a validation query can never mutate the pipeline.