Skip to content

Snowflake Offline to Parquet export failure #3548

@adamschmidt

Description

@adamschmidt

Expected Behavior

Snowflake offline export to Parquet in S3 should successfully export the materialization data set, including timestamp fields.

Current Behavior

Export to S3 integration in Parquet format fails with the following message:

Error encountered when unloading to PARQUET: TIMESTAMP_TZ and LTZ types are not supported for unloading to Parquet. value get: TIMESTAMP_LTZ

This is a known issue with recommended guidance.

Steps to reproduce

  • Configure offline store in Snowflake
  • Attempt to run materialization to external destination (e.g. DynamoDB)
  • Confirm "temporary_" table is created in the offline schema and an attempt was made to write to the storage integration

Specifications

  • Version: 0.29.0+
  • Platform: LInux + EKS
  • Subsystem: Python, Snowflake, DynamoDB

Possible Solution

Two things will assist in resolving this issue:

  • update the data extraction to cast timestamp fields to string or convert to a UTC timestamp integer
  • add an optional configuration option to enable timestamp field treatment, default to None/False

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions