Skip to content

Snowflake-batchsource.md

Latest commit

 

History

History
98 lines (67 loc) · 5.73 KB

Snowflake-batchsource.md

File metadata and controls

98 lines (67 loc) · 5.73 KB

Snowflake Batch Source

Description

Reads data source from Snowflake tables.

Configuration

Basic

Label: Label for UI.

Reference Name: Unique name for lineage.

Account Name: Snowflake account name and region and possibly a cloud specifier. (Part of the URL you use to log in to Snowflake, minus the "snowflakecomputing.com"). E.g. "myaccount.us-central1.gcp".

Database: Database that contains the target table.

Schema: Schema that contains the target table.

Warehouse: Warehouse that provides the compute resources.

Role: Role to use (e.g. ACCOUNTADMIN).

Import Query Type - Method used to retrieve schema from the source.

  • Table Name: The name of the table to retrieve the schema.
  • Import Query: Query for data import.

Credentials

Username: User identity to use to connect to your Snowflake account.

Password: Password to use to connect to your Snowflake account. Not necessary for key pair or OAuth2 authentication.

Key Pair Authentication

Key Pair Authentication Enabled: If true, plugin will perform key pair authentication.

Private Key: Private key contents.

Key File Passphrase: Passphrase for the private key file.

OAuth2

To use OAuth2, user must create a snowflake security integration for it. For more info see Introduction to OAuth in Snowflake

OAuth2 Enabled: If true, plugin will perform OAuth2 authentication.

Client ID: Client id obtained via system function SYSTEM$SHOW_OAUTH_CLIENT_SECRETS

Client Secret: Client id obtained via system function SYSTEM$SHOW_OAUTH_CLIENT_SECRETS

Refresh Token: Token used to receive accessToken, which is end product of OAuth2. Must be generated by user.

Advanced

Maximum Split Size: Maximum split size specified in bytes. If value is set to 0, then result is not split by snowflake.

Connection Arguments: List of arbitrary string tag/value pairs as connection arguments. See: JDBC Driver Connection String.

Data Types Mapping

Snowflake Data Types CDAP Schema Data Type Comment
NUMBER decimal Default precision and scale are (38,0).
DECIMAL decimal Synonymous with NUMBER.
NUMERIC decimal Synonymous with NUMBER.
INT, INTEGER, BIGINT, SMALLINT decimal Synonymous with NUMBER, except that precision and scale cannot be specified (i.e. always defaults to NUMBER(38, 0)).
FLOAT, FLOAT4, FLOAT8 double Snowflake uses double-precision (64 bit) IEEE 754 floating point numbers.
DOUBLE double Synonymous with FLOAT.
DOUBLE PRECISION double Synonymous with FLOAT.
REAL double Synonymous with FLOAT.
VARCHAR string Default (and maximum) is 16,777,216 bytes.
CHAR, CHARACTER string Synonymous with VARCHAR except default length is VARCHAR.
STRING string Synonymous with VARCHAR.
TEXT string Synonymous with VARCHAR.
BINARY bytes
VARBINARY bytes Synonymous with BINARY.
BOOLEAN boolean
DATE date
DATETIME timestamp Alias for TIMESTAMP_NTZ
TIME time
TIMESTAMP timestamp Alias for one of the TIMESTAMP variations (TIMESTAMP_NTZ by default).
TIMESTAMP_LTZ timestamp TIMESTAMP with local time zone; time zone, if provided, is not stored.
TIMESTAMP_NTZ timestamp TIMESTAMP with no time zone; time zone, if provided, is not stored.
TIMESTAMP_TZ timestamp TIMESTAMP with time zone.
VARIANT string A tagged universal type, which can store values of any other type, including OBJECT and ARRAY, up to a maximum size of 16 MB compressed.
OBJECT string This will return a json with the data
ARRAY string This will return a json with the data