Skip to main content
55 questions
Newest Active Bountied Unanswered
Filter by
Sorted by
Tagged with
1 vote
0 answers
86 views

I have got two BigQuery tables: bigquery:ssh-test-project-01.SSh_Dataset_03.SSh_BgQ_Src_01 and bigquery:ssh-test-project-01.SSh_Dataset_03.SSh_BgQ_Dst_BgQ_01. Using Data Transfer with SQL: "...
Sergey Shabalov's user avatar
  • 33
0 votes
0 answers
135 views

I attached some "Column Tags" and "Business terms" using Dataplex and my goal is to fetch the table/column data with the tags information and business terms, is there a way to do ...
Tayzer Damasceno's user avatar
  • 312
0 votes
1 answer
207 views

I want to create Data Catalog tags for specific row which I process in Data Fusion - e.g. Warning tag to underline that specific row not completely passed all validations of pipeline, but it's said ...
Fagoter's user avatar
  • 569
0 votes
1 answer
290 views

I have actually gotten things to work properly (using the policy tag), yet I still don't understand the difference between the data catalog policy tag and the data catalog tag. To be specific, I'm ...
Mike Williamson's user avatar
  • 4,140
1 vote
1 answer
170 views

We are sharing some BQ datasets with a third party (outside our org) using analytics hub. So, the subscriber is able to create a linked dataset in their project and any queries they run are billed to ...
racerX's user avatar
  • 1,130
0 votes
1 answer
1k views

I have a dataset and a table in google big query (BQ). For the dataset, I can add description, and for the table I can add description and column policy tags to control column level access (I am ...
racerX's user avatar
  • 1,130
1 vote
0 answers
310 views

I have generated tag templates with Terraform, intending to associate them with various datasets in GCP. Despite my efforts to identify a connection between the datasets and the tag templates, I was ...
Ahmed sherif's user avatar
  • 63
1 vote
0 answers
207 views

We have a policy tag applied to many columns in BigQuery. Is there anyway to retrieve all the columns that tag applies to?
Hue Truc's user avatar
  • 27
1 vote
1 answer
226 views

Planning to build a data platform with compute as Google Cloud Dataproc storing the data in delta tables (Deltalake). Currently exploring the data catalog available in GCP stack along with open source ...
steve's user avatar
  • 305
0 votes
1 answer
381 views

I'm trying to create a tag template in Google Data Catalog using Terraform. Once created - the tag template's visibility is set to "Private". Looking for a way to set the visibility to ...
Dina Kleper's user avatar
  • 2,101
0 votes
0 answers
273 views

We run 2 queries from a table in 2 scenarios: We run a query to create a table from selecting data of another table (ex: ABC), this table ABC has no any policy tags (data masking rules) on any ...
Hue Truc's user avatar
  • 27
0 votes
1 answer
530 views

I have a table in BQ and I am trying to attach tags to columns depending on the prefix. For ex., all the columns which start with ABC_ have to be tagged with the Private Info tag. I have written below ...
Ananya Dwivedi's user avatar
  • 33
0 votes
2 answers
187 views

Is there any way(with query/python method) to get all the columns a user has in bigquery table? my requirement is userA: TableA.column1, TableA.column2,TableA.column4 userB: TableA.column1, TableA....
develop's user avatar
  • 95
0 votes
0 answers
297 views

Is it possible to also register/add metadata of Kafka topics/producers/consumers and on-premise databases etc in GCP data catalog/Dataplex? Could someone please point me to some examples how to do the ...
Ashok's user avatar
  • 5
0 votes
1 answer
100 views

I have a tag template in data catalog with name 'data check' and i need to add multiple fields to this template using python code by reading from a text file the text file consists the following name,...
Ricky's user avatar
  • 1
0 votes
1 answer
237 views

what is the GCP CLI Command to remove (detach) a Data Catalog Tag from a BigQuery Dataset, and also CLI Command to Update Tag. I am able to do it manually how to do it using Cloud Shell CLI gcloud ...
DSK's user avatar
  • 1
0 votes
1 answer
445 views

Could anyone please share ETL tag template in GCP data catalog? I'd like to refresh a tag value with its ETL status every time a BigQuery table is updated. I'm trying to use gcloud commands to create ...
Muthu lakshmi's user avatar
  • 1
0 votes
2 answers
1k views

I'm on exploration Dataplex API with Python in Google Documentation, there is documentation to Get Lake, Zone, Assets, etc. I've explored that documentation, but I didn't find any documentation ...
MADFROST's user avatar
  • 1,813
0 votes
1 answer
196 views

I would like to update Data Catalog tags through airflow CloudDataCatalog operators and schedule the DAG to update the tag values daily. This is to capture the ETL status on a BigQuery table and feed ...
Muthu lakshmi's user avatar
  • 1
3 votes
1 answer
3k views

I'm trying to update attached tag to tables in data catalog using python : enter image description here And i recieve this error : enter image description here E0102 16:29:36.671000000 13508 src/core/...
Mohamed BOUZIANE's user avatar
  • 31
3 votes
1 answer
615 views

I've been wondering if it is possible to have versions of schema in GCP Data Catalog Service? Or maybe advice on how you deal with Data Catalog entries when schema is changed (e.g. in CloudSQL, GCS ...
bozhyte's user avatar
  • 37
1 vote
1 answer
887 views

As a part of data governance, we have created Taxonomies, Policy Tags Using "Python API". And I am trying to Assign Policy Tags to Columns [Name, Age] for a table Project.Dataset.TMP_TBL. ...
VIJAY BURRA's user avatar
  • 11
0 votes
1 answer
843 views

I need help using the template template created in Google Data Catalog through Python programming to tag several BigQuery tables using the same template template created. I don't know how to use the ...
dsrebechi's user avatar
  • 1
0 votes
1 answer
2k views

Experts, I'm trying to create a Taxonomy in Google Data Catalog using Apache Airflow. When I trigger the job in Airflow, it fails with below error. Error > google.api_core.exceptions....
learner's user avatar
  • 155
0 votes
1 answer
585 views

I'm doing a test to use GCP Data Catalog (DataPlex) to document tables from Big Query. I'm doing in a dev project, in the future, I'd like transfer data that I documented (description of columns, ...
Robson Pelegrini's user avatar
  • 13
1 vote
1 answer
575 views

Following the link I found in Google, I'm trying to do a sample setup to publish message in pubsub and load the same into bigquery table using dataflow sql. But when I create dataflow job am getting ...
Vanaja Jayaraman's user avatar
  • 791
1 vote
1 answer
1k views

I have been trying to implement column level access policies on a Big Query external table. I have tried to create a policy tag via GCP console as well as REST API but getting an error "Policy ...
Despicable me's user avatar
  • 772
2 votes
1 answer
755 views

I am trying to attach tags to data catalog custom entries. I am trying to create a python function to perform data catalog operations i.e. create/delete custom entries, create/delete tag templates, ...
Trishit Ghosh's user avatar
  • 255
1 vote
1 answer
312 views

When searching using Google's data catalog, a scope parameter is required and one of its fields is includeProjectIds. According to the documentation, this parameter represents: The list of project ...
Peter Boone's user avatar
  • 1,368
0 votes
1 answer
175 views

I'm trying to filter the gcp data catalog, in which I try to filter column descriptions, using the description qualifiers: student name attribute. Unfortunately, no return. other qualifiers that I ...
Wesley Rodrigues's user avatar
  • 1
0 votes
0 answers
1k views

I am trying to use Nodejs Library for Google Cloud Data Catalog. For one project, it is working perfectly fine but for others it is giving me Project does not exist error, while I can see the project ...
user2466031's user avatar
  • 11
0 votes
1 answer
1k views

I'm creating tag templates programmatically and it doesn't appear in the section "Tag Templates" (the creation of each tag template was successful because I can attach it in each table), but ...
MarcoGallina's user avatar
  • 1
0 votes
2 answers
2k views

I have multiple tables in BigQuery. I also have a tag template. Is there a way to attach this tag template and fill the details programmatically with python to any table using the google.cloud....
loki's user avatar
  • 972
1 vote
2 answers
541 views

I ran a DLP job saving results to Data Catalog and would like to filter the entries in Data Catalog where the standard tag template (Data Loss Prevention Tags) has a value Contains DLP findings: true. ...
Yann's user avatar
  • 2,572
0 votes
1 answer
690 views

I've been dealing with this issue for weeks. Does anyone know how to create column tags in Data Catalog from DLP results? I wanted to run inspection job on DLP and then tag the inspected tables with ...
N Mohtar's user avatar
  • 1
0 votes
1 answer
460 views

What roles should be properly assigned to a group/service account if we would like to be able to update table descriptions in a centralized dataset and other labels for Data Catalog? We currently have ...
mtryingtocode's user avatar
  • 969
1 vote
1 answer
3k views

We are exploring the use of BigQuery policy tags for implementing column level security. We have discovered that when a column that a user does not have access to is included in a GROUP BY clause then ...
jamiet's user avatar
  • 12.8k
1 vote
1 answer
682 views

I want to use DLP to inspect my tables in BigQuery, and then write the findings to policy tags on the columns of the table. For example, I have a (test) table that contains data including an email ...
Jack Gray's user avatar
  • 11
0 votes
0 answers
120 views

I am working on developing a datacatalog integration pipeline in google cloud composer. Since I am using composer version composer-1.12.4-airflow-1.10.10, its not allowing me to install google-cloud-...
MI Haque's user avatar
  • 51
0 votes
1 answer
184 views

Is it possible to use the Google Cloud Data Catalog to catalog an Azure Data Lake, Blob storage or Azure SQL Database?
Sandy's user avatar
  • 105
1 vote
2 answers
663 views

In Google Data Catalog I have tried to create a new entry group, then created a fileset entry where bucket and file pattern is configured. I have not defined a schema on purpose in the Fileset, as I ...
user2821136's user avatar
  • 13
0 votes
1 answer
917 views

I'm getting an error trying of use the code of this link: Data Catalog Example. At the step 4, just copy pasting all the code provided, authenticating into my GCP Project and testing it. Everything is ...
Emilio's user avatar
  • 13
2 votes
0 answers
1k views

I am attaching a tag-template to a column of a BigQuery table. For this, I am using Terraform and I have just recreated the code in the terraform documentation. resource "...
AudronFS's user avatar
  • 149
0 votes
1 answer
6k views

Is there a way to assign policy tags globally in BigQuery? Rather than entering every single table and putting it there?
Norfecaz's user avatar
  • 47
0 votes
1 answer
244 views

I've added a new field to an existing GCP Data Catalog template which by default adds it to the bottom. I'd like for this new field to appear higher up. Is there some way I'm not seeing to rearrange ...
John Rager's user avatar
  • 3
0 votes
0 answers
80 views

I am using the connector suggested in some GCP documentation to sync metadata from Tableau to Google Data Catalog (https://github.com/GoogleCloudPlatform/datacatalog-connectors-bi). The sync process ...
Enrique Dev's user avatar
  • 119
0 votes
1 answer
719 views

What is the best practice to get a company wide (one or more organisations each with multiple folders and projects) INTO one central and all metadata contained data catalog ? (if "multiple orgs&...
InLaw's user avatar
  • 2,747
0 votes
1 answer
327 views

Take the public github dataset as an example SELECT * FROM `bigquery-public-data.github_repos.commits` LIMIT 2 There are column names like difference.old_mode via search: column:difference....
InLaw's user avatar
  • 2,747
1 vote
1 answer
815 views

How to execute a wildcard/RegEx search in Data Catalog (Google Cloud Platform) ? It would make sense to search metadata across column names and tag attributes (and there values). The current ...
InLaw's user avatar
  • 2,747
1 vote
1 answer
903 views

I am able to search for tag templates using the Java Google Data Catalog library, and do sub search on the tag name to find the big query tables the tags are used, but I can't figure out how to find ...
Scott Nichols's user avatar
  • 21

1
2