bug: error synching cloudflare_dns_records #20393
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
The table cloudflare_dns_records in the Cloudflare plugin has the columns zone_id and zone_name.
The plugin is no longer properly synching the value of these two fields. They are both set to an empty string now.
This applies to both the JSON file and the Postgres output plugins.
Expected Behavior
The column zone_id has a valid zone ID that references the id column from the table cloudflare_zones.
CloudQuery (redacted) config
kind: source
spec:
name: "cloudflare"
path: "cloudquery/cloudflare"
version: "v9.9.10"
tables: ["cloudflare_access_applications", "cloudflare_access_groups", "cloudflare_accounts", "cloudflare_account_members", "cloudflare_dns_records", "cloudflare_zones"]
skip_dependent_tables: true
destinations: ["file_json", "pq_cloudflare"]
spec:
api_token: "REDACTED"
kind: destination
spec:
name: "pq_cloudflare"
registry: "cloudquery"
path: "cloudquery/postgresql"
version: "v8.5.3"
write_mode: "overwrite-delete-stale"
migrate_mode: "forced"
spec:
connection_string: "REDACTED"
kind: destination
spec:
name: "file_json"
path: "cloudquery/file"
version: "v5.2.4"
write_mode: "append"
spec:
path: "/tmp/cq_results/json/{{TABLE}}_-_{{UUID}}.{{FORMAT}}"
format: "json"Steps To Reproduce
Run a sync against a Cloudflare account that has at least one DNS zone with at least one record.
CloudQuery (redacted) logs
2025-03-17T12:03:45Z INF Loading spec(s) args=["./"] invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli
2025-03-17T12:03:47Z INF Plugin server listening address=/tmp/cq-PwyGqYKdsFxviVbP.sock invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cloudflare-source
2025-03-17T12:03:49Z INF Plugin server listening address=/tmp/cq-mrDiptPJbLwobQgL.sock invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli
2025-03-17T12:03:52Z INF Plugin server listening address=/tmp/cq-HuoNULaSrZpaknMl.sock invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli
2025-03-17T12:03:53Z INF started call grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2025-03-17T12:03:53Z grpc.time_ms=0.104 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cloudflare-source peer.address=@ protocol=grpc
2025-03-17T12:03:53Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2025-03-17T12:03:53Z grpc.time_ms=0.167 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cloudflare-source peer.address=@ protocol=grpc
2025-03-17T12:03:53Z INF started call grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2025-03-17T12:03:53Z grpc.time_ms=0.119 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:03:53Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2025-03-17T12:03:53Z grpc.time_ms=0.147 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:03:53Z INF started call grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2025-03-17T12:03:53Z grpc.time_ms=0.103 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:03:53Z INF finished call grpc.code=OK grpc.component=server grpc.method=GetVersions grpc.method_type=unary grpc.service=cloudquery.discovery.v1.Discovery grpc.start_time=2025-03-17T12:03:53Z grpc.time_ms=0.132 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:03:53Z INF Start sync destinations=["pq_cloudflare (cloudquery/postgresql@v8.5.3)","file_json (cloudquery/file@v5.2.4)"] invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli source="cloudflare (cloudquery/cloudflare@v9.9.10)" sync_time=2025-03-17T12:03:53Z transformers=[]
2025-03-17T12:03:53Z INF started call grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:03:53Z grpc.time_ms=0.006 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:03:53Z INF Initializing postgresql destination invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=pg-dest pgx_log_level="invalid level 0"
2025-03-17T12:03:53Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:03:53Z grpc.time_ms=60.13 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:03:53Z INF started call grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:03:53Z grpc.time_ms=0.011 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:03:53Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:03:53Z grpc.time_ms=0.333 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:03:53Z INF started call grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:03:53Z grpc.time_ms=0.01 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cloudflare-source peer.address=@ protocol=grpc
2025-03-17T12:03:54Z INF finished call grpc.code=OK grpc.component=server grpc.method=Init grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:03:53Z grpc.time_ms=831.439 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cloudflare-source peer.address=@ protocol=grpc
2025-03-17T12:03:54Z INF Start fetching resources destinations=["pq_cloudflare (cloudquery/postgresql@v8.5.3)","file_json (cloudquery/file@v5.2.4)"] invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli source="cloudflare (cloudquery/cloudflare@v9.9.10)"
2025-03-17T12:03:54Z INF started call grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:03:54Z grpc.time_ms=0.102 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cloudflare-source peer.address=@ protocol=grpc
2025-03-17T12:03:54Z INF started call grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:03:54Z grpc.time_ms=76.511 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:03:54Z INF started call grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:03:54Z grpc.time_ms=75.741 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:03:54Z INF top level table resolver started client=32a4668d7d4342354c9c6120e470589c:456356a3a870fc08a2acbd6ee1c3d91c invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cloudflare-source table=cloudflare_dns_records
[...]
2025-03-17T12:04:22Z INF table sync finished client=32a4668d7d4342354c9c6120e470589c duration_ms=25066.915702 errors=0 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cloudflare-source resources=37 table=cloudflare_zones
2025-03-17T12:04:22Z INF table sync finished client=32a4668d7d4342354c9c6120e470589c:a14e3f48bbf201bbd654ba73606a9cfa duration_ms=28262.078023 errors=0 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cloudflare-source resources=339 table=cloudflare_dns_records
2025-03-17T12:04:22Z INF finished call grpc.code=OK grpc.component=server grpc.method=Sync grpc.method_type=server_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:03:54Z grpc.time_ms=28522.703 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cloudflare-source peer.address=@ protocol=grpc
2025-03-17T12:04:23Z INF Sync summary invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli summary={"cli_version":"6.8.2","destination_errors":0,"destination_name":"pq_cloudflare","destination_path":"cloudquery/postgresql","destination_version":"v8.5.3","destination_warnings":0,"resources":2056,"source_errors":0,"source_name":"cloudflare","source_path":"cloudquery/cloudflare","source_version":"v9.9.10","source_warnings":0,"sync_id":"1d197272-27a4-43bc-bf25-ec0e21c71eb0","sync_time":"2025-03-17T12:03:53.140426Z"}
2025-03-17T12:04:23Z INF Sync summary invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli summary={"cli_version":"6.8.2","destination_errors":0,"destination_name":"file_json","destination_path":"cloudquery/file","destination_version":"v5.2.4","destination_warnings":0,"resources":2056,"source_errors":0,"source_name":"cloudflare","source_path":"cloudquery/cloudflare","source_version":"v9.9.10","source_warnings":0,"sync_id":"1d197272-27a4-43bc-bf25-ec0e21c71eb0","sync_time":"2025-03-17T12:03:53.140426Z"}
2025-03-17T12:04:23Z INF finished call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:03:54Z grpc.time_ms=29726.404 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:04:23Z INF started call grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:04:23Z grpc.time_ms=0.011 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:04:23Z INF finished call grpc.code=OK grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:04:23Z grpc.time_ms=0.187 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:04:23Z INF finished call grpc.code=OK grpc.component=server grpc.method=Write grpc.method_type=client_stream grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:03:54Z grpc.time_ms=29727.783 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:04:23Z INF started call grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:04:23Z grpc.time_ms=0.012 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:04:23Z INF finished call grpc.code=OK grpc.component=server grpc.method=Close grpc.method_type=unary grpc.service=cloudquery.plugin.v3.Plugin grpc.start_time=2025-03-17T12:04:23Z grpc.time_ms=0.05 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli peer.address=@ protocol=grpc
2025-03-17T12:04:23Z INF Sync summary duration=30s errors=0 invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli resources=2056 result="Sync completed successfully" warnings=0
2025-03-17T12:04:23Z INF End sync destinations=["pq_cloudflare (cloudquery/postgresql@v8.5.3)","file_json (cloudquery/file@v5.2.4)"] invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli source="cloudflare (cloudquery/cloudflare@v9.9.10)" sync_time=2025-03-17T12:03:53Z transformers=[]
2025-03-17T12:04:23Z INF waiting for source plugin to terminate invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli
2025-03-17T12:04:23Z INF Got stop signal. Plugin server shutting down address=/tmp/cq-mrDiptPJbLwobQgL.sock invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli signal=interrupt
2025-03-17T12:04:23Z INF waiting for source plugin to terminate invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli
2025-03-17T12:04:23Z INF Got stop signal. Plugin server shutting down address=/tmp/cq-HuoNULaSrZpaknMl.sock invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli signal=interrupt
2025-03-17T12:04:23Z INF waiting for source plugin to terminate invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cli
2025-03-17T12:04:23Z INF Got stop signal. Plugin server shutting down address=/tmp/cq-PwyGqYKdsFxviVbP.sock invocation_id=1d197272-27a4-43bc-bf25-ec0e21c71eb0 module=cloudflare-source signal=interrupt
CloudQuery version
v6.8.2
Additional Context
No response
Pull request (optional)
- I can submit a pull request