Skip to content

Failed to configure Confluent Schema Registry Cleanup Policies #15124

@manuschillerdev

Description

@manuschillerdev

Describe the bug
With https://github.com/datahub-project/datahub/blob/master/datahub-upgrade/src/main/java/com/linkedin/datahub/upgrade/system/kafka/ConfluentSchemaRegistryCleanupPolicyStep.java#L70 a config step for non-datahub exclusive topics was introduced by setting up cleanup.policy to compactfor the schema-registry topic _schemas. In shared environments, DataHub typically does not have the permissions to alter such topics.

This makes the system-update job fail in our scenario with an authorization error:

Image

To Reproduce
Run self-hosted datahub (https://github.com/acryldata/datahub-helm/releases/tag/datahub-0.7.3) in an environment, where the DataHub principal does not have permissions to configure confluents _schemas topic.

Expected behavior
I'd expect the code to at least check the setting before applying it. DataHub may often have describe permissions on all topics when using the kafka-integration. The default of confluent's schema-registry on cleanup.policy already is compact: https://docs.confluent.io/platform/current/schema-registry/security/index.html#kafka-store - so this should be the common case.

Screenshots
irrelevant

Desktop (please complete the following information):
irrelevant

Additional context
irrelevant

Metadata

Metadata

Assignees

Labels

bugBug report

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions