Skip to content

Add Labels to Source Object #835

@davidheryanto

Description

@davidheryanto

Is your feature request related to a problem? Please describe.
Currently, source in Feast is defined as the following in the FeatureSet spec

spec:
  name: transaction
  maxAge: 86400s
  entities:
  - name: transaction_id
  features:
  - name: amount
    valueType: FLOAT
  ...
  # Source definition
  source:
    kafkaSourceConfig:
      bootstrapServers: mybroker:9092
      topic: transaction
    type: KAFKA

From management perspective, when we list all the sources we use in Feast, we sometimes want to be able to filter or group these sources according to the organisational structure, for instance, team managing the source, the country where the source data comes from.

Current source proto, however, does not allow us to provide such additional information.

Describe the solution you'd like
Add labels field to the Source proto. This is similar to the labels field we currently have in FeatureSet
https://github.com/feast-dev/feast/blob/master/protos/feast/core/FeatureSet.proto#L63

message Source {

    // The kind of data source Feast should connect to in order to retrieve FeatureRow value
    SourceType type = 1;

    // Source specific configuration
    oneof source_config {
        KafkaSourceConfig kafka_source_config = 2;
    }

    // Labels are key-value pairs attached to a Source object. They are intended to help users
    // organize Source objects in Feast. They do not directly imply semantics to the core system.
    //
    // Requirements for label keys and values:
    // - Keys have a min length of 1 character and max length of 63 characters. Values can
    //   be empty and have a max length of 63 characters.
    // - Can only contain lowercase letters, numeric characters, underscores and dashes. All
    //   characters must use UTF-8 encoding.
    // - Keys can only start with a lowercase letter.
    map<string, string> labels = 3;
}

Describe alternatives you've considered
N/A

Additional context
These labels information in the Source object can potentially be used to label ingestion jobs that are using the respective sources as well. In order to help organize ingestion jobs objects.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions