Is your feature request related to a problem? Please describe.
Currently, source in Feast is defined as the following in the FeatureSet spec
spec:
name: transaction
maxAge: 86400s
entities:
- name: transaction_id
features:
- name: amount
valueType: FLOAT
...
# Source definition
source:
kafkaSourceConfig:
bootstrapServers: mybroker:9092
topic: transaction
type: KAFKA
From management perspective, when we list all the sources we use in Feast, we sometimes want to be able to filter or group these sources according to the organisational structure, for instance, team managing the source, the country where the source data comes from.
Current source proto, however, does not allow us to provide such additional information.
Describe the solution you'd like
Add labels field to the Source proto. This is similar to the labels field we currently have in FeatureSet
https://github.com/feast-dev/feast/blob/master/protos/feast/core/FeatureSet.proto#L63
message Source {
// The kind of data source Feast should connect to in order to retrieve FeatureRow value
SourceType type = 1;
// Source specific configuration
oneof source_config {
KafkaSourceConfig kafka_source_config = 2;
}
// Labels are key-value pairs attached to a Source object. They are intended to help users
// organize Source objects in Feast. They do not directly imply semantics to the core system.
//
// Requirements for label keys and values:
// - Keys have a min length of 1 character and max length of 63 characters. Values can
// be empty and have a max length of 63 characters.
// - Can only contain lowercase letters, numeric characters, underscores and dashes. All
// characters must use UTF-8 encoding.
// - Keys can only start with a lowercase letter.
map<string, string> labels = 3;
}
Describe alternatives you've considered
N/A
Additional context
These labels information in the Source object can potentially be used to label ingestion jobs that are using the respective sources as well. In order to help organize ingestion jobs objects.
Is your feature request related to a problem? Please describe.
Currently,
sourcein Feast is defined as the following in the FeatureSet specFrom management perspective, when we list all the sources we use in Feast, we sometimes want to be able to filter or group these sources according to the organisational structure, for instance, team managing the source, the country where the source data comes from.
Current source proto, however, does not allow us to provide such additional information.
Describe the solution you'd like
Add
labelsfield to the Source proto. This is similar to the labels field we currently have in FeatureSethttps://github.com/feast-dev/feast/blob/master/protos/feast/core/FeatureSet.proto#L63
Describe alternatives you've considered
N/A
Additional context
These labels information in the Source object can potentially be used to label ingestion jobs that are using the respective sources as well. In order to help organize ingestion jobs objects.