Expected Behavior
Sources are generic and extensible.
Current Behavior
Currently the sources data model in Feast Core is hardcoded specifically to KafkaSource.
Possible Solution
Make sure none of the columns in our data model is specific to a single source. The pattern we have followed thus far is to use Map<String,String> for Source, Runner, Store configurations. One way to solve this problem is to have source configuration stored as a serialized JSON map in a single column in the Source table.
Expected Behavior
Sources are generic and extensible.
Current Behavior
Currently the sources data model in Feast Core is hardcoded specifically to KafkaSource.
Possible Solution
Make sure none of the columns in our data model is specific to a single source. The pattern we have followed thus far is to use
Map<String,String>for Source, Runner, Store configurations. One way to solve this problem is to havesourceconfiguration stored as a serialized JSON map in a single column in the Source table.