@pradithya @woop @budi This is the PR I mentioned to you in chat. Take a look if you're interested in the implementation details. It's still needs a few changes.

eg

• change the import spec options field to be sourceOptions, but I want to wait for another PR so we have less merge conflicts.
• group by entityname +entitykey, instead of FeatureRowKey, but this is dependent on removing history from the serving stores first.

/wip
/hold

change the import spec options field to be sourceOptions, but I want to wait for another PR so we have less merge conflicts.

Is it #87 ?

group by entityname +entitykey, instead of FeatureRowKey, but this is dependent on removing history from the serving stores first.

is it possible without #87 ?

/hold cancel

/assign pradithya please review

I've included not writing to the feature stores as requested.

Related to #87

Biggest change externally is to the ImportSpec.

From

message ImportSpec {
  string type = 1;
  map<string, string> options = 2;
  repeated string entities = 3;
  Schema schema = 4;
}

To:

message ImportSpec {
  string type = 1;
  map<string, string> sourceOptions = 2;
  map<string, string> jobOptions = 5;
  repeated string entities = 3;
  Schema schema = 4;
}

To turn on coalesceRows you need to pass jobOptions: { coalesceRows.enabled: "true" }

You can also set, coalesceRows.delaySeconds, and coalesceRows.timeoutSeconds, but these settings are only relevant for streaming.

The delay indicates how many seconds the watermark must advance before the rows are flushed, default is 10 seconds.
They timeout indicates how many seconds the watermark must advance before the previous state is cleared. The default is 0, which indicates it will never be cleared.

should this be on or off by default?

should this be on or off by default?

I think it should be on by default.
What is the behavior when it's off?

Is this in-scope for 0.1.0?

/assign zhilingc

Can you run with some of your existing workloads?

@tims done, I think we're good to merge it in

/approve

sorry, sausage fingers :(

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: zhilingc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

/lgtm

@tims: you cannot LGTM your own PR.

boo... @pradithya can you lgtm this?

/lgtm