Context
Came up while reviewing SparkApplication (#6550) against remote materialization (#6590).
Problem
POST /materialize-async returns 202 and runs store.materialize(_force_local=True) in a background executor. That frees the HTTP request coroutine, but the executor thread still runs the engine’s blocking poll.
For SparkApplicationComputeEngine, materialize() creates the ConfigMap + SparkApplication, then calls _wait_for_completion (poll loop up to job_timeout_seconds, default ~1 hour) until the job finishes.
So for a multi-client feature server we still hold a worker thread for the full Spark job. Review feedback on #6550 called this out as a scalability concern (“this should be async”).
What #6590 already solves (correct)
- Clients don’t need local Spark/K8s
- Clients don’t hold a long HTTP connection
- Clients can poll FeatureView state via the registry
What’s still missing for true non-blocking server behavior
On the async path, the server should submit only (create ConfigMap + SparkApplication / kick off the engine) and return 202 — without waiting in-process.
Completion should stay registry-driven (Spark driver pod already calls apply_materialization → AVAILABLE_ONLINE), which the client poller already uses.
Sync /materialize can keep blocking; that’s fine for CLI / single-tenant.
Suggested direction
- Split submit vs wait on engines that run external jobs (at least
spark_application).
- Have
/materialize-async (and /materialize-incremental-async) call submit-only.
- Design cleanup for ConfigMap/CR when the feature-server process no longer waits (
finally: _cleanup today runs after wait).
- Leave sync
POST /materialize behavior unchanged.
Related
Context
Came up while reviewing SparkApplication (#6550) against remote materialization (#6590).
Problem
POST /materialize-asyncreturns 202 and runsstore.materialize(_force_local=True)in a background executor. That frees the HTTP request coroutine, but the executor thread still runs the engine’s blocking poll.For
SparkApplicationComputeEngine,materialize()creates the ConfigMap + SparkApplication, then calls_wait_for_completion(poll loop up tojob_timeout_seconds, default ~1 hour) until the job finishes.So for a multi-client feature server we still hold a worker thread for the full Spark job. Review feedback on #6550 called this out as a scalability concern (“this should be async”).
What #6590 already solves (correct)
What’s still missing for true non-blocking server behavior
On the async path, the server should submit only (create ConfigMap + SparkApplication / kick off the engine) and return 202 — without waiting in-process.
Completion should stay registry-driven (Spark driver pod already calls
apply_materialization→AVAILABLE_ONLINE), which the client poller already uses.Sync
/materializecan keep blocking; that’s fine for CLI / single-tenant.Suggested direction
spark_application)./materialize-async(and/materialize-incremental-async) call submit-only.finally: _cleanuptoday runs after wait).POST /materializebehavior unchanged.Related
feature_server.py(run_in_executor→materialize(_force_local=True))spark_application/compute.py(_wait_for_completion)