Repository navigation
[connectors] A failing preprocessor will drop messages #7606
Description
Describe the bug
Some users implement a preprocessor to handle input data. In some cases, the preprocessor relies on a sidecar container. If the sidecar is unhealthy or broken for some reason, the data is dropped by Feldera.
To Reproduce
- Define a pipeline with a preprocessor that fails/raises an erro
- Start the pipeline and push data
- Observe that messages/data are lost
Expected behavior
If a preprocessor is failing for some reason, we should pause the input connector, hold the data, and raise a clear warning or error for the user. Right now this is just in the log file.
Context (please complete the following information):
- Feldera Version: 0.367.0
- Environment: Cloud
Activity
[-]A failing preprocessor will drop messages[/-][+][connectors] A failing preprocessor will drop messages[/+]mihaibudiu commentedon Oct 23, 2026
"Failing" is not the same as "getting stuck".
mihaibudiu commentedon Oct 25, 2026
The sidecar is logically part of the preprocessor, and the preprocessor is a trusted component. If the preprocessor raises an error, that should surface as a parse error. But otherwise I would not expect data to be lost. Can you be more specific about the scenario which causes data loss? Clearly, if the sidecar ignores some data, that's the best the prepreocessor can do.
What does "preprocessor fails" mean?