Return "Never got TTFT because there was never content in the response." to user #4869
Replies: 2 comments 1 reply
-
|
Hi @quangIO, what's happening here is that there are no chunks with content sent back in the stream. So when we try to collect all the chunks in order to write to clickhouse we notice that there is no data and that causes TensorZero to yell. Since we may have already closed the connection to the client by the time we are doing this there is not in general a way to give the client error info for this case. Today, a workaround would be simply retrying if you get no content back from a stream. We have been considering allowing users to configure TensorZero to error and do retries / fallbacks if a response has no content but have not built this feature yet. Would that be helpful for you? |
Beta Was this translation helpful? Give feedback.
-
|
I think some built-in retry mechanism would be certainly useful for us. More context: we saw Never got TTFT... error log when using the Embedded mode. |
Beta Was this translation helpful? Give feedback.
-
|
In the next release (which will contain #4961), that error should occur less often |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
We are improving some reliability of our client. Sometimes, we received logs like "Never got TTFT because there was never content in the response." and the response stops unexpectedly. I assume this is a network-related issue? However, we only see the log, but no error is propagated to the client. We should have an option to retry failed requests or let the library users explicitly handle this error.
Beta Was this translation helpful? Give feedback.
All reactions