"Echo" mode for regular integration testing #4340
myypo
started this conversation in
Feature Requests
Replies: 1 comment
-
|
Thanks for the feedback. To do this fully correctly, we'd need to thread the request all the way down to each provider and ensure we check this on each of them (e.g. GCP has extra limitations on schemas). So this adds some complexity we'd want to avoid for now. We also use a fake model in CI. I agree this is not ideal and adds some overhead. I'm curious if others have any other ideas for testing here. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hey. What do you think about adding some kind of an "echo" parameter that does not pass the request to an LLM but instead just verifies that the request aligns with the current JSON schema and other configurations? In this case the gateway would respond with the contents of this echo field.
It would have been handy for regular integration tests in CI and could be used with or without
dryrun. Right now you have to get a fake model set up for this.Beta Was this translation helpful? Give feedback.
All reactions