[AI] Add HybridModel to support a fallback model - #16110
Conversation
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. |
Generated by 🚫 Danger |
|
It looks like you're trying to trigger a pull request summary and a code review. The |
Summary of ChangesThis pull request introduces a flexible fallback mechanism for generative models by adding a Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Activity
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a hybrid model feature that allows for a primary model with a secondary fallback. It includes the HybridModel, HybridModelProvider, and HybridModelSession implementations, along with updates to GenerateContentResponse to track the model version used. Feedback focuses on refining the fallback logic: ensuring streaming responses do not produce corrupted output if a failure occurs after data has been yielded, maintaining session consistency in multi-turn conversations by sticking to the secondary model after an initial fallback, and optimizing resource usage by lazily initializing the secondary session.
There was a problem hiding this comment.
Code Review
This pull request introduces a hybrid model feature that enables a primary language model to fall back to a secondary model upon failure. Key changes include the implementation of HybridModel, HybridModelProvider, and HybridModelSession, along with updates to GenerateContentResponse to track the model version. The review feedback identifies a critical bug in the streaming fallback logic that could cause hangs, suggests optimizing resource usage through lazy initialization of the secondary session, and recommends improvements to error message formatting and documentation typos.
There was a problem hiding this comment.
Code Review
This pull request introduces a hybrid model feature to the FirebaseAI library, enabling a primary model to fallback to a secondary model if the primary session fails. It includes the implementation of HybridModel, HybridModelProvider, and HybridModelSession, along with updates to GenerateContentResponse to track the modelVersion. Review feedback recommends making the modelVersion property public for improved debugging and telemetry, refactoring the hybrid session to lazily initialize the secondary model for better resource efficiency, and correcting a version typo in a TODO comment.
2 participants
Added a new
HybridModeltype that is a combination of a primaryLanguageModeland a secondary model to fallback to if the primary model is unavailable or throws an error. This is preparing for hybrid inference using the on-deviceSystemLanguageModelfrom the Foundation Models framework, as wellGeminiModelin the cloud.Since
HybridModelsupports any model conforming toLanguageModel, this also means that fallback support is available for multiple Gemini models or backends (Gemini Developer API and Vertex AI API). This feature is used in the integration tests, which specify invalid model names to simulate a model being unavailable and falling back to another valid model name.#no-changelog