-
-
Notifications
You must be signed in to change notification settings - Fork 12.1k
[Misc] Fix warnings for mistral model #23552
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code Review
This pull request effectively addresses two warnings related to the Mistral model. The first change correctly modifies VoxtralProcessorAdapter to return a BatchFeature object, aligning with the expected behavior of ProcessorMixin. The second set of changes addresses a deprecation warning by explicitly passing the special_token_policy to the underlying tokenizer methods, which is a good practice for future compatibility. The implementation looks solid and I don't see any critical or high-severity issues.
|
cc @patrickvonplaten can you check this? |
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
patrickvonplaten
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice refactor overall - think we need to address some minor parts but looking good otherwise
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Signed-off-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
|
Thanks for your suggestion |
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
|
I think it' ready to merge. @DarkLight1337 |
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> Signed-off-by: Jiangyun Zhu <riverclouds.zhu@qq.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> Signed-off-by: Jiangyun Zhu <riverclouds.zhu@qq.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Purpose
VoxtralProcessorAdapterreturnBatchFeaturepixtral: ditto
pass a special token policy explicitly for self._special_token_policy is deprecated.
Test Plan
Test Result
(Optional) Documentation Update
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.