feat: Add MPNet model family - #434
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds comprehensive support for the MPNet model family to Bumblebee, enabling the use of models like sentence-transformers/all-mpnet-base-v2. The implementation follows the patterns established by similar encoder models (BERT, RoBERTa) in the codebase.
- Implements all standard MPNet architectures (base, masked LM, sequence/token classification, question answering, multiple choice)
- Adds proper tokenizer configuration with MPNet-specific special tokens
- Registers all model types in the main Bumblebee module
- Includes comprehensive test coverage for all architectures using tiny random models from HuggingFace
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
lib/bumblebee/text/mpnet.ex |
Core MPNet implementation with all architectures, configuration options, and HuggingFace parameter mappings |
lib/bumblebee/text/pre_trained_tokenizer.ex |
Adds MPNet tokenizer configuration with appropriate special tokens |
lib/bumblebee.ex |
Registers MPNet model classes and tokenizer type mapping |
test/bumblebee/text/mpnet_test.exs |
Comprehensive tests for all six MPNet architectures |
馃挕 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
6ea3784 to
15041b4
Compare
|
Squashed commits and updated the description. Tested with real models beyond the tiny-random ones. microsoft/mpnet-base outputs match Python transformers exactly, and moshew/mpnet-base-sst2-distilled works for sequence classification. |
15041b4 to
ccbd044
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
馃挕 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
jonatanklosko
left a comment
jonatanklosko
left a comment
There was a problem hiding this comment.
Looks good to me, just one nitpick and good to merge!
MPNet (Masked and Permuted Pre-training) is an encoder model that combines masked language modeling with permuted language modeling objectives. Supported architectures: - :base - :for_masked_language_modeling - :for_sequence_classification - :for_token_classification - :for_question_answering - :for_multiple_choice Key implementation details: - Uses relative position bias shared across all encoder layers - Does not use token_type_ids (unlike BERT/RoBERTa) - Position IDs start at pad_token_id + 1 (matching Python transformers)
ccbd044 to
c474aeb
Compare
Adds the MPNet model family to Bumblebee.
MPNet (Masked and Permuted Pre-training) is an encoder model that combines masked language modeling with permuted language modeling.
Supported architectures
:base:for_masked_language_modeling:for_sequence_classification:for_token_classification:for_question_answering:for_multiple_choiceImplementation notes
token_type_ids(unlike BERT/RoBERTa)pad_token_id + 1(matching Python transformers)Testing
microsoft/mpnet-basemoshew/mpnet-base-sst2-distilled)References