Skip to content

Add support for function_to_apply in classification - #211

Merged
jonatanklosko merged 5 commits into
elixir-nx:mainfrom
coderrg:main
May 23, 2023
Merged

Add support for function_to_apply in classification#211
jonatanklosko merged 5 commits into
elixir-nx:mainfrom
coderrg:main

Conversation

@coderrg

@coderrg coderrg commented May 20, 2023

Copy link
Copy Markdown
Contributor

For classification models, users should have some way of specifying the function to be applied to the logits rather than softmax always being used since the logits do not always correspond to mutually exclusive classes.

Take, for example, a model's logits representing whether a piece of text conveys negative, neutral, or positive sentiment in a text classification task. In this case, the softmax activation function is apt as it ensures that the probabilities in the final output vector sum to 1. However, if the logits denote whether the text is toxic, obscene, insulting, etc. (as seen in Detoxify models), the sigmoid activation function would be more appropriate since the values in the final output vector do not necessarily have to sum to 1.

HF pipelines used softmax as the mandatory activation function over multi-class model logits until huggingface/transformers#8328, when they added support for the function_to_apply parameter which can be specified by the user.

I am proposing support for a function_to_apply parameter here as well. The options are the same as implemented in HF: softmax, sigmoid, and none (none applies no activation function and just returns the raw logits). softmax is the default since it preserves backward compatibility and seems to be the most common case.

Comment thread lib/bumblebee/text/text_classification.ex Outdated
Comment thread lib/bumblebee/text/text_classification.ex Outdated
Comment thread lib/bumblebee/text/text_classification.ex Outdated
@coderrg

coderrg commented May 23, 2023

Copy link
Copy Markdown
Contributor Author

Thank you for your feedback, @jonatanklosko! I have implemented your suggested changes.

Comment thread lib/bumblebee/text/text_classification.ex Outdated

@jonatanklosko jonatanklosko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks :)

@jonatanklosko
jonatanklosko merged commit b9acb4a into elixir-nx:main May 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants