<ul>

<li><a href="/publication/rlwa/">Rewarded soups: towards Pareto-optimality by interpolating weights fine-tuned on diverse rewards</a></li>

</ul>

<div class="media stream-item" itemscope itemtype="http://schema.org/Event">

<div class="media-body">

<h3 class="article-title mb-0 mt-0" itemprop="name">

Rewarded soups: towards Pareto-optimality by interpolating weights fine-tuned on diverse rewards

</h3>

<div class="stream-meta article-metadata">

<div itemprop="author">

Alexandre Ramé, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, Matthieu Cord</div>

soon on arXiv<br>

</div>

<div class="ml-3">

<img src="/publication/rlwa/featured_hu3f61ec8d614379be647f7d7ca55a4688_77418_400x0_resize_box_2.png" itemprop="image">

</div>

<div class="article-style" itemprop="articleBody">

We introduce rewarded soup, a new strategy to trade-off between multiple rewards when fine-tuning foundation models with RLHF; we first learn one network for each reward, and then linearly interpolate their weights.

</div>

<div class="talk-links">

</div><br />

</div>

ICML 2023<br>

Alexandre Ramé, Emilien Garreau, Hedi Ben-Younes, Charles Ollion</div>

arXiv<br>

[{"authors":["Alexandre Ramé","Guillaume Couairon","Corentin Dancette","Jean-Baptiste Gaya","Mustafa Shukor","Laure Soulier","Matthieu Cord"],"categories":null,"content":"","date":1684620000,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1684620000,"objectID":"7c9c768efb988234acbc742398cea03e","permalink":"/publication/rlwa/","publishdate":"2023-05-21T00:00:00+02:00","relpermalink":"/publication/rlwa/","section":"publication","summary":"","tags":null,"title":"Rewarded soups: towards Pareto-optimality by interpolating weights fine-tuned on diverse rewards","type":"publication"},{"authors":["Alexandre Ramé","Kartik Ahuja","Jianyu Zhang","Matthieu Cord","Léon Bottou","David Lopez-Paz"],"categories":null,"content":"","date":1671490800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1671490800,"objectID":"2e3ae69c28a112205a7743774c87e419","permalink":"/publication/ratatouille/","publishdate":"2022-12-20T00:00:00+01:00","relpermalink":"/publication/ratatouille/","section":"publication","summary":"","tags":null,"title":"Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization","type":"publication"},{"authors":["Alexandre Ramé","Matthieu Kirchmeyer","Thibaud Rahier","Alain Rakotomamonjy","Patrick Gallinari","Matthieu Cord"],"categories":null,"content":"","date":1652997600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1652997600,"objectID":"3468ac7cd0e597d0fd2adf7470645538","permalink":"/publication/diwa/","publishdate":"2022-05-20T00:00:00+02:00","relpermalink":"/publication/diwa/","section":"publication","summary":"","tags":null,"title":"Diverse Weight Averaging for Out-of-Distribution Generalization","type":"publication"},{"authors":["Arthur Douillard","Alexandre Ramé","Guillaume Couairon","Matthieu Cord"],"categories":null,"content":"","date":1630965600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1630965600,"objectID":"7541e62fdd60405a9055994724ce4f05","permalink":"/publication/dytox/","publishdate":"2021-09-07T00:00:00+02:00","relpermalink":"/publication/dytox/","section":"publication","summary":"","tags":null,"title":"DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion","type":"publication"},{"authors":["Alexandre Ramé","Corentin Dancette","Matthieu Cord"],"categories":null,"content":"","date":1630965600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1630965600,"objectID":"ac87dc1e61d871e29bbd68d2acf9280c","permalink":"/publication/fishr/","publishdate":"2021-09-07T00:00:00+02:00","relpermalink":"/publication/fishr/","section":"publication","summary":"","tags":null,"title":"Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization","type":"publication"},{"authors":[],"categories":null,"content":"Un condensé de la magie de Liu Cixin: la volonté de coller au plus près à la science malgré quelques pré-requis SF, une poésie toute chinoise sublimant ce monde qui apparaît devant les yeux du lecteur, une réflexion philosophique à tiroirs infinis poussant parfois à relire plusieurs fois les mêmes passages et à y penser sous 4 angles différents, des échos énormes aux plus grandes problématiques actuelles. Nous savons maintenant que notre Terre est finie, limitée. Mais dans cette nouvelle, c\u0026rsquo;est notre lumière dans le noir, notre point de repère universel qui est en danger: c\u0026rsquo;est bien notre Soleil qui est sur le point de mourir.\n","date":1607382000,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1607382000,"objectID":"e5cc49057d4b944e5fceab456ba8e768","permalink":"/books/wandering-earth/","publishdate":"2020-12-08T00:00:00+01:00","relpermalink":"/books/wandering-earth/","section":"books","summary":"Un condensé de la magie de Liu Cixin: la volonté de coller au plus près à la science malgré quelques pré-requis SF, une poésie toute chinoise sublimant ce monde qui apparaît devant les yeux du lecteur, une réflexion philosophique à tiroirs infinis poussant parfois à relire plusieurs fois les mêmes passages et à y penser sous 4 angles différents, des échos énormes aux plus grandes problématiques actuelles. Nous savons maintenant que notre Terre est finie, limitée.","tags":[],"title":"Terre Errante","type":"books"},{"authors":["Alexandre Ramé","Rémy Sun","Matthieu Cord"],"categories":null,"content":"","date":1583017200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1583017200,"objectID":"a95f4e852f0d2d3f9ad26f51e699360a","permalink":"/publication/mixmo/","publishdate":"2020-03-01T00:00:00+01:00","relpermalink":"/publication/mixmo/","section":"publication","summary":"","tags":null,"title":"MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks","type":"publication"},{"authors":["Alexandre Ramé","Matthieu Cord"],"categories":null,"content":"","date":1577833200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1577833200,"objectID":"2b5b8ce4b0f374ae9c148c76097c2837","permalink":"/publication/dice/","publishdate":"2020-01-01T00:00:00+01:00","relpermalink":"/publication/dice/","section":"publication","summary":"","tags":null,"title":"DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation","type":"publication"},{"authors":["Alexandre Ramé","Arthur Douillard","Charles Ollion"],"categories":null,"content":"https://arxiv.org/abs/2010.02849\n","date":1569880800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1569880800,"objectID":"417506d233c5c4f087a23ed0a76dab2e","permalink":"/publication/core/","publishdate":"2019-10-01T00:00:00+02:00","relpermalink":"/publication/core/","section":"publication","summary":"https://arxiv.org/abs/2010.02849","tags":null,"title":"CORE: Color Regression for Multiple Colors Fashion Garments","type":"publication"},{"authors":["Alexandre Ramé","Emilien Garreau","Hedi Ben-Younes","Charles Ollion"],"categories":null,"content":"","date":1543618800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1543618800,"objectID":"b83854a9c7581f99c5d80f267f9e7b4a","permalink":"/publication/omnia/","publishdate":"2018-12-01T00:00:00+01:00","relpermalink":"/publication/omnia/","section":"publication","summary":"","tags":null,"title":"OMNIA Faster R-CNN: Detection in the Wild through Dataset Merging and Soft Distillation","type":"publication"},{"authors":[],"categories":null,"content":"Bouquin unique, belle porte d\u0026rsquo;entrée à la hard sciences, qui plaira aux adeptes de SF comme aux fans de thrillers et aux ufologues. Premier tome d\u0026rsquo;une trilogie qui aura remporté le prix Hugo.\n","date":1526162400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1526162400,"objectID":"1c93f05b818253d22a35052bf95641b8","permalink":"/books/three-body/","publishdate":"2018-05-13T00:00:00+02:00","relpermalink":"/books/three-body/","section":"books","summary":"Bouquin unique, belle porte d\u0026rsquo;entrée à la hard sciences, qui plaira aux adeptes de SF comme aux fans de thrillers et aux ufologues. Premier tome d\u0026rsquo;une trilogie qui aura remporté le prix Hugo.","tags":[],"title":"The Three-Body Problem trilogy","type":"books"},{"authors":["Charles Corbiere","Hedi Ben-Younes","Alexandre Ramé","Charles Ollion"],"categories":null,"content":"https://openaccess.thecvf.com/content_ICCV_2017_workshops/papers/w32/Corbiere_Leveraging_Weakly_Annotated_ICCV_2017_paper.pdf\n","date":1470002400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1470002400,"objectID":"a3a7e1a45c6d8c61fd6c2aaba9d068ea","permalink":"/publication/weakly/","publishdate":"2016-08-01T00:00:00+02:00","relpermalink":"/publication/weakly/","section":"publication","summary":"https://openaccess.thecvf.com/content_ICCV_2017_workshops/papers/w32/Corbiere_Leveraging_Weakly_Annotated_ICCV_2017_paper.pdf","tags":null,"title":"Leveraging Weakly Annotated Data for Fashion Image Retrieval and Label Prediction","type":"publication"}]

<item>

<title>Rewarded soups: towards Pareto-optimality by interpolating weights fine-tuned on diverse rewards</title>

<link>/publication/rlwa/</link>

<pubDate>Sun, 21 May 2023 00:00:00 +0200</pubDate>

<guid>/publication/rlwa/</guid>

<description></description>

</item>