Daheim, Nico ; Möllenhoff, Thomas ; Ponti, Edoardo ; Gurevych, Iryna ; Khan, Mohammad Emtiyaz (2024)
Model Merging by Uncertainty-Based Gradient Matching.
12th International Conference on Learning Representations. Vienna, Austria (07.05.2024 - 10.05.2024)
Konferenzveröffentlichung, Bibliographie
Kurzbeschreibung (Abstract)
Models trained on different datasets can be merged by a weighted-averaging of their parameters, but why does it work and when can it fail? Here, we connect the inaccuracy of weighted-averaging to mismatches in the gradients and propose a new uncertainty-based scheme to improve the performance by reducing the mismatch. The connection also reveals implicit assumptions in other schemes such as averaging, task arithmetic, and Fisher-weighted averaging. Our new method gives consistent improvements for large language models and vision transformers, both in terms of performance and robustness to hyperparameters.
| Typ des Eintrags: | Konferenzveröffentlichung |
|---|---|
| Erschienen: | 2024 |
| Autor(en): | Daheim, Nico ; Möllenhoff, Thomas ; Ponti, Edoardo ; Gurevych, Iryna ; Khan, Mohammad Emtiyaz |
| Art des Eintrags: | Bibliographie |
| Titel: | Model Merging by Uncertainty-Based Gradient Matching |
| Sprache: | Englisch |
| Publikationsjahr: | Mai 2024 |
| Veranstaltungstitel: | 12th International Conference on Learning Representations |
| Veranstaltungsort: | Vienna, Austria |
| Veranstaltungsdatum: | 07.05.2024 - 10.05.2024 |
| URL / URN: | https://openreview.net/forum?id=D7KJmfEDQP |
| Kurzbeschreibung (Abstract): | Models trained on different datasets can be merged by a weighted-averaging of their parameters, but why does it work and when can it fail? Here, we connect the inaccuracy of weighted-averaging to mismatches in the gradients and propose a new uncertainty-based scheme to improve the performance by reducing the mismatch. The connection also reveals implicit assumptions in other schemes such as averaging, task arithmetic, and Fisher-weighted averaging. Our new method gives consistent improvements for large language models and vision transformers, both in terms of performance and robustness to hyperparameters. |
| Freie Schlagworte: | UKP_p_seditrah_factcheck |
| Fachbereich(e)/-gebiet(e): | 20 Fachbereich Informatik 20 Fachbereich Informatik > Ubiquitäre Wissensverarbeitung |
| Hinterlegungsdatum: | 01 Jul 2024 11:05 |
| Letzte Änderung: | 01 Jul 2024 11:05 |
| PPN: | |
| Export: | |
| Suche nach Titel in: | TUfind oder in Google |
 |
Frage zum Eintrag |
Optionen (nur für Redakteure)
 |
Redaktionelle Details anzeigen |
Drucken
Impressum