Model merging

Model merging refers to the process of combining the weights of two or more neural networks with identical architectures to improve robustness.

Approaches

Borrows extensively from Goddard et al. (1)

Linear interpolation/model soup: Multiple models generated on the same data (with the same split) with different hyperparameters. Uniform soup refers to averaging without weighting, while greedy soup refers to averaging each model one at a time but only keeping the change if accuracy is improved. However, it has been shown to lead to catastrophic decreases in model accuracy if interpolation is carried out between models that don’t share an optimization trajectory (2).

Ref (2)

Task arithmetic: Adds and/or subtracts model weights derived from neural networks fine-tuned for discrete tasks.

Ref (3)

Trim, elect sign, merge: Also merges task-specific fine-tuned models (4). Proceeds in three steps: (1) Trim — only the influential parameter values are retained by setting redundant values in each task vector to zero; (2) Elect sign — remove sign conflicts; (3) Merge — merge parameters with consistent sign changes by taking the mean. This approach was found to outperform other approaches for both executing finetuning tasks and out-of-distribution generalization. Task arithmetic was a close second. On soup-like tasks that merge multiple models trained on the same objective, it matches or outperforms all other options, including ensembling. It also works well to generate starting points for initialization.

Fisher merging calculates importance of each parameter for a task by approximating the Fisher Information Matrix. Spherical linear interpolation (SLERP) interpolates between the weight values of two trained neural networks. Model ratatouille allows reuse of multiple fine-tuned models (derived from the same base model). DARE drops certain parameters at a pre-specified rate ($p$) and scales the remainder at an inverse rate ($1/(1-p)$). RegMean computes a model with closest similarity to activations of the two models of interest. LoraHub uses black-box optimization to merge two or more LoRA matrices (5); however this falls short of full LoRA fine-tuning using the data themselves.

Observations

Model merging can overcome reward hacking (overfitting) of RLHF methods for LLMs.

See also

• Weight averaging
• Averaging logits from multiple sources can improve fitness prediction

1.

Goddard C, Siriwardhana S, Ehghaghi M, Meyers L, Karpukhin V, Benedict B, et al. Arcee’s MergeKit: A Toolkit for Merging Large Language Models. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. Association for Computational Linguistics; 2024. p. 477–85. Available from: https://doi.org/10.18653/v1/2024.emnlp-industry.36

2.

Neyshabur B, Sedghi H, Zhang C. What is being transferred in transfer learning? Advances in Neural Information Processing Systems. 2020;33:512–23. Available from: https://proceedings.neurips.cc/paper/2020/hash/0607f4c705595b911a4f3e7a127b44e0-Abstract.html

3.

Ilharco G, Ribeiro MT, Wortsman M, Schmidt L, Hajishirzi H, Farhadi A. Editing models with task arithmetic. In: ICLR 2023. 2023. Available from: https://openreview.net/forum?id=6t0Kwf8-jrj

4.

Yadav P, Tam D, Choshen L, Raffel CA, Bansal M. TIES-Merging: Resolving Interference When Merging Models. In: Advances in Neural Information Processing Systems 36. Neural Information Processing Systems Foundation, Inc. (NeurIPS); 2023. Available from: https://proceedings.neurips.cc/paper_files/paper/2023/hash/1644c9af28ab7916874f6fd6228a9bcf-Abstract-Conference.html

5.

Huang C, Liu Q, Lin BY, Pang T, Du C, Lin M. LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition. 2023; Available from: https://arxiv.org/abs/2307.13269