Math behind Finetuning and Merging Language Models
Loading...
Date
2025-05
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
The Ohio State University
Abstract
To bridge this gap, fine-tuning LMs has become a critical process. By training models on domain-specific datasets or aligning them with specific objectives, organizations can improve their performance, improve accuracy, and ensure ethical considerations in their responses. Fine-tuning allows LMs to adapt to specialized use cases, whether in finance, law, healthcare, or manufacturing, making them more reliable and effective. This report explores the underlying mathematical principles of LMs, model fine-tuning, and certain model merging techniques. Using Microsoft's Phi 2 Model, we finetuned the base model on the Moral Scenarios and Professional Accounting Category of the MMLU Datasets, which a multiple choice task. We reconfigured the model output to answer multiple choice questions. Then, we merged the two fine-tuned models by averaging the weights and a sign-aware merge method. We want to show that finetuning and merging can create a model that specializes in a particular domain. In our case, we want a model that specializes in Moral Scenarios and Professional Accounting.