Research Spotlight 3 Learnings From 3 Use Cases Of Mergekit

Arcee AI | Research Spotlight: 3 Learnings from 3 MergeKit Use Cases

Trinity Large Thinking: Available on OpenRouter.

Try now ↗

ENTERPRISE

Research

COMPANY

Get API

Blog / Research Spotlight: 3 Learnings from 3 MergeKit Use Cases

Research Spotlight: 3 Learnings from 3 MergeKit Use Cases Mariam Jabara ,

•

June 24, 2025

Merging for pre-training, data privacy in healthcare, and language support

MergeKit is the industry-leading tool for Model Merging, a technique that enables you to combine several pre-trained models into a smaller, more efficient model without requiring additional training (and without the need for a GPU). Merging preserves the original capabilities of models while enhancing AI performance and versatility. Arcee designed MergeKit to be both powerful and accessible, putting the power of model merging into the hands of builders of all levels. In this article, we’ll distill the lessons learned from three use cases of model merging published in research papers. Whether you’re interested in improving accuracy, creating domain-specific models, or incorporating advanced reasoning capabilities into diverse language models, we’ll share how model merging plays an important role in these use cases. Model Merging in Pre-training of Large Language Models While model merging is often discussed in the context of post-training , this paper's authors investigate the use of model merging techniques during the pre-training process, a largely unexplored area of research. Pre-training merging typically involves merging checkpoints from a single training run. However, researchers’ access to these intermediate checkpoints is limited, even with open models, which limits their understanding of the utility of merging in pre-training. Builders of DeepSeek and LLaMA-3 leveraged model merging in the development of their models, but they didn’t disclose the exact techniques. In this work, the authors introduce the concept of Pre-trained Model Average (PMA), a strategy for merging model-level weights during pre-training. Across a diverse set of LLMs of varying sizes and architectures, the authors found that merging checkpoints from the stable training phase results in consistent and significant improvements in performance. They also discovered that PMA models initialized with PMA for supervised fine-tuning demonstrated a more stable GradNorm metric compared to the baseline, in addition to reductions in the frequency of loss spikes relative to the baseline. The final performance was not impacted, and the method remained robust across various learning rates. Additionally, the authors performed ablation studies to understand the effect of intervals in the training process, specifically the time at which models are merged and the number of models merged. Intuitively, they found that large intervals resulted in lower performance, likely due to large weight disparities between early-stage and late-stage training, while merging a large number of models consistently led to performance improvements. Finally, they found that pre-training with a constant learning rate combined with model merging can match the performance of models trained with learning rate decay, making training more efficient and providing a way to estimate final performance earlier in the process. At Arcee, we leveraged this pre-training method in the development of our AFM-4.5B, the first Arcee Foundation Model . To build on this technique with more advanced merging, MergeKit also offers Arcee Fusion, a method for selective merging that focuses on the most significant differences between models, rather than indiscriminately merging all parameters. Using important scores for parameters, dynamic thresholding, and selective integration, only the most significant elements are incorporated into the base model, thereby improving results compared to merges based on simple averaging. Key Takeaway: Model merging is not only useful in the post-training phase, but can be a powerful tool to increase model performance in the pre-training phase, stabilize the training process, and monitor the pre-training process to simulate model performance after annealing.

PatientDx: Merging Large Language Models for Protecting Data Privacy in Healthcare Healthcare, as an industry, frequently faces challenges related to data access and privacy, making the development of domain-specific models through fine-tuning a challenging and often costly endeavor. In this work, the authors present PatientDx, a framework for model merging that enables the design of domain-specific large language models (LLMs) for predictive health tasks, without requiring fine-tuning or any adaptation of patient data. Using MIMIC-IV, an open electronic health record dataset, they evaluated the implications of model merging across predictive tasks and the pitfalls of fine-tuning, such as data leakage. Additionally, they explored the use of merged models for downstream tasks, such as identifying relevant keywords for patient profiles to improve tagging and the identification of similar patients.

Given that patient data consists of demographic and clinical features (age, lab values, diagnoses, time series data) in diverse formats (ranges, values, strings), the authors identified two key observations. First, an LLM tasked with predictive analysis on patient health record data needs to have a backbone model adapted to numerical reasoning, such as DART-math. Second, merged math and general-purpose models outperform either model type alone on predictive tasks, suggesting that an optimal merged model exists for these tasks. Thus, the authors optimized task performance by tuning the merge ratios between selected models, without the need for adaptation on patient data. Using the model soup and spherical linear interpolation (SLerp) merge methods, the authors examined whether merged models are more effective than input models for mortality prediction. Specifically, they merged Mistral and Llama models with various combinations of biology, instruct, and math fine-tuned LLMs. The result? Their 8B merged models showed a 7% improvement in AUROC compared to the initial models. They also found that the risk of data leakage was reduced in PatientDx and similar models that were not fine-tuned. Merging did not have a significant impact on information retrieval in downstream tasks, but it is an area that warrants further exploration. This work represents a validation of what the open-source...