A method of model composition that bypasses weight averaging in favor of direct structural splicing.
Frankenmerging is a technique within the fast-maturing field of model merging. Unlike weight averaging, where parameters from two models are interpolated, frankenmerging involves taking blocks of layers from two or more models and concatenating them into a single, taller network. This process is often referred to as a passthrough merge.
This approach was popularized by the mergekit library. While the term remains an obscure coinage, the underlying practice is spreading through the open weights community as a way to manipulate model architecture without the computational cost of full fine-tuning.
To understand frankenmerging, one must distinguish it from a model soup. In a model soup, the weights of multiple models are averaged to create a single set of parameters. In contrast, frankenmerging splices the architecture itself through layer stacking.
Because this method is training-free, it represents the cheapest form of model composition to experiment with. It allows researchers to test new architectural configurations without the massive hardware requirements typically associated with training new parameters from scratch.
The stitched model can pick up behaviors from each donor model. In some instances, this produces capabilities that none of the parent models possessed when they were standalone. This makes it a potent tool for rapid prototyping in the open weights ecosystem.
However, the results are often uneven. Because the layers are being joined without the benefit of a training phase to smooth the transitions between the disparate blocks, the stability and performance of the resulting model can be unpredictable. It remains an experimental method.
How does frankenmerging differ from model soups?
Model soups average the weights of multiple models, whereas frankenmerging stacks layers from different models to create a new, taller architecture.
Is retraining required for a frankenmerge?
No, frankenmerging is a training-free method of model composition, making it a highly efficient way to experiment with new model structures.
Collecting frankenmerge recipes, what breaks, and what unexpectedly works. Leave an email to follow along.