Operators are reusable capability moves
A merging operator is a way to turn parent artifacts into a candidate descendant. Operators should be described plainly, tested under equal budgets, and preserved in lineage.
| Operator | Positive purpose | Best use |
|---|---|---|
| Linear averaging | Blend compatible parent weights. | Shared-base fine-tunes with similar behavior. |
| SLERP | Smooth spherical blending through weight space. | Parent models where geodesic interpolation is more stable than a straight line. |
| Task arithmetic | Add useful capability deltas to a base. | Combining fine-tuned task vectors. |
| TIES | Preserve strong shared directions while reducing interference. | Multi-task merges with conflicting deltas. |
| DARE | Sparse capability transfer through pruning and rescaling. | Adapter or task-vector combinations where most deltas are not needed. |
| WIDEN | Separate magnitude and direction signals. | More controlled exploration of merge geometry. |
| Adapter merge | Combine compact LoRA or adapter deltas. | Frugal skill transfer without storing many full models. |
| Layer recipe | Select or blend layers by role. | Structural recombination when different layers hold different strengths. |
Compatibility first
Merging is strongest when base family, architecture, tokenizer, tensor schema, and license constraints are compatible. When compatibility is weak, use distillation or routing instead of direct weight mixing.
FUNCTION choose_merge_operator(parents, goal)
IF SAME_BASE_AND_TOKENIZER(parents) AND goal == "combine small task deltas"
RETURN ADAPTER_MERGE_OR_TASK_ARITHMETIC
END IF
IF COMPATIBLE_FULL_WEIGHTS(parents) AND goal == "smooth blend"
RETURN SLERP
END IF
IF DELTAS_CONFLICT(parents)
RETURN TIES_OR_DARE
END IF
RETURN ROUTE_OR_DISTILL_INSTEAD
END FUNCTIONThe operator catalog makes experiments easier to repeat and compare.
Source reports used for this guide
These reports are preserved verbatim in the site archive. The guide above is an editorial synthesis and may narrow, qualify, or reorganize claims from the source material.
Architecture and model mergingModel Breeder Architecture Deep DiveSource report · 76.7 KB
Model merging and adaptersModel Merging: Definitions, Taxonomy, Algorithms, Tooling, and Production Pipeline DesignEmerging practice · 35.7 KB
Model merging and adaptersTiny-LLM Adapters and LoRA: Breeding Operators, Compatibility, Evaluation, and PromotionEmerging implementation practice · 15.3 KB