Distillation and specialization

Distillation as breeding

Distillation creates a descendant that approximates a teacher's behavior within a target domain. It is one of the most practical model-breeding operators because it can convert an expensive coalition or generalist into a deployable specialist.

Distillation pipeline

1. Define the specialist contract and explicit non-goals.
2. Select teacher models and judge logic.
3. Build a seed dataset from licensed, representative examples.
4. Generate candidate labels or rationales under quality gates.
5. Filter disagreement, policy violations, duplicates, and low-confidence cases.
6. Train one or more students under different capacity and recipe settings.
7. Evaluate on independent human- or rule-labeled holdouts.
8. Compare against the teacher, champion, and a simple baseline.
9. Package lineage linking every synthetic example to teacher outputs and filters.

PROCEDURE distill_specialist(contract, teachers, seed_inputs)
    accepted_examples <- []

    FOR each input IN seed_inputs
        teacher_outputs <- RUN_INDEPENDENTLY(teachers, input)
        verdict <- JUDGE_TEACHER_OUTPUTS(teacher_outputs, contract)

        IF verdict.accepted AND verdict.confidence >= label_threshold
            APPEND accepted_examples, {
                input: input,
                label: verdict.output,
                teacher_ids: teachers.ids,
                verdict_id: verdict.id
            }
        END IF
    END FOR

    students <- TRAIN_BOUNDED_STUDENT_VARIANTS(accepted_examples, contract)
    RETURN INDEPENDENTLY_EVALUATE(students)
END PROCEDURE

Specialization boundaries

A specialist should abstain outside its niche. Make the niche machine-readable: taxonomy, language, data class, hardware, latency class, and confidence range. Routing should prefer abstention over confident misuse.

Teacher contamination

Synthetic data can magnify teacher biases, factual errors, style artifacts, and policy gaps. Preserve a human- or deterministic-label holdout. Include disagreement cases rather than filtering every difficult example, because uncertainty boundaries are part of the capability.

Capacity selection

Train multiple student sizes. The smallest student that satisfies the contract may offer the best viability even if a larger student wins a generic benchmark. Measure total accepted-result cost, including fallbacks and corrections.

Continual specialization

Do not continuously fine-tune a production artifact in place. Accumulate new evidence, create a descendant, evaluate it, and promote through the release pipeline. This preserves rollback and makes drift attributable.