Resilience must be exercised
A modular ecology is not automatically robust. It can fail through dependency cascades, correlated models, stale contracts, or a router that repeatedly selects the same damaged component. Failure injection reveals whether redundancy and rollback are real.
Failure scenarios
- kill or unload the active specialist during a request burst;
- return malformed or oversized outputs;
- inject latency and timeout variance;
- corrupt a package digest or signature;
- simulate a poisoned federated update;
- make the router misclassify a task family;
- make the judge unavailable or inconsistent;
- exhaust memory or accelerator quota;
- remove network access or cloud fallback;
- shift task distribution toward a rare niche;
- revoke a dataset or license ancestor;
- fail the rollback target.
Experiment structure
PROCEDURE inject_failure(ecology, scenario)
baseline <- RUN_CONTROL_WORKLOAD(ecology)
APPLY_FAILURE(scenario)
stressed <- RUN_CONTROL_WORKLOAD(ecology)
recovery <- OBSERVE_UNTIL_STABLE_OR_TIMEOUT(ecology)
RESTORE_ENVIRONMENT(scenario)
RETURN {
detection_delay: stressed.first_alert - scenario.start,
quality_degradation: COMPARE(stressed, baseline),
containment_scope: MEASURE_BLAST_RADIUS(stressed),
recovery_time: recovery.duration,
policy_behavior: recovery.decisions,
audit_complete: VERIFY_TRACE_CHAIN(scenario)
}
END PROCEDUREExpected controls
The runtime should enforce timeouts and resource ceilings. The router should stop sending traffic to unhealthy packages. The controller should not create uncontrolled descendants during an incident. The release system should retain a verified rollback. Audit should identify every affected request.
Chaos boundaries
Run failures first in simulation, then staging, then low-risk production cohorts. Never inject failures that can cross tenant, safety, or data boundaries. Predefine abort criteria and responsible operators.
Recovery quality
Recovery is not only service availability. Check whether the fallback is safe, calibrated, and permitted for the data class. A system that remains online by routing everything to an expensive or less private model may violate its viability constraints.
Learning from incidents
Convert observed gaps into new evaluator cases, contract requirements, runtime controls, and beads. Do not let the same incident simply become training data without fixing the control-plane weakness that allowed it.
Source reports used for this guide
These reports are preserved verbatim in the site archive. The guide above is an editorial synthesis and may narrow, qualify, or reorganize claims from the source material.