Faults

FaultEffect as the seam, the 14 fault kinds, severity ramping, shared-infra contention and noisy-neighbor asymmetry.

A fault never writes telemetry. It returns a FaultEffect: a bundle of multipliers and addends that a generator applies to its own baseline.

That indirection is the whole design. It is why a new fault works across every one of the 68 placeable workload types without touching any of them, and why noisy-neighbor can hit a whole infra node's tenants at once with two different signatures.

FaultEffect#

proteus/faults/effects.py is frozen. The dataclass carries 14 numeric fields and a narrative payload:

@dataclass(frozen=True, slots=True)
class FaultEffect:
    latency_mult: float = 1.0
    error_rate_add: float = 0.0
    rps_mult: float = 1.0
    cpu_add: float = 0.0
    mem_add: float = 0.0
    disk_used_add: float = 0.0
    queue_depth_add: float = 0.0
    consumer_lag_add: float = 0.0
    conn_used_add: float = 0.0
    packet_loss_add: float = 0.0
    cache_hit_mult: float = 1.0
    eval_score_mult: float = 1.0
    token_mult: float = 1.0
    restart_rate_add: float = 0.0

    log_lines: tuple[tuple[Severity, str], ...] = ()
    span_events: tuple[str, ...] = ()
    span_error_message: str | None = None
    tags: tuple[str, ...] = ()

Multipliers default to 1.0 and addends to 0.0, so FaultEffect() is healthy and can always be combined in safely. The module exports it as HEALTHY.

Units#

Stated once, here, because this is what gets re-broken:

latency_mult applies to both p50 and p99.

The narrative payload#

log_lines, span_events and span_error_message are why a fault leaves human-readable evidence rather than only bent curves. Generators splice them into their output: catalog/_records.py::fault_logs emits one LogRecord per entry in log_lines at the severity given.

Those lines are SCOPE_ASYNC and deliberately carry no trace ids. A fault's narrative describes a condition affecting the whole service, not one request, so pinning it to an arbitrary sampled request would assert a relationship that does not exist. It also could not be correlated at all on a tick that sampled nothing, which would leave the scope contract unsatisfiable rather than merely imprecise.

tags carries the fault kinds that contributed, and catalog/_records.py::log_attrs puts them on every log line as proteus.fault, so you can find the evidence for a specific fault in the console.

combine and scaled#

def combine(self, other: FaultEffect) -> FaultEffect:
    """Compound two effects. Multipliers multiply, addends sum."""

def scaled(self, factor: float) -> FaultEffect:
    """Scale intensity by `factor` (0.0 = healthy, 1.0 = full)."""

combine is how two concurrent faults on one service compound the way real ones do: two latency_mult of 2.0 give 4.0, two cpu_add of 30 give 60. The narrative payloads concatenate, and span_error_message takes the first non-None. combine_all(effects) folds any number and returns HEALTHY for an empty input.

scaled interpolates multipliers toward 1.0 and scales addends linearly. One detail matters: the narrative payload is dropped below a 5% threshold, so a barely-ramped fault does not spam the logs with FATAL lines while its numbers are still within noise.

The fault library#

proteus/faults/library.py holds one builder per FaultKind. Every builder returns a full-intensity effect — the signature at severity 1.0. Callers apply the ramp with scaled(); builders never scale themselves.

Builder = Callable[[Fault, Service, WorldModel], FaultEffect]

Each signature is deliberately multi-signal. A fault that bends one curve is not findable by an operator and not a fair puzzle in hunt mode, so each one moves a recognisable set of signals and leaves matching log evidence.

Params are registered with the builder#

@_fault(
    FaultKind.CPU_SATURATION,
    "CPU pinned near 100%: latency degrades superlinearly, with throttling in the logs.",
    cpu_add="CPU utilisation added, percentage points (default 55)",
    latency_mult="multiplier on latency (default 2.6)",
    error_rate_add="error rate added, absolute fraction (default 0.015)",
)
def cpu_saturation(fault: Fault, service: Service, model: WorldModel) -> FaultEffect:
    """CPU pinned near the ceiling; latency degrades superlinearly."""
    return FaultEffect(
        cpu_add=_num(fault, "cpu_add", 55.0),
        latency_mult=_num(fault, "latency_mult", 2.6),
        error_rate_add=_num(fault, "error_rate_add", 0.015),
        mem_add=6.0,
        log_lines=(
            (Severity.WARN, "cpu throttled: cfs quota exhausted for period"),
            (Severity.WARN, "event loop lag above 250ms, scheduler starved"),
            (Severity.ERROR, "request rejected after queueing beyond deadline"),
        ),
        span_events=("cpu.throttled",),
    ).with_tag(fault.kind)

The decorator registers the builder together with its one-line description and its tunable params, and appends the params to the builder's docstring. So describe(kind) and params_for(kind) come from one source and cannot drift from the code — the AI layer, the CLI and the GUI palette all read them from there.

_num(fault, key, default) reads a numeric tunable from fault.params, falling back to the default if the key is absent or unparseable.

The registry#

effect_for(fault, service, model)            # full intensity; HEALTHY if kind unknown
aggressor_effect_for(fault, service, model)  # the hog's side; HEALTHY for every other kind
known_kinds()                                # in FaultKind declaration order
describe(kind)
params_for(kind)

proteus faults prints describe() for all 14, verified:

queue-depth                 Consumers fall behind: lag and queue depth climb,
                            latency rises moderately, errors flat.
page-load-latency           Front-end page loads slow sharply with a flat error rate.
cpu-saturation              CPU pinned near 100%: latency degrades superlinearly,
                            with throttling in the logs.
oom-kill                    Memory exceeds the limit, pods are OOM-killed and
                            restart; traffic dips in bursts.
disk-full                   The volume fills and writes fail with ENOSPC.
noisy-neighbor              A co-tenant saturates shared infrastructure; victims
                            slow with no local cause.
connection-pool-exhaustion  All pool connections are checked out: callers queue,
                            then time out waiting.
cache-stampede              Cache hit ratio collapses and the miss storm
                            multiplies downstream load.
agent-quality-regression    Agent answer quality drops while latency, errors,
                            CPU and tokens stay healthy.
dependency-timeout          A downstream call hangs to its timeout ceiling,
                            then errors.
network-packet-loss         Packets drop; retries stretch latency and leak a
                            moderate error rate.
slow-query                  A query plan regresses: latency only, with flat CPU
                            and flat errors.
shard-hotspot               Load skews onto one shard: it runs hot and rejects
                            work from a full queue.
thread-pool-rejection       The work queue is full: errors dominate while latency
                            barely moves.

The per-kind parameter tables are in the Appendix.

Three kinds worth reading the source of#

agent-quality-regression is surgically clean on purpose. Latency, error rate, CPU and token usage all stay at baseline, so eval_score_mult (default 0.62) is the only numeric that moves. This is the failure mode traditional monitoring cannot see, and making it visible is much of the reason the Bedrock agent archetype exists.

dependency-timeout caps its latency multiplier rather than letting it grow without bound:

timeout_ms = _num(fault, "timeout_ms", 3000.0)
baseline = max(service.base_latency_p99_ms, 1.0)
cap = _num(fault, "max_latency_mult", 12.0)
mult = min(max(timeout_ms / baseline, 1.0), cap)

The ceiling is the tell an operator recognises: a latency chart that flattens against a round number is a timeout, not congestion.

connection-pool-exhaustion reads the real ceiling out of the model. _pool_ceiling walks service.uses_infra for the first node declaring max_connections in its capacity dict and uses that, falling back to 50. So exhausting the pool lands on the number the estate says the pool is, and the log line quotes it: pool saturated: 200 of 200 in use.

Ramping and the OOM pulse#

Fault.ramp_at(ts) scales severity linearly over the first 10% of the fault window, floored at one second:

span = (self.end - self.start).total_seconds()
elapsed = (ts - self.start).total_seconds()
ramp = min(1.0, elapsed / max(span * 0.1, 1.0))
return self.severity * ramp

Real incidents do not start square. A one-hour fault at severity 0.6 reaches full intensity six minutes in.

faults/resolve.py::intensity_at is what the resolver actually calls, and it adds one exception. OOM_KILL is pulsed, because a pod being reaped and restarting is sharp and intermittent rather than a smooth slope:

period = max(float(fault.params.get("pulse_period_s", 300.0)), 1.0)
elapsed = (ts - fault.start).total_seconds()
phase = (elapsed / period + _unit_hash(fault.id)) % 1.0
pulse = 0.15 + 0.85 * (0.5 - 0.5 * math.cos(2.0 * math.pi * phase)) ** 3
return ramp * pulse

A raised cosine cubed over pulse_period_s seconds (default 300), floored at 0.15 so the memory pressure never fully clears between kills. The phase offset is a blake2b hash of fault.id, not a random draw, so two concurrent OOM faults pulse out of step and both replay identically. _unit_hash exists precisely to replace any use of random here.

How a fault reaches a service#

proteus/faults/resolve.py is the only module that knows how a fault propagates. It is pure: no I/O, no random, and ts is always passed in.

Three things can degrade a service, and an operator has to be able to tell them apart. Origin enumerates them plus two more:

contributions_at(service, model, ts) returns one Contribution per active fault, already scaled:

@dataclass(frozen=True, slots=True)
class Contribution:
    fault: Fault
    origin: Origin
    intensity: float      # the applied scale, 0..1
    effect: FaultEffect   # with that scale already applied
    via: str              # infra node id for infra origins, else the service name
    note: str = ""        # why an INERT fault cannot take effect

effect_at(service, model, ts) is combine_all over those contributions and is the function the renderer calls. It returns HEALTHY when nothing is wrong.

contributions_at applies no filtering on Fault.hidden — it reports facts, and deciding what to reveal is the caller's job.

What depends_on does and does not do#

A fault does not propagate along depends_on. This surprises people, so it is worth stating plainly and showing.

Given api with depends_on=["orders"], and a full-severity cpu-saturation on orders:

orders:  cpu-saturation [f1] directly on orders at intensity 1.00 -
         latency x2.60; error rate +0.015 abs fraction; cpu +55 pp;
         memory +6 pp; 3 log line(s); span events cpu.throttled

api:     (no contributions; effect is HEALTHY)

api stays healthy. depends_on builds topology, not causation: it is what makes the generators emit a CLIENT span per outbound call sharing the SERVER span's trace_id (catalog/_records.py::client_spans), and one outbound-call latency and volume series per dependency (catalog/_appmetrics.py). Those CLIENT/SERVER pairs are what draw the edge in Omni.

If you want a caller to suffer because its dependency is sick, model that explicitly with dependency-timeout on the caller, naming the dependency:

{
  "id": "f-timeout",
  "kind": "dependency-timeout",
  "target": "api",
  "params": {"target_dependency": "orders", "timeout_ms": 3000}
}

target_dependency defaults to service.depends_on[0], or the literal "upstream" when the service declares no dependencies, and it appears in the log line: call to orders timed out after 3000ms.

The design reason for not propagating automatically: cascade depth and blast radius are modelling choices with no correct default. Inferring them would put a number in the data that nobody chose, and a hunt whose answer depends on an inferred cascade is not fair. Shared infrastructure is modelled as propagation, because uses_infra is an explicit statement that two services contend over one resource.

Shared infra#

A fault on an InfraNode degrades every service whose uses_infra names it. Each victim gets a SHARED_INFRA contribution carrying the same builder output as a direct hit, with via set to the node id. One sick RDS instance degrades every service that touches it, and explain says which node:

slow-query [f3] via shared infra rx-primary at intensity 0.84 - latency x4.00; 3 log line(s)

Noisy-neighbor asymmetry#

noisy-neighbor is the one kind with two signatures, and the asymmetry is the whole point: the aggressor looks busy; the victims look broken, with no local cause.

api     (aggressor)  latency x1.06; request rate x1.35; cpu +42 pp
                     log: "batch workload running at full parallelism"

orders  (victim)     latency x2.20; error rate +0.006; cpu +26 pp
                     log: "iowait elevated, storage queue depth above normal"
                          "cpu steal time sustained above 15%"
                          "no local deploy or config change in window"

That last victim line is doing real work. An operator's first move on a slow service is to look for a deploy, and the fault says there wasn't one — which is the clue that points at the shared resource.

library.noisy_neighbor is the victim builder; library.noisy_neighbor_aggressor is the hog's, reached through aggressor_effect_for. Every other kind returns HEALTHY from aggressor_effect_for, because nobody benefits from a disk filling up.

Picking an aggressor#

resolve._resolve_aggressor decides who the hog is, deterministically: a blake2b hash of fault.id indexing the sorted candidate list. Never random.

The candidate list is the subtle part. WorldModel.tenants_on(node_id) answers "who is permitted to share this node" — for a node with shared_by=[] that is every tenant in the estate. That is the wrong set to pick from:

a tenant with no service on the node emits no telemetry there, so naming it produces uniform degradation with no visible culprit and an unsolvable hunt.

So aggressor_candidates narrows to the intersection of the permitted tenants with the tenants that actually run a service using the node. Services with tenant=None are excluded: they can be starved, but they are nobody's noisy neighbor.

An explicit params.aggressor_tenant is honoured only if it is in that list.

Five outcomes, and only one of them names a hog:

flowchart TD
    START["noisy-neighbor<br>on a node id"]
    CAND["candidates:<br>permitted tenants<br>with a service on the node"]
    NAMED{"params.aggressor_tenant<br>given"}
    INSET{"in candidates"}
    USE["that tenant is the aggressor"]
    R1["INERT: it emits no<br>telemetry on this node"]
    COUNT{"how many candidates"}
    R2["INERT: nobody<br>contends for the node"]
    R3["INERT: contention needs<br>an aggressor and a victim"]
    PICK["blake2b of fault.id<br>modulo the list length"]

    START --> CAND
    CAND --> NAMED
    NAMED -- yes --> INSET
    INSET -- yes --> USE
    INSET -- no --> R1
    NAMED -- no --> COUNT
    COUNT -- none --> R2
    COUNT -- one --> R3
    COUNT -- "two or more" --> PICK
    PICK --> USE

The single-candidate case is the one to notice. One tenant on the node is not a degenerate version of contention, it is the absence of it — there is nobody to be the victim — so the resolver refuses rather than degrading the only tenant against itself. The remedy it suggests is structural: set shared_by, or move a second tenant's service onto the node.

The pick being a hash of fault.id rather than random is what makes a hunt replayable: the same model and the same fault id name the same aggressor on every run, so a scenario you demo today behaves identically next month.

INERT, and why nothing happens loudly#

Three situations make a noisy-neighbor fault unmodellable. In each one the resolver produces an INERT contribution with HEALTHY as the effect and a note explaining why, rather than degrading everyone uniformly:

There is a fourth: aiming noisy-neighbor at a service name rather than an InfraNode. Contention is a property of a shared resource, not of a service, so the fault degrades nothing — and it says so, on the service named, so the modelling error surfaces instead of hiding:

noisy-neighbor [f9] on 'checkout' is INERT - it is degrading nothing, and this is
a modelling error, not an incident: 'checkout' is a service, not an infra node; a
noisy neighbor needs a shared InfraNode to contend over

Degrading everyone uniformly would read as "the estate is broken" rather than as an incident, and would leave a hunt unsolvable. GET /api/catalog therefore reports noisy-neighbor with targets: "infra", so the GUI only highlights infra drop targets when it is dragged — see the web GUI.

explain()#

explain(service, model, ts) -> list[str]

One line per fault, naming the kind, how it arrived, the applied intensity, and the signals actually carrying the evidence. This is what proteus hunt reveal shows. It returns [] for a healthy service and applies no hidden filtering.

_signals(effect) builds the evidence list with units attached — latency x2.20, cpu +26 pp, error rate +0.006 abs fraction, 3 log line(s) — and says no measurable deviation yet when an effect moves nothing. Multipliers are compared against 1.0 with a 1e-9 tolerance rather than !=, so float noise does not produce a phantom signal.

Adding a fault kind#

1. Add a member to FaultKind in proteus/model/schema.py. This is a frozen file; the enum is the one thing that has to grow.
2. Write a builder in proteus/faults/library.py decorated with @_fault, giving the one-line description and every tunable param with its unit and default.
3. Return a full-intensity effect. Do not scale — the resolver does that.
4. Move a set of signals and give it log lines. A one-curve fault is not findable.
5. If it needs an aggressor/victim split, extend aggressor_effect_for. If it must target infra, add it to _INFRA_ONLY_FAULTS in proteus/web/catalog_view.py so the GUI does not offer it as a service target.

You do not touch a single archetype or profile. That is what the seam is for.