Overview - How Proteus Works

The world model, the deterministic renderer and the emitters, and how a fault moves through all three.

Proteus is three stages with a hard line between each. A world model describes the estate as data. A renderer turns that model plus a time range into signal-neutral records, as a pure function. Emitters put those records on the wire, and they are the only part that touches the network.

flowchart TD
    WRITERS["plan · rename · chat<br>import-cfn · import-live · the web GUI"]
    MODEL["world model<br>JSON on disk"]
    RENDER["renderer"]
    EMIT["emitters"]
    AWS["CloudWatch OTLP endpoints"]
    RUM["the RUM data plane"]

    WRITERS -- "all they do is edit the model" --> MODEL
    MODEL -- "model plus a time window" --> RENDER
    RENDER -- "spans, metric points, logs, RUM events" --> EMIT
    EMIT -- "SigV4, gzip, OTLP protobuf over HTTP" --> AWS
    EMIT -- "SigV4, PutRumEvents JSON" --> RUM

The world model holds tenants, services, infra and faults, plus a seed. The renderer is pure, deterministic and streaming. The emitters are the only part that touches the network, and they are the only part that can spend money.

Note the last two edges. Three of the four signals go to the OTLP endpoints, and RUM goes somewhere else entirely — a CloudWatch RUM app monitor's data plane, as PutRumEvents JSON rather than protobuf. It is one pipeline with two exits, which is why RUM has its own limits and its own failure modes.

Two consequences fall out of that shape, and both matter when you use the tool. Because the renderer is pure, the same model and the same time range produce identical telemetry on every run — so a demo is reproducible from a JSON file and an integer. And because nothing above the emitters can reach AWS, --dry-run is not a simulation of a run; it is the real run with the last stage swapped out.

The world model#

The model is the single source of truth for the estate. Nothing else holds estate state: the renderer is pure and the emitters are stateless. That is what keeps the AI features deterministic. proteus chat and proteus hunt edit this document; they never generate telemetry themselves.

It holds four kinds of thing:

• Services — a node in the topology. Each has an archetype (what kind of workload it is), a runtime, baseline rate, latency and error rate, and a depends_on list. Those dependency edges become paired CLIENT and SERVER spans, which is what draws the service map.
• Infra nodes — a contended resource: an RDS instance, a Kafka cluster, an EKS nodegroup, an EBS volume. Services declare which ones they use.
• Tenants — fictional customers with a traffic share. An infra node records which tenants share it.
• Faults — a failure aimed at a service or an infra node, bounded by a start and an end.

Plus a region, an environment, and a seed. The model also has a fingerprint: a hash of the estate with the faults excluded, because faults churn constantly while the estate is what the CloudFormation stack provisions for. proteus doctor compares that fingerprint against the deployed stack to tell you the model has moved ahead of your infrastructure.

proteus plan writes a model from a preset. There are seven. The default is northwind-rx — 18 services, a fictional pharmacy-retail business with three tenants sharing eight infra nodes. Five of the others are small quick-start templates, each a shape you already recognize; minimal is three services for a smoke test. The seven presets has the table.

Every preset is deliberately connected. Every service hangs off the same dependency graph, because a set of disconnected workloads renders as islands in Omni and the follow-the-blast-radius story stops working. tests/test_templates.py enforces it.

What a service can be#

The catalog can place 68 workload types, from two layers.

12 hand-written archetypes have bespoke internal logic: an ALB, an nginx fleet, a generic application service, Lambda, RDS Postgres, ElastiCache, Kafka, OpenSearch, EKS and ECS workloads, a Bedrock agent, and a browser RUM app.

56 declarative profiles describe a workload as data instead — a list of metric specifications, each naming its shape, its unit, the AWS metric it corresponds to, and which fault field moves it. One generic generator renders any of them. Between them they cover 56 distinct AWS namespaces across 393 metric specifications, so adding an AWS service is a data edit rather than new code.

Two rules hold across both layers:

• No metric name may begin AWS/. CloudWatch reserves that namespace and rejects the request. The AWS counterpart travels as an attribute instead, which is how the UI can tell you "this is your CPUUtilization" without publishing into a reserved namespace.
• An unconfirmed AWS mapping is left null and shown as unknown. An invented metric name reads as authoritative and sends an operator hunting an alarm that cannot exist.

The renderer#

Rendering walks the time window one sampling instant at a time. At each instant, for each service, it resolves the fault effect in force, builds a Tick, and asks that service's generator for its metrics, then spans, then logs, then RUM events — always in that order.

A Tick carries a random number generator seeded by hashing (model seed, service name, timestamp). Ticks are therefore independent: order does not matter, nothing advances a shared stream, and no generator ever reads the clock. Time is always passed in.

Rendering is a generator, not a list. A 14-day backfill at 60-second intervals across 18 services is roughly 20 million records, so nothing materializes the full range.

The output is signal-neutral: four record types — a span, a metric point, a log event, a RUM event — none of which knows anything about OTLP or AWS. Each carries a resource, the identity of whatever emitted it.

Baselines are not flat. Each tick applies a traffic multiplier for time of day, plus a small jitter. The curve is a sine that bottoms out near 0.35 in the small hours and peaks near 1.35 in the early afternoon, UTC. A flat line reads as synthetic at a glance.

Why resource attributes, not custom metrics#

A record's OpenTelemetry resource attributes are how it states which service it belongs to, and its spans are how it states which service it called. Both are load-bearing, not decoration. So Proteus populates the conventional OTel keys — service.name, service.namespace, deployment.environment, cloud.provider, cloud.region, k8s.cluster.name, k8s.pod.name, host.name, db.system, messaging.system, and the gen_ai.* keys for agent workloads.

PutMetricData has somewhere to put the identity half — an EntityMetricData entry carries an Entity whose KeyAttributes accept Type, ResourceType, Identifier, Name and Environment — but nowhere at all to put the other end of a call. Across MetricDatum, Dimension and Entity there are thirteen fields and not one means peer, parent, caller, callee or dependency; an entity identifies one thing, never a pair. Spans carry the pair explicitly, through peer.service and parent_span_id. A metric can name a node; only a trace can name an edge, and that is the whole reason the tool speaks OTLP.

The emitters#

One emitter per signal, and they are the only code in the project that opens a socket. Each one batches records to the per-request limits, signs with SigV4, gzips, retries throttles with jittered backoff, and self-limits below the documented account request-per-second ceilings so a dense run slows down rather than collecting 429s and leaving gaps in your charts.

Four destinations:

Timestamps must fall inside a −14 day to +2 hour window (+10 minutes for metrics). That is where the 14-day ceiling on proteus seed comes from.

An emitter never raises on a partial failure. A bad batch is recorded and the run carries on, because one rejected request must not abort a 14-day backfill.

The in-memory emitter is what --dry-run, the cost estimator and every test use. Nothing above the emitter layer knows which kind it has.

What a fault is#

A fault is an entry in the model: a kind, a target, a start, an end, and a severity between 0 and 1. There are 14 kinds, from cpu-saturation and oom-kill to cache-stampede and agent-quality-regression. proteus faults lists them with the signature each one produces.

A fault never writes telemetry. It returns a FaultEffect: a bundle of multipliers and addends against baseline — latency_mult, error_rate_add, cpu_add, queue_depth_add, cache_hit_mult, and so on — plus a narrative payload of log lines and span events so the fault leaves human-readable evidence and not only bent curves.

Everything else follows from that one seam:

• Faults compose. Two effects combine by multiplying the multipliers and summing the addends, so concurrent faults on one service compound the way real ones do. An empty effect is "healthy" and can always be combined in safely.
• A new fault kind works everywhere. Because the effect is a bundle of numbers rather than a call into an archetype, adding a kind touches no generator.
• Onsets ramp. Severity is scaled in over the first 10% of the fault's window, because real incidents do not start square. oom-kill is the exception: it pulses on a deterministic cycle, because a pod being reaped and restarting is sharp and intermittent rather than a smooth slope.

How a fault reaches a service#

Three routes, and telling them apart is the exercise Omni is for.

A fault names exactly one target, and only two things can reach a service from it: the service's own name, or an infra node listed in that service's uses_infra.

flowchart TD
    F["fault.target"]
    DIRECT["direct"]
    INERT["inert"]
    NODE[("infra node")]
    SHARED["shared infra"]
    VICTIM["victim"]
    AGG["aggressor"]

    F -- "a service name" --> DIRECT
    F -- "a service name,<br>noisy-neighbor" --> INERT
    F -- "an infra node id" --> NODE
    NODE -- "in uses_infra" --> SHARED
    NODE -- "noisy-neighbor,<br>co-tenant" --> VICTIM
    NODE -- "noisy-neighbor,<br>aggressor's tenant" --> AGG

noisy-neighbor is deliberately asymmetric. By default the aggressor takes 42 percentage points of extra CPU and barely slows down at all, at 1.06x latency, while the victim's latency goes to 2.2x. A symmetric version is not findable, and finding it is the point. That asymmetry is structural rather than simulated: in northwind-rx, one database is genuinely shared by two tenants, and the batch job belonging to one of them genuinely contends with the other's checkout path.

A fault that cannot take effect as modelled resolves to a diagnostic rather than silently doing nothing. Point a noisy-neighbor at a service name instead of an infra node and it degrades nothing — contention is a property of a shared resource — but it says so against that service, so the modelling mistake surfaces instead of hiding.

The cost guard#

Every command that can send data projects first. The projection renders one representative instant near the middle of the window and multiplies, which keeps it O(services) rather than O(records); a 14-day projection takes about as long as a ten-minute one. Metric series are counted differently, by sampling several scattered instants and taking the union, because a series that appears on only some ticks is invisible to a single sample and under-reporting is the unsafe direction for a spend gate.

The projection then meets a policy, which returns a decision: allowed or not, confirmation required or not, forced into dry-run or not. Five policies are available, estimate-confirm by default. They are listed in the cost guard.

Rates come from the live AWS Pricing API, cached locally, falling back to a built-in table; the projection tells you which it used. Figures are list price and never a quote. A spend ledger records what each run actually sent.

What each command does#

Reading the table against the pipeline above tells you roughly what any command will do before you run it.

seed, live and firehose all take --guard and --dry-run. serve binds 127.0.0.1 only and rejects requests carrying a foreign Origin, because that process holds your AWS credentials and can spend money.

Where state lives#

One directory, ~/.proteus/: the world model, the GUI's canvas layout, the pricing cache, the spend ledger, and sealed hunt answers. Set PROTEUS_HOME to keep several estates side by side. Nothing in there belongs in version control.

Next#

Diving Deep covers each subsystem in the detail you need to extend it. Operations has worked examples for the CLI and the GUI.