The renderer

The pure function from (WorldModel, Window) to a stream of records — determinism, Tick, tick_rng, windows and the four record types.

The renderer is (WorldModel, Window) -> Iterator[Record]. A pure function: no network, no file reads, no print, no wall-clock reads, and no use of the random module.

That is what makes faults testable by golden file, and it is why proteus/render/ imports no AWS SDK and needs no credentials.

The record types#

proteus/render/records.py is frozen. Four record types, all frozen dataclasses with slots=True. All timestamps are timezone-aware UTC; all durations are milliseconds unless the field name says otherwise.

Resource#

@dataclass(frozen=True, slots=True)
class Resource:
    attributes: Attributes

    def key(self) -> str: ...            # stable grouping key for batching
    @property
    def service_name(self) -> str: ...   # raises KeyError if absent

An OTel resource: the identity of whatever emitted the record. Load-bearing, not decoration — the resource is how a record says which service it belongs to, so service.name is mandatory and the conventional keys should be populated wherever they apply. The dependency between two services is carried by spans, not by a resource: see peer.service and parent_span_id. Each unique resource must stay under 16 KB serialised.

catalog/_shared.py::base_attributes is the one place they are built. It always sets service.name, service.namespace (the model name), service.instance.id, deployment.environment, cloud.provider, cloud.region, proteus.archetype and proteus.runtime, plus proteus.tenant when the service has one.

_runtime_attributes then layers on what the runtime implies — cloud.platform plus host.id/host.name for EC2, faas.name/faas.version for Lambda, k8s.cluster.name/k8s.namespace.name/k8s.deployment.name for EKS, and the ECS equivalents. An archetype's extra goes on top, and service.attrs from the world model wins over everything.

Resource holds a Mapping and is therefore unhashable, which is why the engine memoises key() by object identity rather than using the resource as a dict key.

SpanRecord#

@dataclass(frozen=True, slots=True)
class SpanRecord:
    trace_id: str          # 32 lowercase hex chars
    span_id: str           # 16 lowercase hex chars
    name: str
    kind: SpanKind
    start: datetime
    end: datetime
    resource: Resource
    parent_span_id: str | None = None
    status: SpanStatus = SpanStatus.UNSET
    status_message: str | None = None
    attributes: Attributes = field(default_factory=dict)
    events: tuple[SpanEvent, ...] = ()

    @property
    def duration_ms(self) -> float: ...

Max 200 KB on the wire, max 10,000 per request. SpanKind is the OTel set — INTERNAL, SERVER, CLIENT, PRODUCER, CONSUMER — and SERVER/CLIENT pairs sharing a trace_id are what build Omni's topology edges. SpanStatus is UNSET, OK, ERROR. A SpanEvent is a name, a timestamp and attributes: an exception, a retry, an OOM kill.

MetricPoint#

@dataclass(frozen=True, slots=True)
class MetricPoint:
    name: str
    kind: MetricKind
    timestamp: datetime
    resource: Resource
    value: float = 0.0
    unit: str = "1"
    attributes: Attributes = field(default_factory=dict)
    is_monotonic: bool = False        # SUM only
    bucket_counts: tuple[int, ...] = ()      # HISTOGRAM only
    explicit_bounds: tuple[float, ...] = ()  # HISTOGRAM only
    count: int = 0                    # HISTOGRAM only
    sum: float = 0.0                  # HISTOGRAM only

Max 1,000 per request — the tightest limit in the system. MetricKind is GAUGE, SUM or HISTOGRAM. For a histogram, populate bucket_counts, explicit_bounds, count and sum; value is ignored. Never emit a metric whose namespace or name begins AWS/ — reserved, and the request is rejected.

Latency histograms bucket over catalog/_shared.py::LATENCY_BOUNDS_MS, 13 bounds in milliseconds:

LATENCY_BOUNDS_MS = (
    1.0, 2.0, 5.0, 10.0, 25.0, 50.0, 100.0, 250.0, 500.0,
    1000.0, 2500.0, 5000.0, 10_000.0,
)

This is a cross-module contract, not a private constant. Both halves of the catalog import it so their histograms stay comparable in Omni, and _compute_shared.py::SLOW_BOUNDS_MS extends it for workloads whose p99 runs past its top bound. It must stay strictly ascending and must stay a strict prefix of any extension. MAX_HISTOGRAM_SAMPLES = 200 caps the synthetic samples behind one histogram datapoint.

LogRecord#

@dataclass(frozen=True, slots=True)
class LogRecord:
    timestamp: datetime
    severity: Severity
    body: str
    resource: Resource
    log_group: str
    log_stream: str
    attributes: Attributes = field(default_factory=dict)
    trace_id: str | None = None
    span_id: str | None = None

    @property
    def severity_number(self) -> int: ...

Max 1 MB each, max 10,000 per request. log_group and log_stream become the x-aws-log-group and x-aws-log-stream headers, so records are batched by that pair and two pairs never share a request.

Severity maps to OTel numbers: TRACE 1, DEBUG 5, INFO 9, WARN 13, ERROR 17, FATAL 21.

Setting trace_id/span_id is what lets Omni correlate a log line to a trace, so populate them whenever the log belongs to a request. The catalog enforces the pairing through an explicit scope attribute rather than by convention:

SCOPE_REQUEST = "request"   # carries trace_id and span_id
SCOPE_ASYNC   = "async"     # carries neither

make_log writes proteus.log.scope onto every line, which turns "every request-scoped line carries a trace id" into a claim you can check in the console.

Group and stream naming lives in catalog/_records.py:

def log_group(model, service) -> str:
    return f"/proteus/{model.name}/{service.name}"

def log_stream(tick) -> str:
    """A log stream name stable per instance, e.g. `checkout-api-002`."""

Which instance emitted this interval's batch is hashed from the tick, so the name is the same whichever signal method asks for it.

RumEvent#

@dataclass(frozen=True, slots=True)
class RumEvent:
    timestamp: datetime
    event_type: str        # RUM's dotted names
    app_monitor: str
    session_id: str
    page_id: str
    attributes: Attributes = field(default_factory=dict)
    user_id: str | None = None

Browser page-load data does not share the OTLP path; it goes to the RUM data plane instead. event_type uses RUM's dotted names, e.g. com.amazon.rum.performance_navigation_event. See emitters.

Routing#

Record: TypeAlias = SpanRecord | MetricPoint | LogRecord | RumEvent

def signal_of(record: Record) -> Signal: ...

Signal is TRACES, METRICS, LOGS, RUM. signal_of is the whole routing layer — emit/base.py::EmitterSet uses it to fan a mixed stream out to the right emitter per signal.

Tick#

proteus/catalog/base.py is frozen and holds the per-interval bundle:

@dataclass(frozen=True, slots=True)
class Tick:
    ts: datetime
    interval_s: float
    service: Service
    model: WorldModel
    effect: FaultEffect
    rng: random.Random

    def jitter(self, spread: float = 0.08) -> float: ...
    def diurnal(self) -> float: ...

jitter returns a multiplier in [1-spread, 1+spread], which keeps baselines from looking fake.

diurnal returns a traffic multiplier for time of day — roughly 0.35 at 04:00 and 1.35 at 14:00 UTC:

hour = self.ts.hour + self.ts.minute / 60.0
return 0.85 + 0.5 * math.sin((hour - 8.0) / 24.0 * 2.0 * math.pi)

Real estates breathe. A flat line reads as synthetic at a glance, and the whole point of the demo is that the data looks like somebody's production.

Determinism#

tick_rng derives a stream by hashing rather than by advancing a shared one:

def tick_rng(seed: int, service_name: str, ts: datetime) -> random.Random:
    material = f"{seed}:{service_name}:{ts.isoformat()}".encode()
    digest = hashlib.blake2b(material, digest_size=8).digest()
    return random.Random(int.from_bytes(digest, "big"))

Ticks are therefore independent and evaluation order does not matter.

Guarantees the engine gives generators#

A Tick is constructed fresh per (service, ts) and never reused across instants, and each of its four signal methods is called exactly once, in the order metrics → spans → logs → RUM. That order is part of the golden-file contract. So a generator that draws from tick.rng stays deterministic even though rng is a single mutable stream shared by the four methods.

The obligation on generators#

Generators must be idempotent per Tick: calling a signal method twice on one Tick must return the same records. estimate_volume builds its own probe Tick for one instant, and a non-idempotent generator makes its projection disagree with the render it is projecting.

Named streams, not tick.rng#

Relying on the call order is legal but brittle, so the catalog does not. catalog/_shared.py::signal_rng(tick, label) derives a named stream:

def signal_rng(tick: Tick, label: str) -> random.Random:
    return tick_rng(tick.model.seed, f"{tick.service.name}:{label}", tick.ts)

Two calls with the same tick and label return identically-seeded streams. That is what lets logs() rebuild exactly the sample list spans() produced, so a log line can carry the trace_id of the span it belongs to without either method knowing the other ran. Request volume comes from the volume stream, so all three signals agree on how busy the interval was.

Child span ids and per-request choices are hashed from the parent id rather than drawn, so they consume no stream at all.

Window and ticks#

proteus/render/timeline.py holds the time-walking, and it is pure: no I/O, no wall-clock reads, no randomness. now is always a parameter.

@dataclass(frozen=True)
class Window:
    start: datetime
    end: datetime
    interval_s: float

A half-open sampling range [start, end). __post_init__ rejects naive datetimes, a non-positive interval and a reversed range. start == end is a legal empty window and yields no ticks.

tick_count and ticks(window) compute instants as integer microsecond offsets from start:

def ticks(window: Window) -> Iterator[datetime]:
    step_us = _interval_micros(window.interval_s)
    for index in range(window.tick_count):
        yield window.start + timedelta(microseconds=index * step_us)

A 14-day walk therefore accumulates no float drift, and two runs produce byte-identical timestamps. window.start is always yielded when the window is non-empty; window.end never is.

clamp_to_ingestible#

MAX_BACKFILL_DAYS = 14
MAX_FUTURE_MINUTES = 10

clamp_to_ingestible(window, now) trims a window to what CloudWatch will accept and returns (window, warnings). The future bound uses the tightest of the three signals: metrics allow +10 minutes where logs and traces allow +2 hours, so one window is safe for everything.

The warnings are not cosmetic. Backfilling outside the range is silently fatal — the request is accepted and the data never appears — so every trim produces a human-readable line rather than a quiet adjustment:

start 2025-12-01T00:00:00+00:00 is older than the 14-day ingestion limit; trimmed
forward to 2025-12-12T09:14:00+00:00 (dropped 273.2 h that would have been
accepted and then discarded)

If the whole window falls outside the range, the result is an empty window and a warning that says nothing will be emitted.

chunk_windows#

The logs and traces endpoints reject any request whose events span more than 24 hours, so a long backfill has to be requested in pieces. chunk_windows(window, max_span_hours=12.0) splits it, leaving margin for clock skew and for a batch that straddles a boundary. Chunk boundaries land on the tick grid, so concatenating ticks() over the chunks reproduces ticks(window) exactly — no instant duplicated or lost.

Batching#

batch_records(records, signal) groups a stream into lists that each fit one request, flushing on whichever of three caps binds first: record count (MAX_RECORDS_PER_REQUEST), estimated uncompressed bytes (MAX_BYTES_PER_REQUEST), and the 24-hour event time gap the logs and traces endpoints impose (MAX_SPAN_HOURS_PER_REQUEST, guarded at 23 hours).

The byte budgets sit deliberately below the hard AWS limits (5 MB, 5 MB, 1 MB) so a batch of unusually fat records cannot blow one. The RUM figures are conservative guesses — AWS publishes no per-request cap.

The emitter layer holds the hard numbers in emit/otlp.py::SPECS and enforces them by serialising and bisecting; these are the planning-side estimates. Both exist, and they are not meant to be equal.

Log records are additionally grouped by (log_group, log_stream), because that pair becomes the required headers and one request can carry only one of each. Batches for different pairs are interleaved as input arrives, so peak memory is bounded by the number of distinct pairs rather than by stream length.

A record that does not belong to signal raises ValueError. A single record larger than the byte budget is emitted alone rather than dropped.

estimate_bytes(record) is a deliberate over-estimate: resource attributes are charged to every record even though OTLP shares one resource across a batch, and protobuf framing is rounded up. Over-estimating keeps batch_records on the safe side of a hard limit; it is not an accounting tool.

render and render_counted#

def render(model, window) -> Iterator[Record]: ...
def render_counted(model, window) -> tuple[Iterator[Record], RenderStats]: ...

Rendering is streaming. A 14-day backfill at 60-second intervals across dozens of services is millions of records, so render is a generator and nothing materialises the full range.

render_counted returns the stats object immediately and mutates it during iteration, so read it after the stream is exhausted. warnings is populated on the first next(), not at call time, because nothing is resolved until then.

@dataclass
class RenderStats:
    spans: int = 0
    metrics: int = 0
    logs: int = 0
    rum: int = 0
    services: int = 0
    ticks: int = 0
    series: int = 0
    warnings: list[str] = field(default_factory=list)

Degrading rather than raising#

Sibling modules are imported lazily inside functions, and a half-built catalog still renders:

• An archetype with no registered generator means the service is skipped with a warning, not an exception.
• If proteus.faults.resolve.effect_at is unavailable, every service renders as healthy with a warning saying faults will not appear.
• An exception raised inside effect_at is deliberately not caught. A broken fault resolver should fail loudly, not silently render a healthy estate.

_adapt_effect_at reads the callee's positional parameter names once and builds the argument order from them, so it keeps working whether effect_at wants the Service or the target name, and in whichever order.

series#

series counts distinct billable metric time series: (metric name, resource identity, datapoint attribute set) triples. That is what CloudWatch charges per metric-month and rate-limits per 10-minute window.

Attribute values are part of the identity, not just the keys. One metric name emitted once per pod or per partition is that many billable series, which is exactly the cardinality a per-metric-month charge counts.

Unlike the record counts, series is never multiplied by the tick count: a series is a property of the estate's shape, so running for a fortnight creates no more of them than running for a minute.

estimate_volume#

def estimate_volume(model, window) -> RenderStats: ...

Projects the record volume of a window without rendering it. The cost estimator calls this ahead of a 14-day backfill, so it must stay O(services) rather than O(records).

Record counts come from one representative tick — the instant nearest the middle of the window, which is more typical than the edges once diurnal traffic is applied — multiplied by window.tick_count.

Two ways the projection is approximate by construction, both documented in the function:

• A fault active for only part of the window either is or is not active at the sampled instant, so fault-driven bursts are over- or under-counted.
• The sampled instant sits at one point on the diurnal curve, so any archetype whose record count scales with traffic skews toward that point. Archetypes that cap their per-tick emissions do not.

Why five probes#

series is the exception to all of that, and it gets a wider sample:

SERIES_PROBE_TICKS = 5

A series that appears on only some ticks — an error counter for a minute with no errors, a per-status-code attribute that varies — is invisible to a single sample, and under-reporting is the unsafe direction for a spend gate. So estimate_volume unions the series seen across five scattered instants while still taking the record counts from the midpoint tick alone.

Five is measured against full-render ground truth for the same window, over eight window positions × three window lengths on the default 18-service estate:

Prefer an odd value. _probe_indices forces the midpoint in, which is why an even value silently samples one extra tick, and its evenly spaced probes can alias against the 24-hour diurnal cycle — that is the 4-probe row above.

The whole estimate stays inside about 30 ms, and a 14-day projection costs the same five probes as a ten-minute one.

The obligation estimate_volume cannot check#

Capping carries the counter rule from the catalog, and this function has no way to verify it. A capped archetype must scale its reported rate counters back up from the sample to the modelled rate, or its volume charts read as a flat line at the cap. Counts here are records, not bytes.