The cost guard

Projecting volume and cost before anything is sent, the five spend policies, live AWS Pricing API rates, and the persistent spend ledger.

The cost guard answers "what is about to happen, and roughly what will it cost" before a single record leaves the machine, then decides whether the run is allowed to proceed.

Two modules, with a deliberate split:

• proteus/guard/estimate.py produces a Projection. It knows nothing about permission.
• proteus/guard/policy.py turns a Projection plus a GuardPolicy into a GuardDecision. It only reports what should happen: it never prompts, never prints, never sends, and never touches the network. The CLI and the web API own all of that.

Projection#

@dataclass(slots=True)
class Projection:
    spans: int = 0
    metric_points: int = 0
    log_events: int = 0
    rum_events: int = 0
    new_series: int = 0
    bytes_traces: int = 0
    bytes_metrics: int = 0
    bytes_logs: int = 0
    requests: int = 0
    est_usd: float = 0.0
    warnings: list[str] = field(default_factory=list)
    pricing: dict[str, Any] = field(default_factory=dict)

Counts are totals for the whole window. bytes_* are uncompressed wire bytes. requests is the total HTTP request count across all signals. est_usd is an approximate figure and must be presented as one.

pricing is the provenance block — source, region, fetched, stale — and it exists so that a fallback or stale price is visible rather than implied to be live. Show it next to the figure.

def project(model, window, *, now, session=None, region=None) -> Projection: ...

window accepts a timedelta (a window of that length ending at now), a (start, end) pair, or any object with .start and .end. now is passed explicitly so projections are reproducible.

Where the counts come from#

Counts come from render.engine.estimate_volume whenever it is importable, and from a documented per-service heuristic otherwise. Any use of the fallback is reported, never blended in silently.

That is not caution for its own sake. guard/_volume.py says why:

A table-derived figure that looks identical to a measured one is how a new_series count stayed frozen at one value while metric volume doubled — the dominant term in the cost estimate, structurally unable to move.

And _engine_counts records the other half of the same lesson:

An earlier version guessed at three possible signatures; none matched, so it silently fell back to the heuristic and overstated span volume by roughly 600x. Guessing hid the mismatch — call it directly and let a real change break loudly instead.

So the probe now calls estimate_volume(model, Window(...)) with its real signature, and every failure lands in _EngineProbe.failure as a string rather than as a bare None. Swallowing the exception is how the previous mismatch survived: a broken engine and an absent engine looked identical downstream.

The heuristic, and when it is used#

_heuristic_counts(model, minutes) derives counts from the model's shape using hard-coded per-archetype tables: request volume is base_rps * 60 * minutes, spans scale by _SPANS_PER_REQUEST, series by _SERIES_PER_INSTANCE × instances, log lines by _LOG_LINES_PER_REQUEST plus an idle floor, and RUM events only for browser services.

It exists only for when the renderer is genuinely absent, and its use produces a warning naming the reason:

proteus.render.engine.estimate_volume was unavailable, so these counts and the
series count come from the per-service heuristic in guard/estimate.py (base_rps x
60 x minutes, scaled per archetype). Treat them as order-of-magnitude only.
Reason: ...

Choosing the series count#

new_series drives most of est_usd, so _choose_series is defensive about it. The engine's measurement wins whenever it supplied a usable one. Two engine answers are rejected as unusable:

• an absent field;
• zero series alongside a non-zero datapoint count, which cannot both be true and is the signature of an unset default rather than a real measurement.

Even when the engine's answer is used, the table is still consulted as a cross-check:

SERIES_DIVERGENCE_FACTOR = 2.0

A disagreement worse than 2x produces a warning saying which way the table is wrong and that the measured figure is the one being used — because the table is still the fallback when the renderer is absent, so a stale table is a latent mispriced spend gate.

Faults inflate records but not series#

amplify = _fault_multiplier(model, start, end)
projection.spans       = int(counts["spans"] * amplify)
projection.metric_points = int(counts["metric_points"])   # not amplified
projection.log_events  = int(counts["log_events"] * amplify)
projection.rum_events  = int(counts["rum_events"] * amplify)

_fault_multiplier adds 0.5 × severity × (overlap / window) per overlapping fault, capped at 4.0 in total. Metric points are not amplified, and new_series is not either: faults do not create series, only the estate's shape does.

Requests#

metric_requests = ceil(metric_points / 1_000)    # the tightest cap
log_requests    = ceil(log_events / 10_000)
trace_requests  = ceil(spans / 10_000)
rum_requests    = ceil(rum_events / 200)

Warnings the projection raises#

Three families, all in Projection.warnings.

Cardinality. NEW_SERIES_LIMIT_PER_10_MIN = 1_000_000 is the account/region ceiling on new time series per 10-minute window. At or above it, the warning says CloudWatch will throttle or reject. Above 50% of it (NEW_SERIES_WARN_FRACTION = 0.5), the warning says high cardinality is the fastest way to both get throttled and run up a bill.

Quota. METRICS_TPS_LIMIT = 500 and LOGS_TPS_LIMIT = 5_000. The wording is careful about what actually happens, because the emitter self-throttles:

QUOTA: 1,240 metric requests/s projected against a 500 TPS account limit. The
emitter self-throttles, so the run will simply take longer than the window it
covers.

The standing caveat, appended to every projection:

Cost is list price only and never a quote: it cannot see negotiated discounts,
credits or private pricing agreements. Custom metric series are billed per
metric-month and this figure charges a full month for every new series rather than
prorating, so a short demo looks dearer than it bills.

That second sentence is worth internalising. The per-series term dominates the estimate on a small estate, and Proteus charges a whole month for a series created minutes ago. The figure is biased high, deliberately.

Pricing#

price_projection sets est_usd and returns the provenance block. It prefers live AWS Pricing API rates through proteus/pricing.py and falls back to a built-in table.

def _estimate_usd_from_card(p: Projection, card) -> float:
    logs_gb = p.bytes_logs / _GIB
    metrics_gb = p.bytes_metrics / _GIB
    return (
        logs_gb * card.usd_per_gb_logs_ingest
        + logs_gb * card.usd_per_gb_logs_storage
        + metrics_gb * card.usd_per_gb_otel_metrics_ingest
        + p.new_series * card.usd_per_metric_month
        + p.requests * card.usd_per_api_request
        + (p.spans / 1_000_000.0) * card.usd_per_million_spans_indexed
        + (p.rum_events / 1_000_000.0) * card.usd_per_million_rum_events
    )

Every term prices a quantity the renderer computes exactly — span count, request count, series count, bytes — against a published list rate. There is no invented constant anywhere in that calculation.

Spans are priced per span, not per GB. AWS publishes no per-GB span-ingest dimension, and the renderer knows the exact span count, so converting a known count into bytes only to divide it back out would add error rather than remove it.

The RateCard#

proteus/pricing.py fetches seven rates via pricing:GetProducts and caches them at $PROTEUS_HOME/pricing-{region}.json.

All seven resolve live from the API, so a healthy card contains no invented constants.

Two useful details about where the rates live. Everything Proteus needs is under the AmazonCloudWatch service code, including Transaction Search span indexing and RUM — which is not under AmazonCloudWatchRUM, since that service code returns no products. AWSXRay holds only trace storage and retrieval, neither of which applies to the OTLP traces path, so it is never queried.

The Pricing API is served from us-east-1 or ap-south-1 regardless of the region being priced; region is passed as a location filter. That filter is a human location name, not a region code, and botocore's own region descriptions are not usable for it: botocore calls eu-west-1 "Europe (Ireland)" while Pricing calls it "EU (Ireland)", a mismatch covering every eu-* region. So pricing_cache.py::REGION_LOCATIONS is an explicit map, verified against the live API, and a region missing from it is not guessed:

'ap-southeast-7' is not in this build's region-to-location map, so the Pricing API
was not queried for it - a guessed location name would silently return another
region's prices. Add it to REGION_LOCATIONS in proteus/pricing_cache.py.

Two biases, both toward over-estimating: where a dimension is tiered the first and most expensive tier is taken, and free-tier allowances are ignored entirely.

Nothing in the pricing path ever raises#

A pricing lookup is a convenience on the way to emitting telemetry, so every failure degrades and records why. The ladder in rate_card is: a fresh cache (unless refresh=True), then a live fetch, then a stale cache with a note giving its age, then BUILTIN_RATES_USD. Default staleness threshold is 24 hours.

source reports which, and it can be mixed:

Treat the mixed value as its own state, not as fully live. It happens where a feature has not launched in a region.

RateCard.notes names every assumption made and every rate that fell back. A card with a plausible-looking number and a note saying it was assumed is the normal case, not an error, so surface notes to a human.

One bug worth knowing about, because the fix reads like an oddity: price_projection constructs a boto3.Session itself when the caller passes none. Without that, project() with no session raised AttributeError inside the pricing lookup, silently fell back to the built-in table, and reported a stale figure that looked live.

The built-in fallback#

APPROX_RATES_USD in guard/estimate.py and BUILTIN_RATES_USD in pricing_cache.py are both hand-copied us-east-1 list prices, stale by construction, used only when nothing else is available. The comment above APPROX_RATES_USD is a standing instruction: check them against the published CloudWatch and X-Ray pricing pages before anyone quotes a number from this tool. Free-tier allowances, volume tiers and regional differences are all ignored.

Offline projection#

project() can reach the network. That is a real exception to the layering rule in CONTRACTS.md, and estimate.py documents it as one rather than pretending otherwise:

that file's style rule says "No network calls anywhere outside proteus/emit/, proteus/cfn/, and proteus/ai/", and proteus/guard/ is not on that list. Live pricing is currently the default path, so a cold or stale cache reaches the network from here. Either the rule needs guard/ added or the lookup needs to become opt-in; this docstring describes what the code does today rather than what the rule says it should.

Pass region="" to disable the lookup entirely. Every test must do this.

So must any caller that has promised the user nothing will leave the machine — which is exactly what the paranoid policy promises. proteus/cli/inject.py does it automatically:

return "" if guard is GuardPolicy.PARANOID else region

The ordering is what makes this necessary: project() runs before evaluate() can refuse anything. A paranoid run that priced itself against the live Pricing API would have already broken its own promise by the time the guard said no.

The five policies#

class GuardPolicy(StrEnum):
    ESTIMATE_CONFIRM = "estimate-confirm"   # the default
    BUDGET_CEILING = "budget-ceiling"
    WARN_ONLY = "warn-only"
    PARANOID = "paranoid"
    UNLIMITED = "unlimited"

evaluate(policy, projection, *, ledger_path=None, monthly_cap_usd=50.0, ttl_seconds=None) returns:

@dataclass(slots=True)
class GuardDecision:
    allowed: bool = True
    require_confirmation: bool = False
    dry_run: bool = False
    ttl_seconds: int | None = None
    banner: str | None = None
    reasons: list[str] = field(default_factory=list)

allowed=False means do not send at all. dry_run forces the emitters into dry-run mode regardless of what the caller asked for. reasons explains the decision and carries the projection's own warnings through.

Two policies carry a prominent banner:

PARANOID GUARD: telemetry will be rendered and costed but NOTHING will be sent to AWS.

UNLIMITED GUARD: every projected record will be sent to AWS with no spend check
and no confirmation. You are responsible for the bill.

paranoid and unlimited are checked first and return immediately, so neither touches the ledger.

The spend ledger#

Only budget-ceiling and record_spend read or write it. It lives at $PROTEUS_HOME/spend.json, keyed by UTC calendar month:

{
  "version": 1,
  "months": {
    "2026-09": {"usd": 12.34, "runs": 3, "updated": "2026-09-23T14:05:00+00:00"}
  }
}

def ledger_path() -> Path:
    home = os.environ.get("PROTEUS_HOME")
    return (Path(home) / "spend.json") if home else DEFAULT_LEDGER_PATH

Resolved at call time, not at import. The docstring says why:

a test or a side-by-side estate that sets PROTEUS_HOME cannot write synthetic spend into the user's real ledger — which would silently refuse their next genuine run with a correct-looking "monthly cap reached" message.

There is exactly one ledger per home, deliberately: the CLI and the web UI must agree on what has been spent.

A missing file is an empty ledger with no warning. An unreadable or malformed one is an empty ledger with a warning, and the warning is explicit about the consequence:

Spend ledger at /Users/<you>/.proteus/spend.json could not be read (JSONDecodeError);
starting a fresh ledger. Previously recorded spend for this month is not being
counted.

It never raises.

ledger_spend tolerates an older bare-number form for a month entry. record_spend adds estimated USD to the month, increments runs, stamps updated, and creates the parent directory if needed. An unwritable path is ignored and the would-be total returned: the run already happened, so failing there would be worse than losing the record.

Call record_spend after a run that actually sent data. The value is estimated dollars, not billed dollars, and everything downstream should keep saying so.

A refusal under budget-ceiling#

BUDGET CEILING REACHED: this run would take estimated 2026-09 spend to ~$61.40,
over the $50.00 cap. Refusing to send.

with the reasons carrying both the arithmetic and the ways out:

Spend ledger /Users/<you>/.proteus/spend.json records ~$38.20 USD estimated for
2026-09; this run would bring it to ~$61.40 against a $50.00 cap.
Refused by the budget-ceiling policy. Raise the cap deliberately, shorten the
window, or run under the paranoid policy to render without sending.

confirmed is load-bearing on the web path#

The browser confirm dialog is not the gate. Under any policy that sets require_confirmation, POST /api/inject refuses with 412 and the projection body unless the request body says "confirmed": true:

if decision.require_confirmation and not inject_request.confirmed:
    return fail(412, ..., detail='re-send with "confirmed": true once the '
                                 'operator has seen the estimate')

A scripted POST hits exactly the same check. Only one inject job runs at a time, process-wide; a second attempt gets 409 with the running job's id.

format_projection#

def format_projection(p: Projection) -> str: ...

Renders a projection as a multi-line console summary. It contains no square brackets, so it is safe to hand to rich.print without anything being swallowed as markup — a small thing that has bitten this project before.

Produced here for a 14-day, 60-second window over the 18-service northwind-rx preset, with region="" so no pricing call was made:

Projected volume
  spans                7,983,360
  metric points        9,354,240
  log events           1,874,880
  RUM events             483,840
  new metric series          469

Projected wire traffic (uncompressed)
  traces                 3.1 GiB
  metrics                1.0 GiB
  logs                 679.4 MiB
  total                  4.8 GiB
  requests                12,762

Estimated cost  ~ $187.50 USD  (approximate, not a quote)

Warnings
  ! Cost is list price only and never a quote: it cannot see negotiated
    discounts, credits or private pricing agreements. ...

Because region="" forced the offline path, Projection.pricing came back as {'source': 'builtin-fallback', 'region': None, 'stale': True} — which is exactly what a caller should be showing next to that dollar figure. With a live rate card the number will differ by region and by how recently AWS changed a rate.

The worked examples that produce numbers for a real estate against live pricing are in Operations.