Field notes from an SRE

Three things I used Proteus for — testing an alarm against a shaped signal, finding the fault class my alarms cannot see, and checking a CloudFormation template for monitoring coverage.

I run on-call for a mid-size AWS estate. I picked Proteus up to solve one specific problem: I had alarm thresholds I believed in and no way to test them short of waiting for an outage. What follows is what I got out of it, including a defect I found in the tool's own monitoring on the way — since fixed, and kept here because a reader who deployed the old stack needs to recognize what they are looking at.

Read the scope line first, because it changes how you should read everything below. Every number on this page came from an offline run. Rendering telemetry, synthesizing the CloudFormation template and importing a template all happen locally and cost nothing. Confirming that an alarm fires in CloudWatch needs a real account and real money, and I say so at each point where the offline work stops.

The estate throughout is the one a fresh proteus plan writes: northwind-rx, 18 services, 8 infra nodes, 3 tenants. Every command ran against a scratch state directory, PROTEUS_HOME=/tmp/proteus-scratch, so nothing touched my real world model or spend ledger.

1. Testing a latency alarm instead of trusting it#

Warning An alarm on a metric that never arrives is indistinguishable from a healthy estate. Everything in this section is a consequence of that sentence.

Our p99 latency alarms were all set at a multiple of each service's steady-state p99. Nobody could tell me why that multiple, or what it would miss. Proteus turned that from an argument into arithmetic.

bootstrap deploys two alarms per service, and their parameters are the ones I wanted to interrogate. They are SynthOptions defaults in proteus/cfn/synth.py, and you can read the whole set out of the synthesized template without deploying it:

proteus bootstrap --print | grep -A2 '"Threshold"'

TreatMissingData: notBreaching is the parameter to hold onto, because it is what makes every failure in this section quiet. An alarm with no data does not go grey and nag at you; it reads OK. The alternative is not obviously better — injection is bursty, so under missing every alarm would sit in INSUFFICIENT_DATA whenever nobody is injecting, which is most of the time, and grey-as-normal trains an operator to ignore grey. That is the same invisibility in a different color.

Two periods at 300 s is the first useful thing that fell out. Nothing in that alarm set can fire in under ten minutes, so a fault window shorter than that is not a test of the alarm — it is a test of nothing. Faults also ramp in over the first 10% of their window (Fault.ramp_at), so a 12-minute fault spends its first 72 seconds below full intensity and has 648 seconds left: 2.16 evaluation periods, which only just covers the two the alarm needs. I place nothing shorter than 30 minutes now.

The first thing I found was a mismatch, and it has since been fixed#

Before any of the arithmetic, I checked that the alarms pointed at metrics the tool emits. They did not. Run this against your own build before you read further — it is how I found it and it is how you confirm which state you are in:

proteus bootstrap --print | grep -o '"MetricName": "[^"]*"' | sort | uniq -c

On the build I started with, every one of the 18 services got the same three names:

  18 "MetricName": "proteus.service.errors"
  18 "MetricName": "proteus.service.latency"
  18 "MetricName": "proteus.service.requests"

The renderer emits none of those three names. Render five minutes of the estate and list what it actually produces:

from datetime import datetime, timedelta, timezone
from pathlib import Path
from proteus.model.store import load
from proteus.render.engine import render
from proteus.render.timeline import Window

model = load(Path("/tmp/proteus-scratch/world.json"))
end = datetime.now(timezone.utc)
window = Window(start=end - timedelta(minutes=5), end=end, interval_s=60)
names = {r.name for r in render(model, window) if type(r).__name__ == "MetricPoint"}
print(len(names), "distinct metric names")
print(sorted(n for n in names if n.startswith(("http.server", "proteus.service"))))

182 distinct metric names
['http.server.active_requests', 'http.server.client_error.count',
 'http.server.error.count', 'http.server.request.count',
 'http.server.request.duration']

182 metric names and not one proteus.service.* among them. So all 36 alarms had no metric to resolve against, and under TreatMissingData: notBreaching they read OK rather than broken.

I could not check what namespace CloudWatch files OTLP-ingested metrics under without an account, so I never claimed the namespace was also wrong. The name mismatch was sufficient on its own: a metric named proteus.service.latency was not being sent.

The interesting part is what happened when it was fixed, because the obvious fix was also wrong. Repointing the three names at http.server.* corrects the services that happen to be app-service or lambda and nothing else, because metric names are chosen per archetype. nginx-web publishes nginx.request.duration; rds-postgres publishes neither. Measured across all seven presets: 88 alarms, of which 24 named a metric their own service emits and 64 did not. On northwind-rx alone, 20 of 36 were dead. So most of those alarms were never merely misnamed — they were structurally incapable of firing, and no single global metric name could have been right for any mixed estate.

The names are now derived per archetype from a table in proteus/cfn/_alarm_metrics.py. Re-running the same command on the current build:

   8 "MetricName": "http.server.request.duration"
   8 "MetricName": "http.server.request.count"
   8 "MetricName": "http.server.error.count"
   1 "MetricName": "alb.httpcode.target.5xx"
   1 "MetricName": "alb.request.count"
   1 "MetricName": "alb.target.response.time"
   1 "MetricName": "elasticache.get.latency"
   1 "MetricName": "nginx.request.count"
   1 "MetricName": "nginx.request.duration"
   1 "MetricName": "nginx.response.5xx"
   1 "MetricName": "opensearch.search.latency"
   1 "MetricName": "postgres.read.latency"
   1 "MetricName": "proteus.agent.errors"
   1 "MetricName": "proteus.agent.invocations"
   1 "MetricName": "proteus.agent.latency"
   1 "MetricName": "proteus.rum.js_errors"
   1 "MetricName": "proteus.rum.page_load.duration"
   1 "MetricName": "proteus.rum.page_views"

Eighteen distinct names for eighteen services, where there used to be three.

northwind-rx now synthesizes about ten fewer alarms than the 36 it used to, and roughly thirty fewer across the seven presets. Every one that went away was an alarm that could not have worked, and the template shrank accordingly — proteus doctor reports its exact size under template size.

Note I am deliberately not printing exact counts, and the reason is worth more than the counts would be. Which archetypes get an alarm is a live decision: while this page was being written it changed five times, oscillating between 26 and 27 alarms for this estate, each time for a defensible reason. Any figure I pin here is true for one commit and silently wrong after the next. Two commands give you yours, and you should prefer them to anything on this page:

copy

proteus bootstrap --print | grep -c AWS::CloudWatch::Alarm
proteus doctor --region us-west-2 2>&1 | grep -A1 'template size'

This is the same discipline as the rest of the page, applied to the page. A number I cannot keep true is worse than a command that always is.

Note Count these yourself rather than trusting the numbers above. Which archetypes get an alarm is a live decision and it moved three times in one afternoon while this page was being written — each time for a defensible reason, and each time changing the totals. proteus bootstrap --print | grep -c AWS::CloudWatch::Alarm takes a second and is true for your build, which these numbers are not.

Warning This does not mean the alarms now fire. Correct metric names are necessary and may not be sufficient. Nobody has verified that these land in the proteus/{model.name} namespace under a service.name dimension once CloudWatch has ingested them over OTLP, and that needs an account. What changed is that the alarms now name metrics the renderer demonstrably emits. Confirm in your own console before you believe a green dashboard.

Two supporting details I could check. service.name is a resource attribute on every metric, so the dimension the alarms use was always the right dimension — it was only the metric name that did not line up. And every metric carries proteus.aws_metric and proteus.aws_namespace attributes naming the AWS metric it models, which is how you find out that http.server.request.duration stands in for Latency under AWS/ApplicationSignals.

Alarmability is decided by the instrument kind, not by the metric name#

This is the part I would not have worked out on my own, and it is the durable lesson rather than the bug.

rds-postgres emits fifteen metrics, two of them named postgres.read.latency and postgres.write.latency. Anything scanning names — me, or a tool — concludes it is alarmable. Both are gauges. Print the kind alongside the name and the picture changes:

archetype        metric                            kind        bounds     top
alb              alb.target.response.time          histogram       13   10000
app-service      http.server.request.duration      histogram       13   10000
elasticache      elasticache.get.latency           histogram       12     250
opensearch       opensearch.search.latency         histogram       13   10000
bedrock-agent    proteus.agent.latency             histogram       14   30000
browser-rum      proteus.rum.page_load.duration    histogram       14   30000
rds-postgres     postgres.read.latency             gauge            0       -
rds-postgres     postgres.write.latency            gauge            0       -

The metric name is not the metric, and two things follow from the kind.

A percentile over a gauge is not the statistic you think you asked for — but it is a statistic. A gauge carries one number per datapoint, so ExtendedStatistic: p99 over it is the 99th percentile of however many readings land in the period: five, at one a minute over a 300-second period. That is effectively the period's worst reading. It is a real signal about a real thing; it is just not a distribution percentile, and saying "no percentile is possible here" overstates it.

And the gauge is on a different scale from the threshold. postgres.read.latency measures storage read latency; base_latency_p99_ms describes the service's request latency. p99 × 2.5 across the two multiplies unrelated quantities. On orders-db it asks for 112.5 ms from a metric that sits flat at 3.15 ms healthy and reaches 12.60 ms under the strongest latency fault the library ships — roughly 9× past anything reachable. Such an alarm deploys, looks right, and never fires.

Whether this archetype should get an alarm at all is unsettled, and both answers are defensible. At the time of writing it gets none, which is the conservative reading: the metric exists but the usual alarm shape does not fit it. I have also seen the build carry this alarm instead, and it is the version worth copying if you are solving the same problem elsewhere:

AlarmName          proteus-northwind-rx-orders-db-latency-max
MetricName         postgres.read.latency
Statistic          Maximum
Threshold          6.0
Period             300        EvaluationPeriods 2
TreatMissingData   notBreaching

Two things make that the good answer rather than a compromise. Maximum over a gauge is unambiguous where a percentile is arguable — it asks for the worst reading in the period, which is precisely what the instrument can answer. And the threshold is declared on the metric's own scale rather than derived from the service baseline. So both halves of the original defect are addressed by changing the alarm, not the name.

The rule that falls out generalizes past this codebase: histogram-backed latency may use a baseline multiple; gauge-backed latency must declare its own threshold and should not borrow a statistic it cannot support.

None of this is an inconsistency beside elasticache and opensearch, which also publish two latency candidates and get ordinary p99 alarms. Those two offer a choice between two histograms — a real question about which signal. This one asks what statistic is meaningful at all, which is a different question with a different answer.

The 6 ms threshold is measured rather than picked. I rendered the gauge healthy and under fault to check it:

The baseline is flat because strain saturates, so there is no jitter to false-positive on — which is why a 1.9× threshold is safe here where it would be reckless on a request-latency metric.

Which instrument CloudWatch can derive a percentile from is unresolved#

I have to be careful here, because the intuition everyone on this reached first may be backwards, and nobody has an account to settle it.

CloudWatch percentiles require raw unsummarized datapoints. For a pre-aggregated statistic set they work only in two narrow cases: SampleCount is 1 with Min, Max and Sum all equal, or Min equals Max. Neither exception can hold for these histograms — count is 200 and the mass is spread across several buckets. Whether CloudWatch derives percentiles from OTLP explicit-bucket counts at all is not documented in either direction, so the histogram-backed p99 alarms rest on an assumption.

A gauge, meanwhile, emits one raw value per datapoint, which is the documented supported case. So the possibility worth holding open is that the 14 histogram-backed p99 alarms are the doubtful ones — the opposite of where I started. I am not asserting that either. Both directions are unverified and I would not change an implementation on a documentation inference.

Note that the database alarm sidesteps the whole question by using Maximum, which is defined for a gauge whatever CloudWatch does with bucket counts. That is worth noticing as a pattern: when you cannot establish that a statistic is supported, a statistic you know is supported is worth more than the one you wanted.

If you have an account, this is cheap to settle and it is the single most valuable thing you could send back upstream. Deploy one p99 alarm, run one short seed, and look at whether the alarm sits in INSUFFICIENT_DATA while its metric visibly has data:

proteus bootstrap
proteus seed --days 0.02

An alarm in INSUFFICIENT_DATA beside a populated metric is the answer. So is an alarm that transitions. Either way you will have learned something no amount of reading the template can tell you — which is the whole thesis of this page, applied to the page itself.

Where there is a real choice, one stated rule resolves it: alarm on the path a user-facing request waits on. So read, get and search latency are chosen over write, set and indexing, which are operator concerns that do not describe what a customer experiences. A rule stated once and applied three times is the difference between a decision and three coincidences.

Then the arithmetic, which is the actual value#

A fault's latency effect is a multiplier, and severity interpolates it toward 1.0 (FaultEffect.scaled: 1 + (mult − 1) × severity). An alarm at 2.5× baseline therefore trips when 1 + (mult − 1) × severity > 2.5. Both halves of that inequality are readable from the source, so I can compute coverage for every fault kind without sending a single datapoint:

from datetime import datetime, timedelta, timezone
from pathlib import Path
from proteus.model.schema import Fault
from proteus.model.store import load
from proteus.faults.library import effect_for, known_kinds

model = load(Path("/tmp/proteus-scratch/world.json"))
svc = next(s for s in model.services if s.name == "orders")
start = datetime.now(timezone.utc)
end = start + timedelta(hours=1)

latency_alarm = max(100.0, svc.base_latency_p99_ms * 2.5)
for kind in known_kinds():
    f = Fault(id="probe", kind=kind, target=svc.name,
              start=start, end=end, severity=1.0)
    e = effect_for(f, svc, model)
    p99 = svc.base_latency_p99_ms * e.latency_mult
    err = (svc.base_error_rate + e.error_rate_add) * 100
    trips = [n for n, hit in (("latency", p99 > latency_alarm),
                              ("errors", err > 5.0)) if hit]
    print(f"{kind.value:28} {p99:7.0f} ms  {err:6.2f}%  {', '.join(trips)}")

For orders — baseline p99 180 ms, so a 450 ms latency alarm, and a 0.18% baseline error rate against a 5% alarm — every one of the 14 fault kinds at full severity:

Seven of the fourteen trip neither alarm at full severity. Those seven are queue-depth, oom-kill, disk-full, noisy-neighbor, cache-stampede, agent-quality-regression and network-packet-loss.

That is not a claim about your thresholds. It is the shape of the answer you get from comparing a threshold multiple against a signature multiplier — an exercise nobody on my team had done, because until this there was nothing to do it against. I re-ran the same script against our own baselines and our own thresholds, which took about twenty minutes and produced the first coverage argument we have had that was not somebody's intuition.

Three details worth stealing:

• slow-query needs severity above 0.5 to trip a 2.5× alarm, because its multiplier is 4.0 and 1 + 3 × 0.5 is exactly 2.5. The comparison operator is GreaterThanThreshold, so a tie does not fire. Severity 0.5 sits on the threshold; 0.51 clears it. Read that as arithmetic on the model, not as a prediction about what a bucket-interpolated p99 will read — see the resolution limit below.
• The 100 ms floor makes your fastest services your least covered. session-cache has a 6 ms baseline p99, so its threshold is the floor: a 16.7× regression before anything fires. event-bus is the sharper case and a different one — it is a kafka service with no latency signal at all, so the floor never enters the question and it gets no latency alarm. A threshold you cannot compute beats a threshold you cannot reach, because only one of them is visible.
• agent-quality-regression moves latency, errors, CPU and tokens by nothing at all. It is visible only in an evaluation-score metric. If you are putting a Bedrock agent behind a conventional RED alarm set, that row is the whole argument.

The resolution limit on all of it#

Latency reaches CloudWatch as an OTLP explicit-bucket histogram, so every percentile is interpolated inside a bucket. A threshold is therefore only resolvable to the width of the bucket containing it, and that width varies by nearly three orders of magnitude across one estate:

There are three bucket-bound sets in play, not one: elasticache uses twelve bounds from 0.05 ms to 250 ms, most archetypes use thirteen to 10,000 ms, and bedrock-agent and browser-rum use fourteen to 30,000 ms. So a general claim about bucket width is wrong for most services — check the set your service actually uses.

Note which service comes out worst. rx-advisor has both the coarsest latency alarm on the estate and the least relevant one, since the Bedrock agent's characteristic failure moves no latency at all.

Two more limits on the same figures. The histogram's count is a capped sample — 200 — rather than request volume, so every percentile here is a sample p99. And the emitter does not set OTLP's optional min and max on the datapoint, which is a one-line gap rather than a design problem, but worth knowing if your percentile derivation wants them.

Two alarms that could not fire, one step further out#

The 30,000 ms ceiling exists because of a defect of exactly the kind this section is about, found in a preset I had not checked. On ml-pipeline, four services baseline in the tens of seconds, so × 2.5 puts their thresholds past anything the histogram can report — but only two of the four were ever alarm candidates:

train-flow     45,000 -> 112,500 ms  stepfunctions.execution.time       suppressed by ceiling
analytics      22,000 ->  55,000 ms  athena.query.total_execution_time  suppressed by ceiling
orchestrator   28,000 ->  70,000 ms  -                                  no latency metric at all
transform      19,000 ->  47,500 ms  -                                  no latency metric at all

A GreaterThanThreshold above the largest bucket bound compares against a number the metric cannot produce — the green-forever failure again, reached by a different route. train-flow and analytics now get no latency alarm rather than a decorative one, which is part of why ml-pipeline synthesizes 2 alarms for 8 services. Keep the two reasons distinct: two services were suppressed by the ceiling, and two never had a metric to alarm on. Both end at no alarm, and only one of them is about the threshold.

And one that is not fixed, which you should know about if you place any of these workloads. The profile renderer buckets every latency metric over the 10-second request bounds regardless of the service's baseline, so metrics that legitimately run longer spill past the top bound:

stepfunctions.execution.time            42% of mass above the 10,000 ms top bound
athena.query.total_execution_time        14%
athena.query.engine_execution_time        9%

None of those is the whole distribution, so the metric is not useless. But a percentile above the overflow fraction lands in a bucket with no upper bound and cannot be interpolated at all. For all three, p99 specifically is unbounded — everything from p58 up for stepfunctions.execution.time. If you are alarming on those, the number you are comparing against does not exist yet.

Notice that those are the same two metrics the ceiling just suppressed alarms for. The ceiling stops you deploying an alarm whose threshold the metric cannot reach; this defect is why the metric cannot reach it. One is the symptom handled and the other is the cause still open, which is worth knowing if you were about to conclude the ml-pipeline estate is now fully covered. It is not — it is now honest about not being.

Which archetypes can carry an alarm at all#

This is the table I now check first on any estate, because it decides whether the coverage arithmetic above is even applicable. Read it as my reading of the output below, not as the primary artifact — the output is the evidence and this is an interpretation you are free to disagree with.

Six archetypes get no error-rate alarm because none of them emits a request counter and an error counter over one population. Inventing a ratio from the two nearest-looking counters would produce well-formed meaningless alarms, which is worse than none. Three archetypes get nothing at all — kafka, eks-workload, ecs-workload — because they report only saturation and health as gauges: consumer lag, partition health, node conditions, pod phase counts, and nothing shaped like a request rate, an error count or a latency. An alarm on saturation would be defensible and is deliberately not there, because that is a different kind of alert from the two this stack deploys.

So on eighteen services, three carry no latency alarm at all — inventory-consumer, claims-batch, event-bus — plus orders-db while its shape is unsettled. Six carry no error-rate alarm, adding session-cache and catalog-search. Not one of those gaps is an oversight; each is a metric that does not exist or a ratio that would have to be invented. That is what I would want to know before telling anyone an estate had been rehearsed, and it is what the old build hid behind 36 green alarms.

Regenerate it for your own build. Note there is no name filter in this, on purpose — see the next section for why:

from collections import defaultdict
from datetime import datetime, timedelta, timezone
from pathlib import Path
from proteus.model.store import load
from proteus.render.engine import render
from proteus.render.timeline import Window

model = load(Path("/tmp/proteus-scratch/world.json"))
archetype = {s.name: str(s.archetype) for s in model.services}
end = datetime.now(timezone.utc)
window = Window(start=end - timedelta(minutes=10), end=end, interval_s=60)

emitted = defaultdict(set)
for r in render(model, window):
    if type(r).__name__ == "MetricPoint":
        svc = r.resource.attributes.get("service.name")
        if svc in archetype:
            emitted[archetype[svc]].add((r.name, r.kind.value))

for arch in sorted(emitted):
    print(f"{arch}  ({len(emitted[arch])} metrics)")
    for name, kind in sorted(emitted[arch]):
        print(f"    {kind:10} {name}")

Twelve archetypes and 175 archetype-attributed metric names on this estate. On a page nothing automatically checks, a list you regenerate stays true for your build after the mapping changes again; a table I transcribed is true only on the day I wrote it.

Why there is no filter in that snippet#

I wrote one first. It cost me two false findings in an afternoon, and it is the most transferable thing on this page.

My filter looked for metric names containing "latency", "duration", "count", "error", "request". It dropped alb.target.response.time, so I reported that alb emits no latency metric. It dropped proteus.rum.page_views, so I reported that browser-rum had no denominator for an error rate. Both archetypes were fine. I caught the first myself and only found the second because a colleague contradicted it.

Then I made the same mistake in the other direction: comparing every histogram against a latency threshold flagged payments' connection-wait histogram and rx-advisor's token-usage histogram as thresholds stranded above their top bound. Neither is a latency metric. That query returned too many rows rather than too few.

All of it is one failure, and it is the same failure as the alarm bug that opens this section:

An empty or over-full result from a badly scoped query is shaped exactly like a real answer, and nothing in the output tells you which you got. The defense is not "beware of filters that hide things" — it is know what you are comparing against what, and look at the unfiltered thing at least once. The three genuinely empty archetypes in the table above survived scrutiny only because I dumped their full metric lists instead of trusting the filter.

I also checked those empty rows against the obvious trap before reporting them. "No metric in a healthy window" is the same shape as the bug that once created log groups by coin flip, because some records only appear while a fault is active. So I re-rendered with all 14 fault kinds active on all 18 services at severity 1.0 over 30 minutes: the per-archetype name sets came back identical. Those rows are structural, not a sampling artifact.

What confirming this in CloudWatch costs#

The arithmetic above is free. Watching the alarm actually transition is not: seed, live and firehose all make real API calls and CloudWatch bills for it. A 14-day backfill of this 18-service estate projects about $153 at us-west-2 list price. Every injecting command routes through the cost guard, which projects volume and dollars and asks before sending. Size a run first:

proteus seed --days 1 --dry-run --yes

--dry-run forces the paranoid policy, which renders and prices everything and makes no network call at all. One run of that command on a fresh 18-service estate came back at ~$144.04 USD — under paranoid that figure comes from the built-in fallback rate table rather than the live Pricing API, and it charges a full metric-month for every new metric series rather than prorating, so it is biased high by design. Read the cost guard before you quote any number it prints.

2. The fault class my alarms structurally could not see#

noisy-neighbor is the row in that table I care about most, because shared-database contention is the incident my team is worst at diagnosing. Look at its numbers again: 396 ms against a 450 ms threshold, 0.78% against a 5% threshold. It trips neither.

That is not a severity problem. Its victim-side latency multiplier is 2.2 at full intensity (proteus/faults/library.py::noisy_neighbor), and 2.2 is below 2.5. A p99 alarm set at 2.5× baseline cannot detect this fault kind at any severity. I would not have found that by reading my own alarm config, because the alarm config does not contain the fault's signature.

Placing it, and reading the asymmetry#

I wanted an exact target, severity and window, so I edited the world model directly rather than going through chat or the GUI canvas. The fault schema is small:

{
  "id": "f-noisy",
  "kind": "noisy-neighbor",
  "target": "rds-orders",
  "start": "2026-09-27T14:38:51+00:00",
  "end": "2026-09-27T15:38:51+00:00",
  "severity": 0.7,
  "params": {},
  "hidden": false,
  "note": "co-tenant contention on the shared orders RDS"
}

target is an InfraNode id, not a service. That is not optional for this kind: aim noisy-neighbor at a service name and the resolver reports it as INERT and tells you it is a modeling error rather than degrading anything.

Warning That distinction will cost you a measurement if you miss it, and it cost one on this project. Pointed at the infra node rds-orders, a full-severity noisy-neighbor takes orders-db's postgres.read.latency from 3.15 ms to 6.93 ms. Pointed at the service orders-db, the same fault leaves it at 3.15 ms flat — because it is inert, not because the fault does not affect that metric. A flat line from a misaimed fault looks exactly like a flat line from a fault that does nothing. proteus show and explain() both say INERT in words; read them before concluding a signal is unmoved.

proteus show confirms the placement, and proteus/faults/resolve.py explains what each service gets. On this estate rds-orders is used by orders (tenant t-alpha), claims-batch (t-beta) and orders-db (untenanted). At severity 0.7 the resolver picked t-alpha as the aggressor — deterministically, from a hash of the fault id, so it replays identically. explain() returns one line per fault; the service name at the left of each block below is my own label for which service I asked about.

orders        noisy-neighbor [f-noisy] as the aggressor tenant on shared
              rds-orders at intensity 0.70 - latency x1.04; request rate x1.25;
              cpu +29.4 pp; 1 log line(s)

claims-batch  noisy-neighbor [f-noisy] as a noisy neighbour on shared rds-orders,
              starved by tenant 't-alpha' at intensity 0.70 - latency x1.84;
              error rate +0.0042 abs fraction; cpu +18.2 pp; 3 log line(s)

The service causing the problem has flat latency and elevated CPU and request rate. It looks like it is working hard. The service suffering has nearly double the latency and a small error rate, with CPU elevated by contention it did not cause. If you triage by "which service looks worst", you start on the victim and stay there.

The victim's log lines include no local deploy or config change in window, which is the clue that is supposed to bounce you off the victim and onto the shared resource. That line existing is the difference between a puzzle and a trap.

What the alarms see of it, now that the database has one#

Worth redoing the coverage question for this specific fault, because the database alarm changes the answer and not by much:

So the fault that is hardest to diagnose is nearly invisible to the alarm set, and the one alarm that could catch it needs the aggressor running flat out. At severity 0.6 — a perfectly realistic noisy neighbor — nothing fires anywhere on the estate while a tenant's latency doubles. A database alarm here moves detection from "never" to "at severity 1.0", which is worth having and is not the same thing as coverage.

This is the argument for alarming on the shared resource's contention signals — CPU steal, iowait, storage queue depth — rather than on per-service latency multiples. Those move first and they move on the aggressor's side too.

Using it to train#

hunt hides a fault and grades the diagnosis on three axes — root cause at 50%, affected resource at 30%, time window at 20% — and withholds its own explanation until the overall score reaches 70, substituting a hint that is screened so it cannot contain the fault kind or any hyphen-separated part of it.

I have not run a graded hunt: hunt start and hunt guess both call Bedrock, and I had no model access in the account I was working in. What I did verify offline is the one property that makes the exercise safe to run in front of people, because a leak here wastes everybody's afternoon. Set a fault's hidden flag and show stops naming it:

1 hidden fault(s) active from an open hunt - details withheld. Use proteus hunt
reveal <id> to see them.

Run that check before you run the exercise. If show names the kind, target or window, the answer is already on screen and you are on an older build.

The training value, to be clear about what I am and am not claiming: an operator who has seen a noisy-neighbor signature once recognizes the second one faster. I have no measurement of that and would not trust one taken on a sample of my team. What I can say concretely is that the estate misleads in the right direction — the culprit looks healthy and the victim looks broken — which is the property a training scenario needs and the property a hand-built one usually lacks.

3. Checking a template for monitoring coverage#

import-cfn reads a CloudFormation template and builds a world model from it. It makes no AWS call and never modifies the template. I used it for something the docs mention almost in passing: the unmapped section of its report is a coverage report on your own monitoring.

Against a trimmed version of our root stack:

proteus import-cfn root.yaml --dry-run

                NOT mapped - no telemetry is generated for these
 logical id    ┃ resource type              ┃ why
━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 PaymentsStack │ AWS::CloudFormation::Stack │ this is a nested stack and its
               │                            │ template was not supplied; pass
               │                            │ --include <LogicalId>=<path> to
               │                            │ expand it, or everything inside
               │                            │ it is absent from the estate
1 resource(s) are absent from the estate. Any alarm or dashboard that depends on
them is not being exercised by this replica.

That is the sentence to act on. Our payments tier lives in a nested stack. Had I built the replica, seeded it and pronounced our alarms rehearsed, I would have rehearsed nothing about payments — and nothing in the console would have looked wrong, because an absent service produces no failing signal. The report also refuses to follow the TemplateURL it prints; you name every child explicitly with --include, which is a structural boundary rather than a policy.

The same run told me something I had not thought to ask:

no dependency edges were found. Every service will draw as an island in
CloudWatch Omni.
  note: the derived estate is 4 disconnected groups, not one topology

Omni builds topology from dependency edges, and my template states which resources exist without stating which calls which. So the service map in my rehearsal would have been four unconnected nodes. Finding that at my desk rather than in a go-live review is worth the ten seconds the command takes. Proteus invents no edges to paper over it, which is the right call — a fabricated edge in a service map is worse than a missing one.

--strict exits non-zero when anything is unmapped, so this runs as a CI gate:

proteus import-cfn root.yaml --strict --dry-run

I confirmed the exit codes: 1 with --strict and an unmapped resource, 0 without. What that gate buys you is a build failure when somebody adds a resource type the replica cannot model, which is the moment your rehearsal quietly stops covering a piece of your estate.

What this will not do for you#

The reason to trust the three sections above is that I am equally specific here.

Synthetic data is shaped, not real. Proteus reproduces a class of pathology, not your service's pathology. A slow-query fault gives you the signature of a query plan regression: latency up, CPU and errors flat, matching log lines. It does not give you the particular way your ORM misbehaves under that condition. If your question is "would my alarm catch a latency-only regression of roughly this size", you get a real answer. If your question is "would my alarm catch the incident we had in March", you do not.

Correct metric names are not proof an alarm fires. The stock alarms now name metrics the renderer demonstrably emits, which was not true when I started. Nobody has yet confirmed they resolve in CloudWatch once ingested over OTLP — the namespace, the dimension and whether ExtendedStatistic: p99 resolves against an explicit-bucket histogram all need an account. Run the reproduction command on your own build to see which state you are in, then check the console rather than the template.

import-live's baselines are less trustworthy than they look. Its metric names and dimensions were written from AWS documentation and have not been validated against a live account response. A wrong dimension name does not raise — it returns an empty series, which the absent-metric logic then correctly reports as absent, which the report then correctly describes as a metric that was never published. Every layer tells the truth and the conclusion is still wrong: a busy service reads as unmonitored, and an absent you cannot distinguish from a real monitoring gap is worse than no output. Run --discover-only first, then one narrow --services read against an account whose answer you already know, and treat any absent on a service you know is busy as evidence the tool asked the wrong question. See import-live.

It spends money, and the estimate is deliberately pessimistic. A short run projects almost as much as a long one because every new metric series is charged a full metric-month at the most expensive tier. Do not size a budget ceiling off a one-minute projection — the default --cap 50 refuses even a one-minute run of this estate. Use --dry-run while you are learning what a command does.

Faults do not propagate along depends_on. This caught me out. A full-severity cpu-saturation on orders leaves checkout completely healthy, even though checkout declares depends_on: [cart, payments, orders]. Propagation happens through shared infrastructure via uses_infra, or explicitly with dependency-timeout placed on the caller. That is a defensible modeling choice — cascade depth has no correct default — but if you place a fault and see no effect on the service you expected, this is probably why.

The short version#

Three things a console cannot give me: the severity at which each fault kind crosses my alarm threshold, proof that one fault class sits permanently under it, and a list of the resources in my own template that my rehearsal does not cover. All three were free and offline. The expensive part — watching the alarm transition in CloudWatch — I now only do for the cases the arithmetic says are interesting, which is a better use of the money than seeding fourteen days and hoping something goes red.

The thing I did not expect was that the most useful output of testing a monitoring tool was a defect in its own monitoring, and that the defect and my two mistakes finding it were the same mistake. A query that returns nothing, or too much, looks exactly like an answer. That is true of an alarm on a metric nobody publishes, of a CloudWatch dimension that does not exist, and of a grep over names you did not write. If you take one thing from this page, take that rather than the numbers — the numbers are for the build I ran, and every command that produced them is here so you can get your own.