Troubleshooting

Symptom-first fixes for every failure this project has actually hit, starting with the four that return HTTP 400.

Every entry below is a failure that really happened during development, not a hypothetical. Each one is now enforced or detected in code, which is why proteus doctor is the first thing to run.

Symptom table#

Start with doctor#

proteus doctor

It is read-only — it never changes an AWS resource and never sends telemetry — it takes seconds, and every failing check prints the exact command that fixes it. Most entries on this page are a symptom doctor would have named before you hit it.

Reading the output#

 check         ┃      ┃ detail
━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 region        │ OK   │ us-west-2 supports CloudWatch Omni
 credentials   │ FAIL │ could not resolve credentials: TokenRetrievalError: Error when
               │      │ retrieving token from sso: Token has expired and refresh failed
 world model   │ OK   │ northwind-rx rev 1: 18 services, 8 infra, 3 tenants, 0
               │      │ fault(s), fingerprint d7caf912a5a9862f
 template size │ WARN │ template 37,797 bytes across 47 resources, sent inline (74% of the
               │      │ 51,200-byte limit)

suggested fixes
  credentials: aws sso login   (or refresh whatever provides this profile)
  template size: no action needed: bootstrap stages large templates to S3 automatically

Four verdicts:

Every failing and every actionable warning check prints its fix under suggested fixes, keyed by check name. So the recovery loop is: read the failing row, run its fix, run doctor again.

Two structural details that explain output you might find odd.

The offline checks run before the credentials gate. world model and template size need no AWS call, so they still report on a machine with no working profile — a template that cannot deploy is worth knowing about either way.

Everything after credentials FAIL is absent, not passing. With no session, transaction search, log retention, log groups, stack, model vs stack and bedrock are not attempted at all. An empty row is not a green row.

The same checks are available in the GUI under the header's doctor button, with identical verdicts — see the GUI walkthrough.

Every span is rejected with HTTP 400#

Symptom. seed, live or firehose reports every trace batch rejected. The traces row shows accepted 0 and a non-zero rejected, and the error mentions UpdateTraceSegmentDestination. Metrics and logs may be fine.

Cause. CloudWatch Transaction Search is enabled but its destination is XRay. The OTLP traces endpoint requires destination CloudWatchLogs. It is not enough for Transaction Search to be active — with the wrong destination, every span is rejected with a 400.

Fix.

proteus bootstrap

bootstrap sets the destination as part of the stack. If you would rather do it by hand, that is a resource policy granting xray.amazonaws.com logs:PutLogEvents on aws/spans, then:

aws xray update-trace-segment-destination --destination CloudWatchLogs

Confirm the fix. proteus doctor reports:

 transaction search │ OK │ destination CloudWatchLogs, ACTIVE

If it reports destination CloudWatchLogs but status <something else>, that is a WARN: the setting is right and has not finished propagating. Wait a few minutes and re-run doctor rather than changing anything.

Every log batch is rejected with HTTP 400#

Symptom. The logs row shows accepted 0 and a non-zero rejected. Traces and metrics are accepted normally. Often it is some services rather than all of them.

Cause. The log group does not exist. The logs OTLP endpoint returns HTTP 400 for any group it cannot find — there is no implicit creation.

The reason this bites selectively is worth understanding, because it shaped the fix. Some services only log while a fault is active: their declared log lines are driver-gated, so they are silent in a healthy window and log heavily the moment a fault lands. An earlier version of the group list was sampled from a short render. A 5-minute probe is five ticks; an undriven line at frequency 0.12 misses all five about 53% of the time. So edge-cdn's log group was created by coin flip, and orders-queue and rx-inference — whose only declared lines are driver-gated — got no group at all. Measured afterwards, all 13 applicable fault kinds make each of those services emit 58–120 log records per 30 minutes, every one of which would have been rejected with a 400 — at exactly the moment you most needed them.

The group set is now derived per service from the model, unioned with a rendered probe as a safety net, rather than sampled.

Fix.

proteus bootstrap

That re-applies the stack with the full derived group set.

Confirm the fix. doctor has a dedicated log groups check that names the missing ones:

 log groups │ FAIL │ 3 log group(s) the renderer can write to do not exist:
            │      │ /proteus/northwind-rx/edge-cdn, /proteus/northwind-rx/orders-queue,
            │      │ /proteus/northwind-rx/rx-inference. Every log batch for those
            │      │ services will be rejected with HTTP 400 - including services that
            │      │ only log while a fault is active, which is when you most need them.

After the fix it reports all 22 group(s) the renderer can write to exist.

Logs are accepted with HTTP 200 but never appear#

Symptom. The logs row shows healthy accepted counts and rejected 0. The run looks like a complete success. Then the log events are not retrievable in CloudWatch — not in Logs Insights, not in the log group, not correlated to a trace.

Cause. The log group's retention period is shorter than your backfill window. CloudWatch Logs accepts a backdated event with HTTP 200 and then discards it if its timestamp predates the retention period. Nothing reports the loss.

Verified directly: events at −25 hours and −10 days into a group with 1-day retention returned 200 and were never retrievable.

This is the worst failure mode on this page, because every signal your tooling has says the run succeeded.

Warning There is no way to recover discarded events. They were never stored. Fix the retention and seed again.

Fix. Default retention is 14 days, which matches the maximum backfill horizon. If you lowered it, raise it:

proteus bootstrap --log-retention-days 14

bootstrap now refuses to synthesize a template that would lose data, so this cannot be configured wrong going forward:

refusing to synthesise a template that would silently lose telemetry.
LogRetentionDays=1 days is shorter than the 14-day backfill horizon; events older than
the retention period are accepted with HTTP 200 and then silently discarded. Raise
retention to at least 14 days.

If you genuinely want a shorter retention, tell it the shorter horizon too and it will accept the pair:

proteus bootstrap --log-retention-days 7 --seed-days 7

Confirm the fix.

proteus doctor --days 14

 log retention │ OK │ 22 log group(s), all retaining at least 14 days

Pass the --days you actually intend to seed. Checking with --days 7 and then seeding 14 passes a check that should have failed.

Telemetry lands in the wrong region#

Symptom. Either telemetry appears in a region you did not target, or every batch fails with HTTP 400 for log groups that you can see exist.

Cause. Region split-brain. The emitters build their endpoint URL from an explicit region argument, but every other client — CloudFormation, Logs, X-Ray, Bedrock, Pricing — is created from the boto3 session and inherits its region. When --region reached the emitters but the session was left unset, the session resolved to the profile's ambient region. The stack went to us-east-1 while telemetry went to us-west-2, so the log groups did not exist where the logs were being sent, and every log batch failed with a 400.

The region is now set on the session as well as passed to the emitters. What is left to you is consistency across commands.

Fix. Pass --region the same way to every command in a session:

proteus bootstrap --region us-west-2
proteus doctor    --region us-west-2
proteus seed      --region us-west-2

Or set it once and stop thinking about it:

export PROTEUS_REGION=us-west-2

PROTEUS_REGION is the default for every command's --region, falling back to AWS_REGION and then to us-west-2.

Confirm the fix. proteus about prints the resolved default region. proteus doctor --region <region> reports stack <name> is ... for the stack in that region — if it says the stack does not exist, you are looking in the wrong one.

Note CloudWatch Omni is documented in us-east-1, us-west-2 and eu-west-1. Outside those, doctor warns: telemetry is still ingested by CloudWatch, but the Omni experience may not be available.

Nothing appears in Omni for a few minutes#

Symptom. A run completed with rejected 0 and no errors, and the service map, traces or logs are still empty.

Cause. CloudWatch indexes newly ingested telemetry asynchronously. Acceptance and queryability are not the same event.

Fix. Wait. The CLI prints this itself after any run that actually sent data:

CloudWatch indexes newly ingested telemetry asynchronously; allow a few minutes
before it is queryable in Omni.

Cost Do not re-inject. A second run doubles the spend, doubles the data volume, and does nothing to make the first run's data index faster. If you re-inject a seed, you also get duplicated history, which makes the demo look wrong in a way that is tedious to undo.

Confirm. Come back in a few minutes and refresh. If the data is still absent after that, go back to the 400-level entries above: a run can report success on metrics and traces while logs were being dropped.

bootstrap fails with a template-size error#

Symptom. bootstrap aborts complaining about template size. Because it aborted, the new services' log groups were never created — and the next seed partially fails on log groups that do not exist, which looks like an unrelated problem three commands later.

Cause. CloudFormation caps an inline template body at 51,200 bytes. Every service adds a log group, and most add one or two alarms, so the template grows with the estate and eventually crosses that line.

A fresh 18-service plan synthesizes 47 resources — 18 log groups, 27 alarms, 1 SNS topic and 1 RUM app monitor — at 37,797 bytes, which is 74% of the inline limit. That leaves roughly 13 KB of headroom, so a handful of extra services is comfortable and a few dozen is not.

Fix. Nothing. Proteus stages templates above the inline limit to S3 automatically, which lifts the ceiling from about 21 services to roughly 400 — at which point you meet CloudFormation's own 1 MB limit rather than this one. The staging bucket is created with public access fully blocked and a 7-day expiry on staged objects.

Confirm. doctor has a template size check that reports the headroom. It is offline — it synthesizes the template in memory and measures it, with no AWS call.

On a fresh 18-service estate, under the limit:

 template size │ WARN │ template 37,797 bytes across 47 resources, sent inline (74% of the
               │      │ 51,200-byte limit)

Grow the estate far enough and the same check switches wording: it reports being over the 51,200-byte inline limit, says staging is required, and gives its percentage against the 1,000,000-byte limit instead. That second percentage is the one that tells you how much further the estate can grow, because staging is what you are measuring against by then.

WARN is the expected steady state either way, and the fix line says so explicitly rather than leaving a yellow row with no action attached. Read the wording, not just the color. sent inline (N% of the 51,200-byte limit) means you have headroom; language about being over the limit means staging is carrying you, and that is not a problem to solve.

No second transcript is shown here on purpose. Reproducing one means constructing an estate large enough to cross — an estate nobody would otherwise build — and the figure would then be an artifact of this page rather than of the tool. The mechanism is the part worth knowing, and it does not depend on a number: each service adds a log group plus a variable number of alarms, the template grows, and at some size bootstrap stages to S3 without being asked.

The cost estimate looks absurd on a short run#

Symptom. On the 18-service estate a fresh proteus plan gives you, a one-minute run projects about $138–$141, a one-day run $141.54, and a fourteen-day run $152.53. The one-minute figure is obviously wrong for a minute of data, and the three barely differ despite covering wildly different amounts of it.

Cause. Two deliberate pessimisms in the projection, both in the same direction:

1. Custom metric series are charged a full metric-month each, never prorated. The 18-service estate touches 469 distinct series over 14 days and around 460 over any given minute. At $0.30 per metric-month that is roughly $140 either way — almost the whole total, and almost independent of how long the run lasts. That is why all three figures above sit within $15 of each other.
2. The most expensive pricing tier is taken. Custom metrics are tiered across four tiers; the projection uses the first and dearest, so volume discounts are not reflected.

A short-run figure is also not stable to the dollar. How many distinct series a one-minute window touches depends on where that minute falls on the diurnal curve, so the same command run at different times of day projects anywhere from $137.70 to $140.70 on this estate. Treat a short-run projection as an order of magnitude, not a number.

Fix. Nothing. Over-estimating is the safe direction for something that spends your money without asking twice. Real cost for a one-minute run is pennies.

Note These figures are for the 18-service estate. A larger estate has more series, so every figure rises: a 22-service estate touches 509 series over 14 days and projects $165.20, with its one-minute runs landing between $149.70 and $152.70. The shape of the problem is identical.

The projection says so itself, in the notes:

Warnings
  ! custom metrics per metric-month: the published rate is tiered across 4 tiers; the
    first and most expensive tier was taken, so volume discounts are not reflected and
    this is an over-estimate.
  ! Cost is list price only and never a quote: it cannot see negotiated discounts,
    credits or private pricing agreements. Custom metric series are billed per
    metric-month and this figure charges a full month for every new series rather than
    prorating, so a short demo looks dearer than it bills.

In the GUI those are behind the projection notes drawer in the footer. Expand it before quoting the number to anyone.

Two consequences worth internalizing:

• Do not size a budget ceiling off a short-run projection. budget-ceiling refuses when the ledger total plus this run's estimate exceeds the cap, so the default --cap 50 refuses even a one-minute run of the 18-service estate — its projection is around $140. The refusal says so and names the three ways out: raise the cap deliberately, shorten the window, or run under paranoid to render without sending.
• The series term is fixed, so the marginal cost of more data is small. On the 18-service estate, going from one day to fourteen takes the projection from $141.54 to $152.53 — about $11 for fourteen times the data, because the series count does not change.

See the cost guard for how the projection is built.

A pricing-cache warning names a path under ~/.cwomni/#

Symptom. A projection carries a warning naming a pricing cache under ~/.cwomni/ rather than ~/.proteus/ — for example:

the pricing cache at /Users/you/.cwomni/pricing-us-west-2.json used an old schema and
was ignored

Cause. ~/.cwomni/ is the pre-rename state directory. The project, package and CLI are all proteus now and the state directory is ~/.proteus/. The warning names whatever path it actually read, so seeing the old path means PROTEUS_HOME still points there — or a cache file from a pre-rename build is still being reached.

The cache is versioned, so a file in the old format is discarded rather than misread. Nothing is wrong with your rates; the cache was not used, and the projection fell back to the built-in table.

Fix. Refresh the rates. In the GUI that is refresh rates in the footer next to the cache provenance line. Otherwise the next projection re-fetches by itself once the stale file is out of the way:

unset PROTEUS_HOME     # if it was pointing at the old directory
proteus doctor

Confirm the fix. The cost meter's provenance line should read cache <region> <n> ago, and the projection notes should include 7 of 7 rates came from the live Pricing API.

Warning Confirm it from the provenance line, never from the dollar figure. A failed live lookup falls back to a rate table that produces a figure close to the live one — a 14-day seed of the 18-service estate projects about $153 either way — so there is no dollar amount that tells you which rates you got. The only signal is the source on the provenance line and the notes beside it.

Note Rates are cached per region for 24 hours. A cache from the future — clock skew, or a hand-edited file — also counts as stale, since its age cannot be trusted.

doctor says credentials FAIL#

Symptom.

 credentials │ FAIL │ could not resolve credentials: TokenRetrievalError: Error when
             │      │ retrieving token from sso: Token has expired and refresh failed

Cause. Whatever provides your credentials has expired or cannot be resolved. This is the single most common reason a demo fails five minutes before it starts.

Fix. The check prints it:

aws sso login

Or refresh whatever else provides the profile. Then:

proteus doctor --profile my-profile

Confirm the fix. The row becomes OK and names the account and role:

 credentials │ OK │ account <your-account-id> as <your-role-session>

And the six checks that were skipped now run.

Every alarm stays green through an injected fault#

Symptom. bootstrap created the alarms, seed or live reported success, the service map and traces look right — and every alarm is still OK. Raising the fault severity changes nothing.

Cause. Most often the alarms have no data at all. They report OK rather than INSUFFICIENT_DATA because they set TreatMissingData: notBreaching, and that is what makes this hard to spot: an alarm on a metric that never arrives is indistinguishable from a healthy estate. Green is not evidence that anything is being measured.

This happened once for a concrete reason worth knowing if you are running an older deployed stack. The alarm metric names were global — one triple applied to every service — and the renderer emitted none of them, so every alarm evaluated a metric that did not exist. Names are now derived per archetype, and tests/test_cfn_alarm_metrics.py asserts that every name an alarm uses is one the renderer emits.

Note what that test does not prove. It establishes that the names agree; it cannot establish that the alarms fire, because nobody on this project has an account to check against. Whether the metrics arrive in the expected namespace with a matching service.name dimension, and whether ExtendedStatistic: p99 resolves against a duration metric, are all unverified. See the deep dive.

Confirm. Offline and free. Read the metric names out of the template your own build produces, rather than trusting any name written on this page:

proteus bootstrap --print | grep -o '"MetricName": "[^"]*"' | sort -u

Then run the suite — if tests/test_cfn_alarm_metrics.py passes, the names agree with what the renderer emits and the cause is elsewhere. Check in this order: that seed or live actually sent (--dry-run sends nothing), that you are looking at the region you injected into, and that enough time has passed for datapoints to land.

Also check whether the service you are watching has an alarm at all. Not every service gets one — an archetype exposing no sensible request-and-error pair, such as a queue consumer or a database, gets none, and a service with no alarm cannot go red. That is deliberate: an alarm nothing can trip is worse than no alarm, because it looks like coverage.

Fix. If the metric names disagree with the emitted set, that is a defect in proteus/cfn/synth.py rather than a misconfiguration, and no flag reconciles them. Either way, do not build a demo on an alarm firing, or on AWS DevOps Agent being handed an alert, until you have watched one reach ALARM on your own estate. The metrics, logs, traces, RUM and the service map are unaffected and are what a demo should rest on.

A latency histogram has no detail above ten seconds#

Symptom. A slow workload's latency looks truncated. Percentiles above ten seconds read as ten seconds, or a chart flattens at the top, and the shape of the tail is missing.

Cause. A known defect, not yet fixed. proteus/catalog/generic.py gives every latency metric the same bucket ladder — boundaries suited to request latency, topping out at 10,000 ms — regardless of the metric's own baseline. A workload whose normal duration is several seconds therefore puts a large part of its distribution in the overflow bucket, where it has no resolution at all.

ml-pipeline is the clearest case. Sampled over eight 20-minute windows spread across a day, at a 300-second interval — the figures move with the diurnal traffic curve, so they are ranges rather than constants:

Confirm. Offline and free. Render the preset and read the last bucket against the total:

import datetime as dt
from proteus.model import presets
from proteus.render import engine
from proteus.render.timeline import Window
from proteus.render.records import MetricPoint

end = dt.datetime.now(dt.timezone.utc).replace(microsecond=0)
window = Window(start=end - dt.timedelta(minutes=20), end=end, interval_s=300.0)
for record in engine.render(presets.build("ml-pipeline", "us-west-2"), window):
    if isinstance(record, MetricPoint) and record.explicit_bounds:
        overflow = record.bucket_counts[-1] / sum(record.bucket_counts)
        print(record.name, f"{overflow:.1%}", max(record.explicit_bounds))

Your numbers will not match the table exactly. The window is anchored to now and the traffic curve is diurnal, so run it twice an hour apart and the shares move by several points. What does not move is the top bucket boundary: 10,000 ms for every latency metric in the estate, whatever its baseline.

Fix. None available to you; the bucket boundaries are not configurable. The bulk of each distribution is still inside the ladder, so the mean and the lower percentiles are sound — it is the upper tail that is unresolvable, and any percentile above ten seconds that you read from these two metrics is not measuring anything.

This is separate from the alarms. No alarm threshold sits above its metric's widest bucket any more, so an alarm cannot be silently unreportable for this reason — see the alarm section.

Bedrock is unavailable#

Symptom. chat or hunt fails, or doctor reports bedrock WARN.

Cause. No model access in the region, or the configured model id is not available to your account.

Fix. Request model access in the Bedrock console for the region you are passing, or point at a different model:

export PROTEUS_MODEL=<model-or-inference-profile-id>

Confirm the fix. doctor's bedrock check reports OK with an explanation of what it found. Note this is a WARN, not a FAIL: seed, live, firehose, show, bootstrap and teardown all work without Bedrock. Only chat and hunt need it.

The estate has moved ahead of the stack#

Symptom. Some services have no log group or no alarm, and doctor reports:

 model vs stack │ WARN │ world model fingerprint <a> does not match the deployed <b> -
                │      │ the estate has changed since bootstrap, so some services may
                │      │ have no log group or alarm

Cause. You added services — through chat, or on the GUI canvas — after the stack was deployed. The stack records the fingerprint of the model it was built from; that no longer matches.

This is the same failure as a missing log group, caught one step earlier.

Fix.

proteus bootstrap

Confirm the fix. doctor reports fingerprint <a> matches stack, and the log groups check reports that all of them exist.

A hunt is leaking its answer#

Symptom. proteus show names the hidden fault's kind, target or window while a hunt is open.

Cause. This was a real bug: the first live hunt printed its own answer through show, which defeats the mode entirely, since the presenter need only run one command. Redacting the kind column alone was not enough either — the fault id embedded the kind.

Fix. Nothing on your side. show now filters hidden faults out of the table completely and prints only a count. If you are seeing the answer, you are on an older build.

Confirm. After proteus hunt start, run proteus show. Correct output is:

1 hidden fault(s) active from an open hunt - details withheld. Use
proteus hunt reveal <id> to see them.

If show names it, do not run the exercise. See fault hunting for the other three places the answer is withheld.

serve will not start#

Symptom.

could not start on port 8787: [Errno 48] Address already in use
try proteus serve --port 8788

Cause. Another process — often an earlier proteus serve you forgot to stop — holds the port.

Fix. Take the suggestion, or stop the other server. The error names the next port for you.

Warning Do not try to make the server reachable from the network. It binds 127.0.0.1 only and rejects foreign Origin headers by design, because the process holds your AWS credentials and can spend money.

Only one job runs at a time#

Symptom. In the GUI, pressing inject or dry run offers to attach to a running job instead of starting a new one.

Cause. The server holds a single job slot. This is intentional: two concurrent injectors would race on the spend ledger and make the throttling ceilings meaningless.

Fix. Attach and watch it finish, or cancel it. A running dry run is not a malfunctioning inject button.