Troubleshooting¶

Start from the refusal, then read the machine's timeline. The commands use the walkthrough's setup.

Reading a refusal¶

Every refusal is application/problem+json. Its type is urn:bronzeward:problem:<code>, and detail, when present, says which check refused. A 401 and an identity-revoked carry no detail: go by the code. The response carries no deeper cause: a node or provider failure behind it is recorded in the operation's events, a use-time-check timeline entry or an observation's unread. The response headers are in $H.

call POST /plans "$publisher" "$body" troubleshoot-plan-1 | jq '{type, status, detail}'

Code	Status	Usual cause	What to do
`idempotency-key-required`	428	A write without `Idempotency-Key`	Send one, unique per intended act
`invalid-request`	400	A malformed body or header, such as an `Idempotency-Key` outside 16 to 128 characters of `[A-Za-z0-9_-]`; `detail` names it	Fix the request
`idempotency-key-reused`	422	The same key with a different request	Use a new key; the same key replays only the same request
`precondition-required`	428	A draft change without `If-Match`	Send the draft's current ETag
`precondition-failed`	412	The draft moved since you read its ETag	Read the draft again, reapply your change
`stale-input`	409	A publication input moved since it was read	Publish again
`scope-busy`	409	An operation holds the machine's scope	Wait for it, or settle it (recovery)
`recovery-mode-active`	409	Recovery mode, or a scope not yet released	Finish the restore
`conflict`	409	A state the act does not apply to; `detail` names it	Read the object, then act on its current state
`identity-revoked`	403	The subject or service identity is revoked, or in `auth.deniedSubjects`	Revocation is permanent and a new token does not lift it: act as another human subject, or issue a new service identity (`bronzeward token issue`) under a name `auth.deniedSubjects` does not list
`forbidden`	403	The token lacks the role, or a human-only act from automation	Check `auth.roles` and the groups claim
`machine-identity-mismatch`	409	The node's SMBIOS UUID, node ID or cluster ID is not the recorded one	Check the endpoint points at the right machine
`talos-access-unavailable`	503	No talosconfig in the provider, access denied, or the node unreachable	Check OpenBao, the access path and the network
`dependency-unavailable`	503	PostgreSQL, the identity provider or OpenBao is unreachable or not configured	Fix the dependency, then retry
`transient-conflict`	503	Concurrent writes kept colliding	Retry with the same key
`epoch-superseded`	503	This process started before the current recovery epoch	Restart it
`internal-error`	500	Unknown; the write may or may not have happened	Retry with the same key, which returns the first answer if it was recorded

Reading the timeline¶

Each machine has one append-only timeline: every plan, approval, observation, commitment, attempt, response, accounting, drift, freeze and ownership transition, in revision order.

call GET "/machines/$machine_w/timeline?limit=500" "$viewer" | jq -c '.items[] | [.revision, .at, .kind, .entry]'

Pages hold 50 entries by default and at most 500, oldest first, so recent entries are on the last page: follow next with &cursor=<next>, or use the items helper. Other lists, such as observations, page the same way but are in id order, not time order. Timeline and operation-event responses carry no configuration text or secrets. Other reads can: source reads (fragments, profiles, assignments) carry sanitized documents, and an ingestion's review carries the staged document, which can hold a value nobody marked. That review is no-store and answers any human with the author role, not only the ingestion's creator.

From it, for any operation, you can answer:

Who approved what: the plan and approval entries, and any approval-revocation.
What each attempt carried, and when: attempt entries, each with its number, route, controller and transport and verification deadlines.
How each attempt is accounted for: a response, an accounting decision, or neither (still unknown). An attempt is recorded before the send, so the timeline cannot always say whether the request left; an accounting records what was decided from observations, not that it was sent.
What was seen on the node: observation-started and observation entries, each naming its purpose.
Why the scope was held: commitment, operation-state and scope-state entries.
Who decided an accounting, and on what: a decided accounting entry names who decided it, the start and end observations and the facts.
How it ended: the last operation-state entry.
Who owned it: ownership-transition entries, each with its generation.

To follow one operation:

call GET /operations/$op "$viewer" | jq .
call GET /operations/$op/events "$viewer" | jq -c '.items[]'
curl --disable -sS -N -H @- -H 'Accept: text/event-stream' "$A/api/v1/operations/$op/events" \
  <<<"Authorization: Bearer $viewer"

The stream ends when the operation ends, or when the token expires. Resume with a fresh token and Last-Event-ID set to the last event's id. An event: reset means the installation entered a new recovery epoch: read the events again from the start.

Common situations¶

An operation stays unresolved. Find out why before taking it over: read its attempts and timeline. An unaccounted attempt, a verification deadline reached without a decisive observation, a closed scope gate or a revoked approval each leave it there with a healthy owner, and a takeover only advances the generation. Account, observe or cancel for the cause you find; take it over only when its owner stopped. See interrupted operations.
A plan is approved but nothing happens. Look on the timeline for a refusal entry, which names the transaction and the comparison that failed, or a use-time-check entry, which gives each dependency's result.
A machine is frozen. Its last freeze entry says who froze it, in which role and why. An approver unfreezes it with POST /machines/$machine/unfreezes and a body {"reason": "..."}.
Every call answers 409 recovery-mode-active. The instance was started with -recovery-start and no entry is recorded. Until the entry commits it serves only /livez and POST /recovery/entries, GET /recovery included; the detail says so.
Acts on one machine answer 409 recovery-mode-active. The installation is in recovery mode and that machine's scope is not released. On any instance past the entry: call GET /recovery "$viewer" | jq '{recoveryMode, scopes}'.
serve exits at start. An unmigrated or differently migrated database: stop every server, run migrate (migrate, then serve).