Key loss and restoration: applying a retained artifact versus regenerating it¶
| Date | 24 September 2026 |
| Work item | Experiment E5 - test key loss and restoration |
| Design reference | §7.5 Rotation and retention contract, §14.4 Recovery dependencies, §14.6 Explicit recovery mode after restoration, §18.1 Phase 0; execution and recovery contract §7 |
| Artifacts | experiments/e5-key-loss-restoration/, evidence under experiments/e5-key-loss-restoration/evidence/ |
| Decision enabled | What each backup family must hold, and in what order and age relative to the others, for a release to stay applicable or regenerable after a restore; what the recovery check must verify and how it stops. Input to retention and recovery policy, execution and recovery contracts and deployment profile selection. No policy, window or provider is selected here. |
1. Question¶
The design keeps two recovery paths apart. Applying a retained release uses its exact encrypted artifact; regenerating compiles a new release from source versions (design:287, design:404). "Losing a source secret version can block regeneration while a retained encrypted artifact remains applicable. Losing the only usable artifact decryption key blocks application of that artifact. Preserving ciphertext alone does not preserve executability" (design:404). The recovery set includes releases, secret versions, artifact keys, credentials and provider unlock material, and "backups need not have identical timestamps, but every dependency required by the selected recovery action must be available" (design:866). After a restore the operator enters recovery mode, verifies decryption and credentials under the actual recovery identities, and marks each scope ready, blocked or unresolved (design:876-880; spec §7 steps 3 and 6, spec:448 and spec:454).
The issue asks for three things:
- restoration cases for older and newer application and provider backups, missing keys and missing secret versions;
- applying a stored encrypted artifact kept apart from regenerating the configuration, with ciphertext presence alone not counted as executability;
- the recoverable, blocked and irrecoverable outcomes, the custody and unlock prerequisites, and the safe stopping behaviour.
2. What was built¶
2.1 Three backup families, three generations¶
The investigation fixtures snapshot three families independently and restore them in any combination. This experiment gives each a role:
| Family | Snapshot, restore | Holds here |
|---|---|---|
| management database (PostgreSQL) | db-snapshot, db-restore |
table kl_release: id, generation, source path and version, key name, stored ciphertext, artifact SHA-256 |
| provider (OpenBao, integrated Raft) | bao-snapshot, bao-restore |
KV v2 source versions under secret/kl/, Transit keys kl-*, policies, and every token it has issued |
local credential store (.state/data) |
store-snapshot, store-restore |
the compiler's and the executor's tokens (0600), with their accessors |
run/scenario builds three generations:
| Generation | Built | Snapshotted |
|---|---|---|
| g1 | KV kl/app v1; key kl-art v1; release R1 = kl/app@1 under kl-art v1; compiler and executor1 issued |
all three families (row 001) |
| g2 | KV kl/app v2; kl-art rotated to v2; R2 = kl/app@2 under v2; R1 rewrapped to v2 in the database (002); decryption floor raised to 2 (003); executor2 issued (004), executor1 revoked (005, shown effective in L, 102) |
all three families (006) |
| g3 | KV kl/late v1; key kl-late; R3 = kl/late@1 under kl-late v1 |
never |
Row 007 is the release table at the end of g3: R1 and R2 under kl-art v2, R3 under kl-late v1.
The injection log (evidence/injections.log) fixes which snapshot each restore used.
2.2 Identities and the three actions¶
| Identity | Policy |
|---|---|
| compiler | read secret/data/kl/*; update on transit/encrypt/kl-* |
| executor | update on transit/decrypt/kl-* |
| admin | the root token, standing for the operator's administrator |
| operator | runs the fixture's injections and the recovery check |
run/release takes a release id and reads everything else from the restored state: the record
from the database, the credential from the store, the key and source from the provider.
- apply (executor): decrypt the stored ciphertext and compare the plaintext's digest with the recorded one. This is application eligibility; nothing is sent to a machine.
- regen (compiler): read the recorded source version, render the artifact, compare its digest, and encrypt it under the release's key. Nothing is stored: a regenerated release is a new release that needs its own approval.
- check (operator): run both, then stop. It prints
stopped: nothing dispatched, no release stored, confirms that the release table's digest is unchanged, and ends with one class:applicable(the stored artifact decrypts to its digest),regenerable(it does not, and the source does regenerate it),blocked(neither), orabsent(the restored database has no such release). The check is not read-only at the provider. Its regeneration step sends a real Transit encrypt, and Transit turns an encrypt under a missing key into a key creation for an identity that holdscreate(§3.2, case D). The scenario therefore reads the provider's Transit key state as the administrator before and after every check, and records a change as the classprovider-changed. The comparison bracketscheckrows only (run/scenario:111-124, called fromjudge_release, 130-134). A standaloneregenrow is not bracketed. Row 037, for one, sends a Transit encrypt to the deletedkl-artoutside any check. A key created there would show only as part of the next check's "before" state. None did: the checks either side of 037 (035, 038) both count one key.
The artifact is a one-field YAML document, machine:\n token: <value>\n, rendered
deterministically so that a regeneration from the same source version reproduces its digest.
2.3 Rows¶
Every observation is one row of evidence/verdicts.tsv: the case, release, action and identity,
the state bundle that is its ground truth, the expected outcome written into the script before the
run, and the outcome observed from the command's exit status and output alone. For apply and
regen the observation is ok, denied (the CLI's Code: 403, matched case-sensitively so that
a local "Permission denied" on a file is not taken for the provider's) or fail; for check it is
the class printed. A mismatch is recorded and the run continues. fail is matched on the exit
status alone, so a row expecting fail would also match a failure for another reason; the reason
each such row failed is quoted in §3.2 from its transcript.
The state column names the last bundle taken, not always the state the row ran in. Rows 040-043,
065-067, 084-086 and 110-112 name a bundle taken before an in-case change: the administrator's
upsert of kl-art (039), the floor changes (064, 066) and the executor reissues (083, 109). The
bundle does not show that change; the change's own row and its transcript do.
2.4 Pinned versions¶
From fixtures/versions.env: OpenBao ghcr.io/openbao/openbao:2.6.1@sha256:5b2486ab…, PostgreSQL
docker.io/library/postgres:17-alpine@sha256:f02121de…, one-share unseal, single-node Raft. Talos
v1.13.6 runs as part of the fixture and is not used here.
2.5 Synthetic values and the leak scan¶
The three source values are BWSYNTH-kl-<name>-<24 hex>, generated per run under KL_OUT, outside
the checkout and outside every backup family, and handed to the CLIs over stdin. A rendered
artifact exists only in a temporary file for the one call that needs it. Every decryption is
reduced to a digest verdict. Tokens pass through the environment; only accessors are printed.
The fixture's leak scan (fixtures/lib.sh:923, patterns including BWSYNTH-) ran on each of the
14 bundles, over the bundle, .state/backups and .state/data. Each scan found its positive
control three times (the live store and the two store archives the bundle expands) and nothing
else: 42 control hits, 0 other hits (evidence/leak-scans.txt, evidence/summary.txt).
2.6 Reproduction¶
fixtures/bin/up
KL_OUT=<empty directory on disk> experiments/e5-key-loss-restoration/run/all
KL_OUT=<the same directory> experiments/e5-key-loss-restoration/run/collect-evidence
fixtures/bin/down
The committed capture: 112 rows, 0 mismatches, 14 state bundles.
The fixture records the commit it was brought up at in each bundle's versions.txt: created at
5b29fed, in all 14. The capture ran on 24 September 2026. The fixture wrote its own secret
fixture/inline at 22:02 CEST (20:02:37Z), the scenario wrote its first source version at
20:04:23Z, and the final bundle was taken at 20:07:57Z (openbao-metadata.jsonl and
versions.txt of each bundle). It ran from a working tree whose run/ scripts were last modified
at 22:01:47 CEST, before the capture began. Those scripts are committed unchanged in 3c715e6,
together with the evidence. The evidence itself does not record the scripts' commit or their
hashes, so the tie between the capture and the committed scripts rests on those times and that
commit, not on anything the capture recorded. A future capture should record git rev-parse HEAD
and the hashes of run/.
3. Results¶
3.1 Case table¶
"Live" is the state of each family at that point; gN names the snapshot a family was restored
from. A live family carries every earlier case's loss until a restore undoes it: in D, kl/app@1
is still destroyed from B and kl-late still deleted from C.
| Case | Database | Provider | Store | Release | apply | regen | check | Outcome | Rows |
|---|---|---|---|---|---|---|---|---|---|
| A nothing lost | live | live | live | R1, R2, R3 | ok | ok | applicable | recoverable | 009-017 |
B source version destroyed (kl/app@1) |
live | live | live | R1 | ok | fail | applicable | recoverable by apply; regeneration blocked | 020-022 |
| R2 | ok | ok | applicable | recoverable | 023-025 | ||||
C key and source lost after the last snapshot (kl-late, kl/late@1) |
live | live | live | R3 | fail | fail | blocked | irrecoverable: no snapshot holds either (E) | 028-030 |
D artifact key deleted (kl-art) |
live | live | live | R1 | fail | fail | blocked | blocked; recovered by E | 033-035 |
| R2 | fail | denied | blocked | blocked; recovered by E | 036-038 | ||||
| E all restored | g2 | g2 | g2 | R1, R2 | ok | ok | applicable | recoverable | 052-057 |
| R3 | — | — | absent | irrecoverable | 058 | ||||
| G database older than provider | g1 | g2 | g2 | R1 | fail (below floor) | ok | regenerable | recoverable: regeneration, or an administrator lowers the floor | 060-067 |
| R2 | — | — | absent | no record to apply | 063 | ||||
| H provider older than database | g2 | g1 | g1 | R1 | fail (key version too new) | ok | regenerable | recoverable by regeneration | 072-074 |
| R2 | fail | fail | blocked | blocked in this combination | 075-077 | ||||
| I store newer than provider | g1 | g1 | g2 | R1 | denied | ok | regenerable | recoverable: an administrator reissues the executor | 080-086 |
| J credentials lost | g1 | g1 | none, then g1 | R1 | fail, then ok | fail, then ok | blocked, then applicable | recoverable from the store backup that matches the provider | 088-094 |
| K provider sealed | g1 | g1 (sealed) | g1 | R1 | — | — | blocked, then applicable | recoverable with the unseal key share | 096-098 |
| L store older than provider | g2 | g2 | g1 | R1, R2 | denied | ok | regenerable | recoverable: an administrator reissues the executor | 102-108 |
| R2 after reissue | ok | ok | applicable | recoverable | 110-112 |
3.2 What each case showed¶
B — a destroyed source version. The admin read shows kl/app version 1 destroyed: true
(019). R1's stored artifact still decrypts to its digest (020), while its regeneration fails: "No
data found at secret/data/kl/app" (021). The check classes R1 applicable (022). This is
design:404 observed: the artifact stays applicable while regeneration is blocked. A later provider
restore brought the version back (048), and R1 regenerated again (053).
C and E — irrecoverable, and why. After kl-late was deleted and kl/late@1 destroyed, R3
neither applies ("encryption key not found", 028) nor regenerates (029). Restoring all three
families to g2 brings back what B destroyed and what D deleted: kl/app@1 undestroyed (048) and
kl-art at latest version 2 with its floor (047); version 1, below the floor there, is shown
decryptable again in G (064, 065). But kl-late and kl/late are absent from that snapshot (049,
050), and so is R3's record (051, 058). R3 is irrecoverable because no snapshot holds its key, its
source or its record, not because they were deleted: the same restore undid a destruction and a key
deletion.
D — a deleted artifact key, and a key recreated under its name. With kl-art deleted (032),
no stored artifact applies (033, 036). R1's regeneration fails for a reason carried over from B,
not from the deletion: kl/app@1 is still destroyed ("No data found at secret/data/kl/app", 034). R2's source is still readable and renders to its digest, but
encryption is refused with HTTP 403 (037). Transit's encrypt endpoint creates a missing key, which
needs the create capability, and the compiler holds update only. Rows 039-043 show what that
refusal prevents. The administrator, who holds create, encrypted a probe under the missing name
(039). That minted a new kl-art whose only version is 1 (040). Against that key, R2's stored
artifact still fails ("invalid ciphertext: version is too new", 041). R2's regeneration now goes
through and encrypts under "kl-art key version 1" (042), a label the lost key's own version 1 also
carried. The check classes R2 regenerable (043). Nothing in the ciphertext's vault:v1: prefix tells
the two keys apart.
G — the database older than the provider. The g1 database holds R1 as it was before the
rewrap, under key version 1, and the g2 provider enforces a decryption floor of 2. Apply fails:
"ciphertext or signature version is disallowed by policy (too old)" (060). Regeneration from
kl/app@1 succeeds (061), so the check says regenerable (062). R2 was recorded after the g1
database snapshot, so the restored database has no record of it (063), although the provider still
holds its source and key. The administrator lowered the floor to 1 (064). The same artifact then
applied (065), and failed again once the floor was put back (066, 067).
H — the provider older than the database. The g2 database holds R1 rewrapped and R2 encrypted
under kl-art version 2. The g1 provider has only version 1 (071). Both applications fail with
"invalid ciphertext: version is too new" (072, 075). R1 regenerates from kl/app@1 (073), but R2
cannot: the compiler's read of kl/app version 2 answers "No value found at secret/data/kl/app"
(076). Case H has no kv_truth row; the H bundle's provider metadata lists version 1 of kl/app
only. The rewrap in g2 is what cost R1 its applicability here. Before it, R1's artifact was under
version 1, which this provider holds.
I — credentials newer than the provider. The g2 store holds executor2, which the g1 provider never issued: apply is denied with HTTP 403 (080). The compiler token dates from g1 and still works, so R1 regenerates (081, 082). The administrator issued a new executor token on the restored provider (083), after which R1 applies (084, 086). The artifact and its key were intact; only the credential was missing.
J — credentials lost. With the credential store removed, neither path opens: "no executor credential in the restored store" (088), "no compiler credential" (089). Restoring the g1 store, the one matching the g1 provider, restores both (092-094).
K — a sealed provider. With every dependency retained and every credential valid, the check is blocked: both paths got HTTP 503 "Vault is sealed" (096). After unsealing with the fixture's key share (097), R1 is applicable again (098).
L — credentials older than the provider. The g1 store holds executor1, which the g2 provider revoked in setup (005). The provider has no token for its accessor ("invalid accessor", 102), and apply is denied with HTTP 403 for both releases (103, 106), although key, artifact and source are all present. The compiler token dates from g1, was never revoked and still works, so both regenerate (104, 107). After the administrator issued a new executor token (109), R2 applies again (110, 112). This is I in the other direction: a store backup that does not match the provider loses the credential whichever of the two is older.
3.3 Custody and unlock prerequisites per case¶
| Case | What was missing | Where it is kept | What made the release usable again |
|---|---|---|---|
| B | source kl/app@1 |
provider KV; provider backup | a provider restore (E, 048); apply was never affected (020) |
| C | key kl-late, source kl/late@1, R3's record |
provider and database; no backup holds them | nothing |
| D | key kl-art |
provider Transit; provider backup | a provider restore (E, 047) |
| G | key version 1 above the floor; R2's record | the provider's key configuration; database backup | an administrator floor change (064); R2 needs a database backup no older than R2 |
| H | key version 2; kl/app@2 |
provider backup no older than the database backup | a newer provider backup; R1 regenerates meanwhile (073) |
| I | an executor credential this provider issued | credential store backup matching the provider | an administrator reissue (083) |
| J | every credential | credential store backup | the store restore matching the provider (091) |
| K | an unsealed provider | the unseal key share, held apart from every backup family | unsealing (097) |
| L | an executor credential this provider has not revoked | credential store backup matching the provider | an administrator reissue (109) |
4. Failures hit while building it¶
4.1 The expected outcome of regenerating against a deleted key¶
The first full capture expected row 037 to fail and observed denied: it was the one mismatch in
93 rows. The script was right and the expectation was wrong. Transit's encrypt treats a missing key
as an upsert, and the compiler's policy lacks create, so the provider refused the request with a
403 rather than reporting a missing key. That capture was discarded and is not committed; the
mismatch count comes from its run, not from evidence in this repository. Later captures expect
denied and add the administrator's upsert and its consequences (039-043) as the control for what
the refusal prevents.
4.2 The check was described as read-only¶
A review of the second capture found that the report, run/release and the README called the
check read-only, while its regeneration step sends a real encrypt (row 038 sends one to the deleted
kl-art). The third, committed capture compares the provider's key state across every check,
adds case L (a store older than the provider, which also shows the executor1 revocation took
effect) and matches denied on the CLI's Code: 403 only. Rows 001-098 have the same case,
release, action, identity, state, expectation, observation and verdict in both captures. The
second capture is not part of the current evidence: it was committed in 5b29fed and replaced by
this one in 3c715e6, so the comparison rests on that earlier commit's verdicts.tsv, not on
anything under evidence/ now.
4.3 Verbatim transcripts fail git diff --check¶
The indented error text inside the check transcripts carries trailing whitespace. .gitattributes
exempts this experiment's evidence/transcripts/*.txt, as it already does for the
provider-capability run's.
5. What this decides¶
5.1 Criterion 1: restoration cases¶
Each combination the issue's run design names was restored, and both paths were tried after it: application older than provider, provider older than application, a destroyed secret version, a deleted Transit key and missing management credentials. Of the eight database/provider/store age combinations, six were set up: g2/g2/g2 (E), g1/g2/g2 (G), g2/g1/g1 (H), g1/g1/g2 (I), g1/g1/g1 (J after the store restore, K) and g2/g2/g1 (L). The other two were not (§6).
| Asked for | Case |
|---|---|
| application backup older than provider | G (database g1, provider g2) |
| application backup newer than provider | H (database g2, provider g1) |
| provider backup older | H |
| provider backup newer | G |
| missing key | C (never snapshotted), D (deleted, restorable) |
| missing secret version | B (destroyed, restorable), C (destroyed, never snapshotted) |
| missing management credentials | I (store newer than the provider), L (store older), J (credentials gone) |
| explicit restoration | E (all three families to g2), J (store alone) |
5.2 Criterion 2: applying is not regenerating, and ciphertext is not executability¶
The two paths failed independently in both directions:
- Artifact applies, regeneration blocked: B, where the source version was destroyed (020, 021).
- Artifact does not apply, regeneration works: G (060, 061), H for R1 (072, 073), I (080, 081), L (103, 104), and D after the key was recreated (041, 042).
- Both blocked: C, D, H for R2, J and K.
In every "does not apply" row the stored ciphertext was present: each transcript's first line names its key version. The release then applied again once the missing piece came back: the key version (E, 052), the floor (065) or the credential (084, 110). That it was the same ciphertext is by construction, not observed: the scenario writes a release's ciphertext only when it makes the release and in the rewrap (002), and the evidence records its key-version prefix (007, 051), not its bytes. What made it executable was the key version, the provider's floor, the executor's credential and an unsealed provider. The ciphertext alone never did.
5.3 Criterion 3: outcomes, prerequisites and stopping¶
Outcomes. In this run, recoverable meant that a restore, an administrator action or a regeneration made the release usable again. Blocked meant a missing piece that another backup or action could supply. Irrecoverable meant that no snapshot held the piece:
- recoverable by apply: A, B (R1), E (R1, R2), I and L after reissue, J after the store restore, K after unsealing;
- recoverable by regeneration: H (R1), and I and L before the executor was reissued; each makes a new release that needs its own approval;
- recoverable by regeneration, or by apply once an administrator lowers the floor: G (R1, 064, 065);
- blocked, then recovered by a restore: D (by E);
- blocked in this combination: H (R2), which needs a provider backup at least as new as g2;
- irrecoverable: R3 (C, E). Its key, its source version and its record were all made after the last snapshot.
Custody and unlock prerequisites observed, by path (per case: §3.3):
| Path | Needs |
|---|---|
| apply | the release record (database); the key version it names, present and at or above the decryption floor (provider); an executor credential the restored provider recognises (store matching provider); an unsealed provider (the unseal share) |
| regen | the source version (provider KV); a key to encrypt under (provider); a compiler credential the restored provider recognises; an unsealed provider; then a new approval |
| repair actions seen | lowering a decryption floor (064), reissuing a credential (083, 109), creating a key (039): all administrator operations on the restored provider |
Safe stopping. The check stores no release and dispatches nothing. Each of the 25 check rows
prints stopped: nothing dispatched, no release stored, and for every release present the release
table's digest was the same before and after the check (for example 030 and 043). The check is not
read-only at the provider, though. Its regeneration step sends a real Transit encrypt, and Transit
creates a missing key on encrypt for an identity that holds create (039). So the scenario read
the provider's Transit key state, as the administrator, before and after every check. It was
unchanged in the 24 checks where the provider could answer, and not compared in the one where the
provider was sealed (096). For the irrecoverable R3 the check reports absent (058) or blocked
(030) and stops. It creates no new release and no substitute record. The one way this run found to
"recover" past a lost key without a backup was the recreated key in D (042). There a regeneration
succeeds under a different key that carries the same name and version number. The recovery check
must not do that on its own. The compiler's lack of create is what kept D's checks (035, 038)
from doing it (037); a recovery identity that held create would have minted the key.
5.4 Alternative recovery paths compared¶
| Path | Works when | Costs | Seen in |
|---|---|---|---|
| apply the retained artifact | the record exists, its key version is at or above the floor, the executor credential is one the provider honours, and the provider is unsealed | nothing beyond the original approval; the bytes approved are the bytes applied (design:287) | A, B, E, J, K, and I and L after reissue |
| regenerate from source as a new release | the source version, a key to encrypt under and a compiler credential | a new release and a new approval; under another renderer the bytes could differ (not exercised: the renderer is fixed here) | G, H (R1), I, L |
| restore another backup generation | a snapshot holds the missing piece | everything after that snapshot in the restored family is lost (R3 in E), and the pairing with the other families can break (G, H, I, L) | E, which undid B's and D's losses |
| administrator repair on the restored provider | lowering the floor works while the key version still exists below it; reissuing a credential works while the policy exists | an administrator action outside the check, to be recorded (and, for the floor, reverted) | G (064-066), I (083), L (109) |
| recreate a lost key under its name | for apply, never: the new key decrypts nothing stored (041). A regeneration does succeed under it (042) | the regenerated ciphertext reuses the lost key's name and its vault:v1: label, so a record of key name and version cannot tell the new key from the lost one |
D (039-043) |
Inference: apply loses least. Next come a restore or repair that makes apply possible again, then
regeneration with re-approval. Recreating a key under the lost key's name, reusing its name and
vault:vN labels, is not a recovery path. Regenerating under a new key with a new name, as a new
release, was not exercised as a recovery path.
6. Limits¶
- One provider and one topology. OpenBao 2.6.1, single node, one unseal share. Every provider restore went back onto the same cluster, which keeps its unseal key. Restoring onto a new cluster, where unseal or recovery keys come from another custody path, was not exercised.
- The administrator ran as root for every change it made (002-005, 039, 064, 066, 083, 095, 109) and every read of the ground truth. A least-privilege administrator policy for floor changes, credential reissue and key creation is still unwritten.
- Apply is eligibility, not application. Nothing was sent to a Talos machine. The artifact is a one-field synthetic document, not a rendered machine configuration.
- The release table is a stand-in. It has no approvals, operation journal or recovery epoch. The check models spec §7 steps 2, 3 and 6 for one release. It does not model recovery-mode entry, executor quiescence or approval invalidation.
- Token expiry was not exercised. Tokens used the provider's default TTL, well beyond the run. A restored provider whose tokens have expired, or a lease revoked after a snapshot, was not tested.
- One pass per case, in sequence, on one fixture. Each case's ground truth is its bundle, and the injection log fixes the order. Case G's floor change was reverted within the case (066).
- Two age combinations were not set up. Database/provider/store at g1/g2/g1 and at g2/g1/g2 (§5.1). The issue's verification method asks for each backup-age combination; these two are open.
- The capture does not record its scripts. Each bundle records the fixture's commit (5b29fed,
§2.6), not the commit or the hashes of the
run/scripts that drove it. The tie to 3c715e6 rests on the scripts' modification times, before the capture began. A future capture should recordgit rev-parse HEADand therun/hashes. - The key-state comparison brackets
checkrows only (§2.2). A key created by a standaloneregenrow, such as 037, would show only in the next check's "before" state. run/alltells the control by its file name. It counts a leak-scan hit on any path ending in/data/canary-control.txtas the control (run/all:32-33). The fixture's own control test also checks the file's content and links, but its verdict goes tostates-<label>.stderr, whichrun/alldoes not read andrun/collect-evidencedoes not copy.- Smaller harness properties.
revoked()(run/scenario:278-285) takes any lookup error as revoked; row 102 shows the error wasinvalid accessor, so the result stands.key_state(run/scenario:96-101) takes a failed listing with empty output as no keys.make_release(run/scenario:66-80) would run as the root token if the compiler's token file were empty, since the fixture'sbaofalls back to it (fixtures/lib.sh:687).run/collect-evidence:38rewrites the checkout path beforeKL_OUT, so a sibling<checkout>-evidenceoutput, whichrun/lib.sh:28-31allows, would come out as<repo>-evidence. The committed evidence holds neither<KL_OUT>nor<repo>-evidence, and does not record whichKL_OUTthe capture used. That guard also comparesKL_OUTas given, not canonicalized, so a path reaching the checkout through..or a symlink would pass it.key_statefolds the listing's stderr into its output, so a mount with no key whose listing prints an error would read asunreadable, not as no keys. No check row had an empty mount: every check compared one to three keys, except the sealed row 096. - The store is a file copy. It held the token and accessor files and the fixture's own leak-scan
control (
canary-control.txt); no process held a database open in it. - What stays out of the repository. The bundles under
KL_OUTinclude the expanded store archives, which hold the run's tokens.run/collect-evidencecopies none of them and refuses collected evidence that looks like an OpenBao token. The command column ofverdicts.tsvholds the probe ciphertext's base64 plaintext (kl-probe, 039) and no synthetic value.
7. Recommendation¶
The recommendations below are inferences from the cases above. They are not observations.
- Record a release's key by identity, not by name and version alone. D shows that a key
recreated under a lost key's name produces ciphertexts labelled
vault:v1:like the lost key's (042). Contracts for issue 19 and the provider profile for issue 13 should pin something the provider cannot reissue, such as the key's creation time for that version. The compiler should also be kept withoutcreateon the key path (037). - Pair backups by age. A provider backup older than the database backup it is restored with loses what the database references (H). A database backup older than the provider loses release records and can fall below a raised floor (G). For issue 15: take the provider backup no earlier than the database backup it pairs with. Keep the credential store's backup matched to the provider's: a store newer than the provider (I) and one older (L) each lost the executor credential, and a lost store needed the matching backup (J).
- Tie floor raises and rewraps to the database retention window. Rewrapping made every rewrapped artifact depend on the newest key version (H), and raising the floor made every older database's artifacts undecryptable (G). While a database backup that references a key version may still be restored, keep that version decryptable, or accept regeneration and re-approval as the recovery path for those releases.
- Make the recovery check verify, stop, and name the missing piece. The check in this run tells applicable from regenerable from blocked by testing decryption and the source under the actual recovery identities, which spec §7 step 3 asks for. Its output names what failed. That is enough to mark a scope blocked with the missing dependency (spec:454). Administrator repairs (floor, reissue) stay explicit and outside the check.
- Irrecoverability rests on the backup inventory. A restore undid destruction and deletion here (047, 048). Anything created after the newest snapshot of any family is what can be lost for good (R3). The backup interval bounds that exposure.
8. Hand-off¶
- Issue 15, retention and recovery policy: the pairing rule and the floor and rewrap window (§7 items 2 and 3); the exposure bounded by the newest snapshot (item 5); the case table (§3.1) as the restoration matrix to keep.
- Issue 19, execution and recovery contracts: the prerequisites table (§5.3), the check's stop-and-name behaviour, key identity in the release record (item 1), and that a regeneration is a new release needing approval (G, H, I).
- Issue 13, deployment profile: Transit's
upsert-on-encrypt and the
createcapability (D); the unseal share as a custody item separate from every backup family (K). - Not covered, for whoever takes it up: the age combinations g1/g2/g1 and g2/g1/g2, restore onto a new provider cluster, token expiry across a restore, a least-privilege administrator policy, and a capture that records its own scripts (§6).