All field notes

Recovery architecture

Organizational State Is Restorable. Organizational Permission Is Not.

A recovery architecture for autonomous organizations must restore evidence and unfinished work without automatically reviving revoked authority, stale observations or already completed external effects.

MP
Max PerfiljevFounder & CEO, AES · Architect of Autonomous Organizations
Read in Russian

A backup is usually framed as a promise about time: how much data can be lost, and how quickly can service return? That is necessary, but it is not enough for an autonomous organization. Its runtime may hold workflow positions, agent memory, queued messages, knowledge versions, policy references, identities, leases, obligations and records of intended actions. Restoring those materials can reconstruct an organization’s past. It must not, by itself, re-establish its right to act in the present.

The core recovery rule should be simple: organizational state is restorable; organizational permission is not. A restored runtime may recover durable evidence and unfinished work, but executable authority must be checked again against the live identity and policy planes. This is not an argument against backing up credential material or token records. It is an argument against allowing their presence in a snapshot to become an authorization decision.

A checkpoint must describe an organization, not just its database

A useful checkpoint begins with a more complete definition of state. Chandy and Lamport define a distributed global state as the combined state of processes and communication channels. Checkpointing is one of the applications of that model. For an autonomous organization, a runtime image that captures workflow rows but omits pending messages may therefore fail to represent a consistent organizational state. A message in transit can be the difference between an obligation that remains open and an action that appears never to have been requested.

The checkpoint should span at least process state, messages, memory, knowledge and policy version references, identity bindings, leases, obligations, and effect records. This does not mean all of these objects have the same recovery treatment. It means the organization needs a coherent record of what it knew, what it had started, what it believed it was allowed to do, and what it may already have done.

Effect records deserve special attention. A distributed snapshot does not prove the state of a payment processor, mail system, regulator portal or other external party. An agent may have formed an intention to send a payment; the runtime may have recorded a request; the external system may have completed, rejected or ambiguously received it. Recovery must preserve the evidence needed to investigate that boundary. It cannot infer external completion from an internal checkpoint alone.

Restore into quarantine, not into execution

The first destination for a restored organization should be quarantine: a contained runtime that can read restored state, test its integrity and prepare a recovery decision, but cannot exercise normal external authority. This is an operational boundary, not a claim that a snapshot is untrustworthy. A valid historical state can still be dangerous when reconnected to the present.

HashiCorp, for example, instructs operators to test Vault snapshot restoration in an isolated network because a restored instance could interact with live third-party credentials, including by attempting revocations that affect production. The point generalizes. Control-plane state is not inert merely because it came from a backup. Once network access and executors return, historical state can create fresh external effects.

Quarantine makes several necessary checks possible before the organization resumes. Verify backup integrity. Confirm that the checkpoint is internally consistent. Identify its policy, knowledge and schema references. Inventory identities, grants, tokens and leases that appear in the restored state. Compare them with live authority sources. Then decide which parts of the restored organization can cross from evidence into execution.

Authority must be reissued from live control planes

No restored agent, worker or workflow should receive executable authority merely because it held authority at checkpoint time. Identity bindings should be revalidated against the live identity plane. Policy references should be evaluated against the active policy plane. Leases should be recomputed or renewed through their live authority source. Fresh execution grants should be issued only after these checks, with the scope and duration appropriate to the work that remains.

This distinction matches how authorization systems already behave. OAuth 2.0 token revocation can invalidate a token and may also invalidate related tokens or the underlying authorization grant. A restored token record does not establish that the authorization server still recognizes that grant. Likewise, Vault documents dynamic secrets and service tokens with durations, renewal rules and revocation behavior. Stored lease state is evidence of a past condition, not proof of present executability.

The operating consequence is deliberately conservative: unfinished work may be restored as a claim, not resumed as a command. A workflow can retain its position, inputs, approvals and next intended transition. But the transition must pass current authorization and current policy before it invokes an executor. If the policy has changed, the identity was disabled, the grant was revoked or the lease expired, the work must be discarded, adapted, reassigned or escalated—not silently continued.

Invalidate the past before accepting the present

A recovery design also needs an epoch boundary. Kubernetes notes that an etcd snapshot contains all Kubernetes state and critical information, while its restore procedure stops API servers and restarts control-plane components so they do not continue from stale state. etcd’s disaster-recovery guidance further warns that restoring an older revision can leave controllers and cached clients unpredictable or inconsistent; it recommends revision bumps and compaction markers to invalidate watches and caches.

An autonomous runtime should apply the same principle explicitly. A recovery epoch should invalidate cached policy evaluations, identity assertions, authorization decisions, knowledge-derived conclusions, service discovery data and message-consumer positions where appropriate. Agents must not interpret a pre-failure cache as current organizational truth simply because it survived in memory or returned in a snapshot.

This is more than cache hygiene. A stale observation can be operationally equivalent to a stale permission. An agent that acts on a superseded customer status, an expired risk limit or an old policy binding may produce a validly formatted request that the organization should no longer make. Recovery has to re-establish a present-tense view before it permits present-tense action.

Reconcile effects; do not replay intentions

After authority and caches are addressed, the remaining hard problem is ambiguity at external boundaries. For each recorded intended or in-flight effect, the runtime should reconcile with the relevant external system where possible. The result should be classified: completed, not completed, safely retryable, requires compensation, or ambiguous and requiring escalation. An intention is not a licence to repeat an action after recovery.

NIST SP 800-53 separates backup from recovery and reconstitution. Its controls include integrity testing and test restoration, transaction recovery, and compensating security controls. That separation is the right architectural posture here: successful restoration is a governed validation process, not a file-copy operation. For high-consequence effects, an unresolved classification should block automation rather than invite optimistic retry.

Make the recovery decision inspectable

Before normal execution resumes, the runtime should issue a signed recovery manifest. This is a proposed architectural control, not an established standard. Its role is to state the recovery epoch, checkpoint identity and validation results, then enumerate what was restored as evidence, reauthorized for execution, discarded, reconciled, or escalated for human or designated-governor review.

The manifest turns a technical restart into an accountable organizational transition. Operators can inspect why a workflow resumed. Auditors can distinguish an old grant from a newly issued one. A downstream executor can require proof that a recovered task crossed the recovery boundary correctly. Most importantly, the organization gains a clear answer to the question that ordinary recovery metrics leave open: what, exactly, was allowed to act again?

Recovery planning for autonomous organizations should therefore measure more than recovery point and recovery time. It should test whether a checkpoint includes the organization’s relevant global state; whether restoration is isolated; whether live authority is revalidated; whether stale caches are invalidated; whether external effects are reconciled; and whether the final disposition is recorded. A system that comes back quickly but resumes revoked powers, duplicates payments or acts on stale truth has not recovered safely. It has only restarted.

BUILD WITH AES

Turn architecture into an operating company.

AES connects strategy, tasks, organizational memory, knowledge, agents, people and approvals in one execution environment.