AES FIELD NOTES · OPERATING EVIDENCE
We Cut Seven Checks to Two—and the Automation Became More Honest
A build-in-public account of why AES reduced its editorial supervision from seven daily checkpoints to two, preserved failures before recovery, and separated safety gates from constant intervention.
We had built a diligent supervisor for the AES editorial loop. It checked the system at 08:10, 10:10, 12:10, 13:10, 15:10, 17:10 and 19:10. Each checkpoint could inspect state, repair a schedule and move work forward. On paper, that looked safe. In practice, it made the system look healthier than it was. The supervisor often arrived quickly enough to repair the visible symptom before we had preserved the failure that produced it.
So we removed five supervisory checkpoints. The loop now has two: 10:10 and 19:10, Moscow time. The workers and durable schedulers continue operating between them. More importantly, every checkpoint records the observed pre-repair state before it is allowed to recover anything. This was not a reduction in safety. Duplicate detection, idempotency, channel limits, asset validation and publication evidence all stayed in place. We reduced intervention, not control.
A green status can be a damaged signal
Frequent supervision creates a subtle failure mode. A worker misses a handoff, a scheduler does not fire, or a candidate generator returns nothing. The next check notices the gap and repairs it. By the time a human looks, the final state is green. The delivery exists, but the operating evidence no longer clearly shows whether the worker succeeded or the supervisor rescued it. Reliability and rescue are collapsed into one number.
That distinction matters for autonomous systems. If a recovery mechanism is always standing next to the worker, the organization cannot tell which component is dependable. It can count completed outputs, but it cannot attribute them. The result is false confidence: a workflow appears autonomous because the supervisor quietly performs the missing work. A healthy final state is useful; a healthy final state without causal history is dangerous.
Do not repair the evidence away. Observe first, preserve the trace, then recover.
The new contract: observe, preserve, recover
- Observe the externally meaningful state: public Blog pages, scheduled work, registered assets, channel limits and worker health.
- Preserve the pre-repair snapshot with timestamps, identifiers, status and the exact missing evidence.
- Recover only an authorized overdue stage whose handoff, asset, safety checks and idempotency key are already valid.
This contract follows a basic observability principle: a system should let operators ask not only what is broken, but why. Google SRE makes the same distinction between symptoms and causes, and warns that noisy alerting can mask the incidents that actually matter. OpenTelemetry frames observability as the ability to investigate unknown problems through correlated signals rather than through a single health indicator. Our editorial loop is smaller than a distributed production platform, but the architectural problem is the same.
What the first real failure taught us
The new cadence immediately made an uncomfortable pattern easier to see. The editorial engine kept proposing variations of the same architecture-and-governance thesis. Under the old rhythm, another recovery could have generated a cover, created a draft and pushed the pipeline toward publication. Under the new contract, the semantic novelty gate stopped the candidate before writing and visual generation. The day remained visibly blocked instead of being cosmetically completed.
That blocked state was valuable. It showed that the missing capability was not another scheduler check. The missing capability was topic diversity: a stronger archive comparison, several editorial lanes and a retry policy that changes the candidate rather than paraphrasing the same idea. We fixed that mechanism. Weekdays now lead with material AI news, Saturday is reserved for practical operating evidence, and Sunday carries the deeper Systems Architect point of view.
What we removed—and what we kept
It is easy to hear “seven checks became two” and assume that the system now observes less. That is not the design. The delivery workers still emit state. Schedules remain durable. Public endpoints, database state and asset records remain inspectable. What disappeared was five opportunities for the supervisor to mutate the workflow during the day. Observation can be continuous; intervention should be sparse, explicit and recorded. This separation gives telemetry a chance to describe the worker before the supervisor changes its environment.
- We kept hard gates at irreversible boundaries: no publication without validated bilingual copy, a registered unique asset and a stable idempotency key.
- We kept channel policy outside the recovery decision: a Saturday article cannot silently become a Telegram post, and a paused LinkedIn module cannot become a blocker.
- We added negative evidence: an expected output that did not appear is recorded as a fact, not overwritten by the next successful attempt.
This produces a cleaner accountability model. The worker owns first execution. The scheduler owns timing. The supervisor owns diagnosis and bounded recovery. The audit trail owns the truth about which one actually completed the stage. When those responsibilities are blended, every success looks collective and every failure looks ambiguous. When they are separated, a repair can be celebrated as a successful recovery without being mislabelled as a successful first attempt.
Control is not the same as intervention
The lesson is not that fewer checks are universally better. Measurement resolution should match the behavior and risk of the system. A payment authorization path and a daily editorial workflow do not need the same polling frequency. The lesson is narrower: supervisory frequency must not destroy diagnostic value, and recovery must not be confused with successful execution. Keep high-frequency telemetry where it answers a real question. Keep safety gates at every irreversible boundary. But make supervisory intervention deliberate and attributable.
For us, two checkpoints are now enough to answer three questions. Did the autonomous worker do what it was assigned? If not, what exact evidence is missing? Can the authorized recovery complete the stage without creating a duplicate or violating a channel rule? That is a more useful operating model than seven opportunities to make the dashboard green. The automation did not become trustworthy because it failed less. It became easier to trust because it could no longer hide how it succeeded.
