Proposed thesisDelegated autonomy needs an explicit account of prevention, detection, intervention, and recovery, not merely an estimate that an action is likely to be correct.
Assumptions behind the argument
  • The action creates consequences that matter beyond the generated output.
  • Some relevant deviations are observable, but signals may be delayed or incomplete.
  • An accountable owner can define authorized actions and response options.

01 / Confidence answers only part of the question

Suppose an AI system is highly confident that a proposed action will achieve a goal. What follows? We have an estimate about the action, provided that confidence has a defensible meaning. We still need to know whose goal it serves, which constraints apply, and who is entitled to make the commitment.

A prediction concerns what may be true or what may happen. Permission concerns which actions an institution authorizes. The first can inform the second; it cannot supply it. A more accurate model does not automatically acquire a larger mandate.

My proposal is to evaluate autonomy through a theory of correction: what can be prevented, what can be detected, how intervention works, and what remains possible after consequences occur.

02 / What the control perspective contributes

Feedback control studies how observations of a system can inform actions that influence its behavior. Åström and Murray’s Feedback Systems develops this relationship and the constraints involved in designing feedback. [1] It offers a useful source of questions about an AI-enabled workflow.

What is the desired condition? Which signals indicate a meaningful deviation? How delayed or noisy are those signals? Which interventions are available? Who can use them?

The analogy should remain disciplined. An organization is not a fully specified physical plant, and a dashboard is not evidence of formal controllability or stability. Business objectives can change, people can disagree, and important harms may remain unobserved. I use the control perspective to expose those limitations, not to claim that a diagram solves them.

03 / The interval that matters

Consider an illustrative scheduling assistant authorized to confirm an appointment when a qualified specialist is available. After approval, the specialist is reassigned. If the change is visible before confirmation, the assistant can pause and seek another valid option. If it is discovered afterward, the organization may need to contact the customer and arrange a replacement.

The same deviation has different consequences depending on when it is detected and what can still be changed. The relevant interval runs from the change in conditions to the point beyond which a particular remedy is no longer available.

A useful intervention path must fit within that interval. This includes the time to detect the change, understand it, obtain the necessary authority, and carry out the response. A nominal human review adds little protection if the reviewer receives the signal too late or lacks the ability to alter the commitment.

04 / Separate prevention, containment, and recovery

Correction should not be used as a vague synonym for undo. I would distinguish three design responsibilities.

  • Prevention: restrict the action before an unacceptable commitment can occur, including checks, limits, or deliberate non-action.
  • Containment: stop additional consequences once a deviation is detected, such as suspending related actions while preserving useful work.
  • Recovery: restore what can be restored and authorize compensating actions for what cannot.

05 / The counterexample: sometimes action cannot wait

A theory that permits autonomy only when every action is reversible would be too restrictive. A time-critical protective action can be justified even when it has irreversible consequences, and waiting for a person could make the situation worse. Conversely, a reversible database update can still trigger an external commitment that is difficult to recover.

These cases limit the role of feedback. Some decisions require prior constraints, carefully bounded authority, and assurance appropriate to the consequence, rather than a promise to learn afterward. Others should remain outside the system’s mandate. This essay provides no safety certification or universal rule for those judgments.

The narrower claim is that autonomy should have an explicit response theory. The organization should know which failures it intends to prevent, which it can contain, and which residual consequences it is choosing to accept through an authorized decision.

06 / A test of the response theory

In a simulator or controlled replay, hold the model, task set, and initial permissions constant. Compare a policy that executes after a confidence threshold is met with one that also checks relevant changes and provides an authorized interruption path. Restrict the experiment to cases where the added signal could actually change the permissible action.

Introduce the same disturbances under both policies, varying when they become visible relative to the commitment boundary. Include false alarms and cases with no disturbance. Measure preventable violations, successful interventions, unnecessary interruptions, completion time, and residual recovery work.

The hypothesis predicts that the intervention-aware policy will help when the deviation can be detected and acted on in time. The advantage should diminish when feedback arrives after the relevant remedy is unavailable. If violations persist despite timely signals and sufficient response capacity, the proposed control path is inadequate. If interruptions dominate without reducing consequential errors, the policy may be too broad or the signal unhelpful.

07 / Feedback must be able to change authority

Repeated evidence that a system operates outside its intended conditions should trigger a decision about scope. The response might be a narrower mandate, a new source check, or suspension of a particular action. It should not depend on the system silently rewriting its own permissions.

That is where control and institutional design meet. Someone must own the right to change the rules of action, not merely observe the outcomes. Mature autonomy includes both effective execution and a credible account of how its mandate can be questioned, constrained, and revised.

Before expanding autonomy, explain how the organization will prevent, detect, contain, and respond to the failures that matter.
A conceptual essay with proposed hypotheses and experiments. These tests have not been run as part of this site. Examples are illustrative, not employer or client case studies. Views are my own.

Sources & intellectual foundations

  1. [1] Karl J. Åström & Richard M. Murray, Feedback Systems Foundation: feedback as a relationship among observations, system behavior, and action. The organizational correction framework and experiment are conceptual proposals, not an application of a proved stability result or a safety assurance method.