Agentic Covenants

Recover (RC) · Authorization

Authorization at the server side layer

external · Outside the agent entirely

How do I get back to a known-good state and not repeat this?

What this cell does

Reapply RBAC, Kyverno, IAM from declarative source; audit for drift between cluster state and source.

Artifacts (2)

agent-restore-authorization-serverview on GitHub
#!/usr/bin/env bash
# ABOUTME: Server-side authorization-rebuild runbook. Removes emergency-deny, reapplies RBAC + Kyverno + IAM from declarative source.
# ABOUTME: Source-of-truth manifests must be intact (signed commits, signed tags, off-cluster mirror); script does not check repo integrity.

set -euo pipefail

if [[ $# -lt 2 ]]; then
  echo "Usage: agent-restore-authorization-server <AGENT_NAME> <INCIDENT_ID>" >&2
  exit 64
fi

AGENT_NAME="$1"
INCIDENT_ID="$2"
NAMESPACE="agent-${AGENT_NAME}"
MANIFESTS_DIR="${MANIFESTS_DIR:-./manifests}"

# 1. Remove emergency Kyverno deny-all ClusterPolicies (there may be multiple,
# each tagged with an incident ID).
mapfile -t EMERGENCY_POLICIES < <(
  kubectl get clusterpolicies -o name 2>/dev/null \
    | grep -E "emergency-deny-(all|agents|image-)"
)
for cp in "${EMERGENCY_POLICIES[@]}"; do
  kubectl delete "$cp" 2>/dev/null || true
done

# 2. Reapply the agent's Role and RoleBinding from declarative source.
ROLE_FILE="$MANIFESTS_DIR/rbac/${AGENT_NAME}-role.yaml"
BINDING_FILE="$MANIFESTS_DIR/rbac/${AGENT_NAME}-rolebinding.yaml"
for f in "$ROLE_FILE" "$BINDING_FILE"; do
  if [[ -r "$f" ]]; then
    kubectl apply -f "$f"
  else
    echo "WARN: $f not found in source; skipping" >&2
  fi
done

# 3. Reapply the operational Kyverno policies.
if [[ -d "$MANIFESTS_DIR/kyverno" ]]; then
  kubectl apply -f "$MANIFESTS_DIR/kyverno/" 2>/dev/null || true
fi

# 4. Remove any lingering EmergencyDenyAll IAM policies on the agent role.
mapfile -t IAM_POLICIES < <(
  aws iam list-role-policies --role-name "$AGENT_NAME" \
    --query 'PolicyNames[?starts_with(@, `EmergencyDenyAll`)]' \
    --output text 2>/dev/null | tr '\t' '\n'
)
for p in "${IAM_POLICIES[@]}"; do
  [[ -z "$p" ]] && continue
  aws iam delete-role-policy --role-name "$AGENT_NAME" --policy-name "$p" 2>/dev/null || true
done

# 5. Audit drift between cluster state and source.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
if [[ -x "$SCRIPT_DIR/drift-audit.sh" ]]; then
  echo "Drift audit:"
  "$SCRIPT_DIR/drift-audit.sh" "$NAMESPACE" || true
fi

logger -t agent-recovery -p user.notice \
  "$(jq -n \
      --arg event "authorization_restored_server" \
      --arg agent "$AGENT_NAME" \
      --arg incident "$INCIDENT_ID" \
      --arg actor "$(whoami)" \
      --arg ts "$(date -Iseconds)" \
      '{event:$event, agent:$agent, incident:$incident, actor:$actor, ts:$ts}')"

echo "Server-side authorization restored for $AGENT_NAME"
echo "Review drift-audit output before declaring recovery complete."
drift-audit.shview on GitHub
#!/usr/bin/env bash
# ABOUTME: Drift-detection helper. Lists Kubernetes resources in a namespace not present in the declarative source-of-truth.
# ABOUTME: Anything in cluster but not in source is suspect after a recovery; review and decide delete or capture.

set -euo pipefail

NAMESPACE="${1:-}"
MANIFESTS_DIR="${MANIFESTS_DIR:-./manifests}"

if [[ -z "$NAMESPACE" ]]; then
  echo "Usage: drift-audit.sh <NAMESPACE>" >&2
  exit 64
fi

# Resource kinds worth auditing. Add or remove for your environment.
KINDS=(
  configmaps secrets serviceaccounts
  roles rolebindings
  deployments daemonsets statefulsets
  services networkpolicies
  pods
)

echo "Drift audit for namespace: $NAMESPACE"
echo "Source manifests: $MANIFESTS_DIR"
echo

for kind in "${KINDS[@]}"; do
  mapfile -t CLUSTER_NAMES < <(
    kubectl get "$kind" -n "$NAMESPACE" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null \
      | tr ' ' '\n' | sort -u
  )

  for name in "${CLUSTER_NAMES[@]}"; do
    [[ -z "$name" ]] && continue
    # Heuristic: assume manifest filename pattern is <name>-<kind>.yaml or <kind>/<name>.yaml.
    # Drift detection here is a starter; tighten for your repo layout.
    if ! grep -rqE "^\s*name:\s*$name\s*$" "$MANIFESTS_DIR" 2>/dev/null; then
      echo "DRIFT: $kind/$name in cluster but not referenced in $MANIFESTS_DIR/"
    fi
  done
done

echo
echo "Review each DRIFT line. Anything not deliberate during recovery should be deleted."

Cell notes

Restorations, Authorization / Server-side

Precondition. Interventions L3-C2 has fired (Kyverno deny-all in effect, agent Role empty, IAM deny-all attached). Restorations identity row complete. The declarative source-of-truth has been verified intact (signed commits, signed tags, off-cluster mirror).

Authority. On-call plus security review.

Tooling

  • - kubectl with permission to apply RBAC and Kyverno resources.
  • - AWS CLI (or GCP/Azure equivalent) for IAM resync.
  • - kubectl-neat or kubediff for drift detection (optional).
  • - Source-of-truth manifests: manifests/rbac/, manifests/kyverno/, infrastructure/iam/.

Files in this directory

  • - agent-restore-authorization-server, runbook script. Removes emergency Kyverno deny ClusterPolicy, reapplies the agent's Role from source, reapplies the operational Kyverno policies, re-removes any EmergencyDenyAll IAM policies, audits cluster drift vs source.
  • - drift-audit.sh, drift-detection helper. Lists resources in the agent namespace not present in manifests/. Anything in cluster but not in source is suspect.

Verification


# 1. Emergency Kyverno deny-all removed
kubectl get clusterpolicy emergency-deny-all-agents 2>&1 | grep -i "not found"

# 2. Agent Role has rules from source
kubectl get role -n agent-claude-code-prod claude-code -o jsonpath='{.rules}'
# expected: matches manifests/rbac/<agent>-role.yaml

# 3. Operational Kyverno policies in effect
kubectl get clusterpolicies | grep -E "(agents-no-cluster-roles|verify-image-signatures)"

# 4. Drift audit clean
./drift-audit.sh agent-claude-code-prod
# expected: no entries (or all entries explained)

# 5. Test action that was denied during incident now succeeds
kubectl --as=system:serviceaccount:agent-claude-code-prod:claude-code get pods
# expected: success

Common failure modes

  • - Declarative source itself was compromised during incident. Verify source repo integrity (signed commits, signed tags) before reapply.
  • - Reapply does not delete drift; only adds/updates declared resources. Run drift audit and explicitly delete unknown resources.
  • - Emergency-deny ClusterPolicy not removed. The agent has its Role back but is still globally denied by the emergency policy.

Citation

NIST CSF 2.0 RC.RP-01, RC.IM-01; PR.AA-05 (recovery dimension). NIST SP 800-207 (Zero Trust). CIS Kubernetes Benchmark. NIST AI RMF MANAGE 4.1.

Primary failure modes

Documented, not hypothetical. A control whose bypass is undocumented is worse than no control, because somebody trusted it.

  • declarative source itself was compromised during incident
  • reapply does not delete drift; only adds/updates declared resources
  • emergency-deny ClusterPolicy not removed

Crosswalk

NIST CSF 2 0RC.RP-01, RC.IM-01, PR.AA-05
NIST AI RMFMANAGE 4.1
OWASP AGENTICASI02, ASI03, ASI05
OTHERNIST SP 800-207, CIS Kubernetes Benchmark