Recover (RC) · Authorization
Authorization at the server side layer
external · Outside the agent entirely
How do I get back to a known-good state and not repeat this?
What this cell does
Reapply RBAC, Kyverno, IAM from declarative source; audit for drift between cluster state and source.
Artifacts (2)
agent-restore-authorization-serverview on GitHub#!/usr/bin/env bash
# ABOUTME: Server-side authorization-rebuild runbook. Removes emergency-deny, reapplies RBAC + Kyverno + IAM from declarative source.
# ABOUTME: Source-of-truth manifests must be intact (signed commits, signed tags, off-cluster mirror); script does not check repo integrity.
set -euo pipefail
if [[ $# -lt 2 ]]; then
echo "Usage: agent-restore-authorization-server <AGENT_NAME> <INCIDENT_ID>" >&2
exit 64
fi
AGENT_NAME="$1"
INCIDENT_ID="$2"
NAMESPACE="agent-${AGENT_NAME}"
MANIFESTS_DIR="${MANIFESTS_DIR:-./manifests}"
# 1. Remove emergency Kyverno deny-all ClusterPolicies (there may be multiple,
# each tagged with an incident ID).
mapfile -t EMERGENCY_POLICIES < <(
kubectl get clusterpolicies -o name 2>/dev/null \
| grep -E "emergency-deny-(all|agents|image-)"
)
for cp in "${EMERGENCY_POLICIES[@]}"; do
kubectl delete "$cp" 2>/dev/null || true
done
# 2. Reapply the agent's Role and RoleBinding from declarative source.
ROLE_FILE="$MANIFESTS_DIR/rbac/${AGENT_NAME}-role.yaml"
BINDING_FILE="$MANIFESTS_DIR/rbac/${AGENT_NAME}-rolebinding.yaml"
for f in "$ROLE_FILE" "$BINDING_FILE"; do
if [[ -r "$f" ]]; then
kubectl apply -f "$f"
else
echo "WARN: $f not found in source; skipping" >&2
fi
done
# 3. Reapply the operational Kyverno policies.
if [[ -d "$MANIFESTS_DIR/kyverno" ]]; then
kubectl apply -f "$MANIFESTS_DIR/kyverno/" 2>/dev/null || true
fi
# 4. Remove any lingering EmergencyDenyAll IAM policies on the agent role.
mapfile -t IAM_POLICIES < <(
aws iam list-role-policies --role-name "$AGENT_NAME" \
--query 'PolicyNames[?starts_with(@, `EmergencyDenyAll`)]' \
--output text 2>/dev/null | tr '\t' '\n'
)
for p in "${IAM_POLICIES[@]}"; do
[[ -z "$p" ]] && continue
aws iam delete-role-policy --role-name "$AGENT_NAME" --policy-name "$p" 2>/dev/null || true
done
# 5. Audit drift between cluster state and source.
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
if [[ -x "$SCRIPT_DIR/drift-audit.sh" ]]; then
echo "Drift audit:"
"$SCRIPT_DIR/drift-audit.sh" "$NAMESPACE" || true
fi
logger -t agent-recovery -p user.notice \
"$(jq -n \
--arg event "authorization_restored_server" \
--arg agent "$AGENT_NAME" \
--arg incident "$INCIDENT_ID" \
--arg actor "$(whoami)" \
--arg ts "$(date -Iseconds)" \
'{event:$event, agent:$agent, incident:$incident, actor:$actor, ts:$ts}')"
echo "Server-side authorization restored for $AGENT_NAME"
echo "Review drift-audit output before declaring recovery complete."
drift-audit.shview on GitHub#!/usr/bin/env bash
# ABOUTME: Drift-detection helper. Lists Kubernetes resources in a namespace not present in the declarative source-of-truth.
# ABOUTME: Anything in cluster but not in source is suspect after a recovery; review and decide delete or capture.
set -euo pipefail
NAMESPACE="${1:-}"
MANIFESTS_DIR="${MANIFESTS_DIR:-./manifests}"
if [[ -z "$NAMESPACE" ]]; then
echo "Usage: drift-audit.sh <NAMESPACE>" >&2
exit 64
fi
# Resource kinds worth auditing. Add or remove for your environment.
KINDS=(
configmaps secrets serviceaccounts
roles rolebindings
deployments daemonsets statefulsets
services networkpolicies
pods
)
echo "Drift audit for namespace: $NAMESPACE"
echo "Source manifests: $MANIFESTS_DIR"
echo
for kind in "${KINDS[@]}"; do
mapfile -t CLUSTER_NAMES < <(
kubectl get "$kind" -n "$NAMESPACE" -o jsonpath='{.items[*].metadata.name}' 2>/dev/null \
| tr ' ' '\n' | sort -u
)
for name in "${CLUSTER_NAMES[@]}"; do
[[ -z "$name" ]] && continue
# Heuristic: assume manifest filename pattern is <name>-<kind>.yaml or <kind>/<name>.yaml.
# Drift detection here is a starter; tighten for your repo layout.
if ! grep -rqE "^\s*name:\s*$name\s*$" "$MANIFESTS_DIR" 2>/dev/null; then
echo "DRIFT: $kind/$name in cluster but not referenced in $MANIFESTS_DIR/"
fi
done
done
echo
echo "Review each DRIFT line. Anything not deliberate during recovery should be deleted."
Cell notes
Restorations, Authorization / Server-side
Precondition. Interventions L3-C2 has fired (Kyverno deny-all in effect, agent Role empty, IAM deny-all attached). Restorations identity row complete. The declarative source-of-truth has been verified intact (signed commits, signed tags, off-cluster mirror).
Authority. On-call plus security review.
Tooling
- -
kubectlwith permission to apply RBAC and Kyverno resources. - - AWS CLI (or GCP/Azure equivalent) for IAM resync.
- -
kubectl-neatorkubedifffor drift detection (optional). - - Source-of-truth manifests:
manifests/rbac/,manifests/kyverno/,infrastructure/iam/.
Files in this directory
- -
agent-restore-authorization-server, runbook script. Removes emergency Kyverno deny ClusterPolicy, reapplies the agent's Role from source, reapplies the operational Kyverno policies, re-removes any EmergencyDenyAll IAM policies, audits cluster drift vs source. - -
drift-audit.sh, drift-detection helper. Lists resources in the agent namespace not present inmanifests/. Anything in cluster but not in source is suspect.
Verification
# 1. Emergency Kyverno deny-all removed
kubectl get clusterpolicy emergency-deny-all-agents 2>&1 | grep -i "not found"
# 2. Agent Role has rules from source
kubectl get role -n agent-claude-code-prod claude-code -o jsonpath='{.rules}'
# expected: matches manifests/rbac/<agent>-role.yaml
# 3. Operational Kyverno policies in effect
kubectl get clusterpolicies | grep -E "(agents-no-cluster-roles|verify-image-signatures)"
# 4. Drift audit clean
./drift-audit.sh agent-claude-code-prod
# expected: no entries (or all entries explained)
# 5. Test action that was denied during incident now succeeds
kubectl --as=system:serviceaccount:agent-claude-code-prod:claude-code get pods
# expected: success
Common failure modes
- - Declarative source itself was compromised during incident. Verify source repo integrity (signed commits, signed tags) before reapply.
- - Reapply does not delete drift; only adds/updates declared resources. Run drift audit and explicitly delete unknown resources.
- - Emergency-deny ClusterPolicy not removed. The agent has its Role back but is still globally denied by the emergency policy.
Citation
NIST CSF 2.0 RC.RP-01, RC.IM-01; PR.AA-05 (recovery dimension). NIST SP 800-207 (Zero Trust). CIS Kubernetes Benchmark. NIST AI RMF MANAGE 4.1.
Primary failure modes
Documented, not hypothetical. A control whose bypass is undocumented is worse than no control, because somebody trusted it.
- declarative source itself was compromised during incident
- reapply does not delete drift; only adds/updates declared resources
- emergency-deny ClusterPolicy not removed
Crosswalk
| NIST CSF 2 0 | RC.RP-01, RC.IM-01, PR.AA-05 |
|---|---|
| NIST AI RMF | MANAGE 4.1 |
| OWASP AGENTIC | ASI02, ASI03, ASI05 |
| OTHER | NIST SP 800-207, CIS Kubernetes Benchmark |
Cite this cell:
https://agenticcovenants.com/recover/authorization/server-side/