Agentic Covenants

Recover (RC) · Blast radius

Blast radius at the server side layer

external · Outside the agent entirely

How do I get back to a known-good state and not repeat this?

What this cell does

Restore data from immutable backups verified pre-incident, redeploy IaC, reapply NetworkPolicy and ResourceQuota.

Artifacts (1)

agent-restore-blast-radius-serverview on GitHub
#!/usr/bin/env bash
# ABOUTME: Server-side blast-radius rebuild runbook. Removes emergency NetworkPolicy, reapplies operational policies, restores data from pre-incident backup.
# ABOUTME: Caller MUST pass a backup snapshot ID dated BEFORE the earliest indicator of compromise. Refuses without it.

set -euo pipefail

if [[ $# -lt 3 ]]; then
  echo "Usage: agent-restore-blast-radius-server <AGENT_NAME> <INCIDENT_ID> <BACKUP_SNAPSHOT_ID>" >&2
  echo "  BACKUP_SNAPSHOT_ID must be a snapshot dated BEFORE the earliest IoC from Sentinels." >&2
  exit 64
fi

AGENT_NAME="$1"
INCIDENT_ID="$2"
BACKUP_SNAPSHOT="$3"
NAMESPACE="agent-${AGENT_NAME}"
SOURCE_DIR="${SOURCE_DIR:-$(pwd)/controls/blast-radius/server-side}"
INFRASTRUCTURE_DIR="${INFRASTRUCTURE_DIR:-$(pwd)/infrastructure}"

# 1. Remove the emergency NetworkPolicy from Interventions.
kubectl delete networkpolicy -n "$NAMESPACE" emergency-deny-everything 2>/dev/null || true

# 2. Reapply operational NetworkPolicies from source.
if [[ -r "$SOURCE_DIR/networkpolicy-default-deny.yaml" ]]; then
  kubectl apply -n "$NAMESPACE" -f "$SOURCE_DIR/networkpolicy-default-deny.yaml"
fi
if [[ -r "$SOURCE_DIR/networkpolicy-allowlist.yaml" ]]; then
  kubectl apply -n "$NAMESPACE" -f "$SOURCE_DIR/networkpolicy-allowlist.yaml"
fi

# 3. Reapply ResourceQuota and LimitRange.
if [[ -r "$SOURCE_DIR/resourcequota.yaml" ]]; then
  kubectl apply -n "$NAMESPACE" -f "$SOURCE_DIR/resourcequota.yaml"
fi
if [[ -r "$SOURCE_DIR/limitrange.yaml" ]]; then
  kubectl apply -n "$NAMESPACE" -f "$SOURCE_DIR/limitrange.yaml"
fi

# 4. terraform plan + apply against fresh state. Operator must review the
# plan output before this script proceeds to apply.
if [[ -d "$INFRASTRUCTURE_DIR" ]] && command -v terraform >/dev/null 2>&1; then
  ( cd "$INFRASTRUCTURE_DIR" && terraform init -backend-config=backend.hcl ) >/dev/null
  ( cd "$INFRASTRUCTURE_DIR" && terraform plan -out=/tmp/restoration.tfplan -no-color )

  echo
  read -r -p "Review the terraform plan above. Apply now? (type 'apply' to proceed): " CONFIRM
  if [[ "$CONFIRM" == "apply" ]]; then
    ( cd "$INFRASTRUCTURE_DIR" && terraform apply /tmp/restoration.tfplan )
  else
    echo "Skipping terraform apply; recovery is not complete until apply is run."
  fi
fi

# 5. Restore data from the named pre-incident backup snapshot.
# This step is data-store-specific. Stub here invokes a restore-from-snapshot
# helper that the operator must provide for the relevant data store
# (RDS, Postgres backup, S3 versioned object copy, etc.).
if [[ -x "/usr/local/bin/restore-from-snapshot" ]]; then
  /usr/local/bin/restore-from-snapshot "$BACKUP_SNAPSHOT" "$AGENT_NAME"
else
  echo "WARN: /usr/local/bin/restore-from-snapshot not present; data restore must be performed manually" >&2
  echo "Backup snapshot to restore from: $BACKUP_SNAPSHOT"
fi

logger -t agent-recovery -p user.notice \
  "$(jq -n \
      --arg event "blast_radius_restored_server" \
      --arg agent "$AGENT_NAME" \
      --arg incident "$INCIDENT_ID" \
      --arg snapshot "$BACKUP_SNAPSHOT" \
      --arg actor "$(whoami)" \
      --arg ts "$(date -Iseconds)" \
      '{event:$event, agent:$agent, incident:$incident, snapshot:$snapshot, actor:$actor, ts:$ts}')"

echo "Server-side blast-radius restore complete for $AGENT_NAME (snapshot $BACKUP_SNAPSHOT)"

Cell notes

Restorations, Blast radius / Server-side

Precondition. Interventions L3-C3 has fired (NetworkPolicy default-deny applied, Deployments scaled to zero, pods deleted). Identity and Authorization restorations rows complete. The earliest indicator of compromise from Sentinels has been identified, pick a backup from before that timestamp.

Authority. On-call plus security review.

Tooling

  • - AWS CLI (or GCP/Azure equivalent) with permission to copy from immutable backup buckets.
  • - terraform (or tofu) for IaC redeploy.
  • - kubectl for NetworkPolicy and ResourceQuota reapply.
  • - The point-in-time-recovery tool for your data store: pg_restore, RDS PITR, etc.

Files in this directory

  • - agent-restore-blast-radius-server, runbook script. Removes emergency NetworkPolicy, reapplies operational NetworkPolicies + ResourceQuota + LimitRange from source, runs terraform apply against fresh state, optionally re-creates the namespace, restores data from a specified pre-incident backup snapshot.

Verification


# 1. Emergency NetworkPolicy removed
kubectl get networkpolicy -n agent-claude-code-prod emergency-deny-everything 2>&1 | grep -i "not found"

# 2. Operational NetworkPolicies in effect
kubectl get networkpolicy -n agent-claude-code-prod
# expected: default-deny-all + claude-code-egress

# 3. ResourceQuota in effect
kubectl get resourcequota -n agent-claude-code-prod

# 4. terraform plan is clean
terraform plan -no-color | grep -E "Plan: 0 to add"

# 5. Data restored to expected pre-incident state
# (data-store-specific verification; check row counts / object inventory against pre-incident snapshot)

Common failure modes

  • - Backup taken after contamination point. Always pick a backup from before the earliest indicator of compromise from Sentinels, not the most recent backup.
  • - Backups stored in same account as production. Immutable backups (S3 Object Lock, GCP Bucket Retention) reduce but do not eliminate this risk if the credentials that wrote them were compromised.
  • - Emergency NetworkPolicy not removed. Workloads come back up but cannot communicate.
  • - terraform apply against drifted state, the recovery applies the state-of-the-incident, not the desired state. Always terraform plan first and review the diff.

Citation

NIST CSF 2.0 RC.RP-01, RC.IM-01; PR.IR-01 (recovery dimension); PR.DS-11 (backup restoration). NIST SP 800-34 Rev. 1 (contingency planning). NISTIR 8596. NIST AI RMF MANAGE 4.1.

Primary failure modes

Documented, not hypothetical. A control whose bypass is undocumented is worse than no control, because somebody trusted it.

  • backup taken after contamination point
  • backups stored in same account as production
  • emergency NetworkPolicy not removed

Crosswalk

NIST CSF 2 0RC.RP-01, RC.IM-01, PR.IR-01, PR.DS-11
NIST AI RMFMANAGE 4.1
OWASP AGENTICASI05, ASI08
OTHERNIST SP 800-34 Rev. 1, NISTIR 8596