Recover (RC) · Blast radius
Blast radius at the server side layer
external · Outside the agent entirely
How do I get back to a known-good state and not repeat this?
What this cell does
Restore data from immutable backups verified pre-incident, redeploy IaC, reapply NetworkPolicy and ResourceQuota.
Artifacts (1)
agent-restore-blast-radius-serverview on GitHub#!/usr/bin/env bash
# ABOUTME: Server-side blast-radius rebuild runbook. Removes emergency NetworkPolicy, reapplies operational policies, restores data from pre-incident backup.
# ABOUTME: Caller MUST pass a backup snapshot ID dated BEFORE the earliest indicator of compromise. Refuses without it.
set -euo pipefail
if [[ $# -lt 3 ]]; then
echo "Usage: agent-restore-blast-radius-server <AGENT_NAME> <INCIDENT_ID> <BACKUP_SNAPSHOT_ID>" >&2
echo " BACKUP_SNAPSHOT_ID must be a snapshot dated BEFORE the earliest IoC from Sentinels." >&2
exit 64
fi
AGENT_NAME="$1"
INCIDENT_ID="$2"
BACKUP_SNAPSHOT="$3"
NAMESPACE="agent-${AGENT_NAME}"
SOURCE_DIR="${SOURCE_DIR:-$(pwd)/controls/blast-radius/server-side}"
INFRASTRUCTURE_DIR="${INFRASTRUCTURE_DIR:-$(pwd)/infrastructure}"
# 1. Remove the emergency NetworkPolicy from Interventions.
kubectl delete networkpolicy -n "$NAMESPACE" emergency-deny-everything 2>/dev/null || true
# 2. Reapply operational NetworkPolicies from source.
if [[ -r "$SOURCE_DIR/networkpolicy-default-deny.yaml" ]]; then
kubectl apply -n "$NAMESPACE" -f "$SOURCE_DIR/networkpolicy-default-deny.yaml"
fi
if [[ -r "$SOURCE_DIR/networkpolicy-allowlist.yaml" ]]; then
kubectl apply -n "$NAMESPACE" -f "$SOURCE_DIR/networkpolicy-allowlist.yaml"
fi
# 3. Reapply ResourceQuota and LimitRange.
if [[ -r "$SOURCE_DIR/resourcequota.yaml" ]]; then
kubectl apply -n "$NAMESPACE" -f "$SOURCE_DIR/resourcequota.yaml"
fi
if [[ -r "$SOURCE_DIR/limitrange.yaml" ]]; then
kubectl apply -n "$NAMESPACE" -f "$SOURCE_DIR/limitrange.yaml"
fi
# 4. terraform plan + apply against fresh state. Operator must review the
# plan output before this script proceeds to apply.
if [[ -d "$INFRASTRUCTURE_DIR" ]] && command -v terraform >/dev/null 2>&1; then
( cd "$INFRASTRUCTURE_DIR" && terraform init -backend-config=backend.hcl ) >/dev/null
( cd "$INFRASTRUCTURE_DIR" && terraform plan -out=/tmp/restoration.tfplan -no-color )
echo
read -r -p "Review the terraform plan above. Apply now? (type 'apply' to proceed): " CONFIRM
if [[ "$CONFIRM" == "apply" ]]; then
( cd "$INFRASTRUCTURE_DIR" && terraform apply /tmp/restoration.tfplan )
else
echo "Skipping terraform apply; recovery is not complete until apply is run."
fi
fi
# 5. Restore data from the named pre-incident backup snapshot.
# This step is data-store-specific. Stub here invokes a restore-from-snapshot
# helper that the operator must provide for the relevant data store
# (RDS, Postgres backup, S3 versioned object copy, etc.).
if [[ -x "/usr/local/bin/restore-from-snapshot" ]]; then
/usr/local/bin/restore-from-snapshot "$BACKUP_SNAPSHOT" "$AGENT_NAME"
else
echo "WARN: /usr/local/bin/restore-from-snapshot not present; data restore must be performed manually" >&2
echo "Backup snapshot to restore from: $BACKUP_SNAPSHOT"
fi
logger -t agent-recovery -p user.notice \
"$(jq -n \
--arg event "blast_radius_restored_server" \
--arg agent "$AGENT_NAME" \
--arg incident "$INCIDENT_ID" \
--arg snapshot "$BACKUP_SNAPSHOT" \
--arg actor "$(whoami)" \
--arg ts "$(date -Iseconds)" \
'{event:$event, agent:$agent, incident:$incident, snapshot:$snapshot, actor:$actor, ts:$ts}')"
echo "Server-side blast-radius restore complete for $AGENT_NAME (snapshot $BACKUP_SNAPSHOT)"
Cell notes
Restorations, Blast radius / Server-side
Precondition. Interventions L3-C3 has fired (NetworkPolicy default-deny applied, Deployments scaled to zero, pods deleted). Identity and Authorization restorations rows complete. The earliest indicator of compromise from Sentinels has been identified, pick a backup from before that timestamp.
Authority. On-call plus security review.
Tooling
- - AWS CLI (or GCP/Azure equivalent) with permission to copy from immutable backup buckets.
- -
terraform(ortofu) for IaC redeploy. - -
kubectlfor NetworkPolicy and ResourceQuota reapply. - - The point-in-time-recovery tool for your data store:
pg_restore, RDS PITR, etc.
Files in this directory
- -
agent-restore-blast-radius-server, runbook script. Removes emergency NetworkPolicy, reapplies operational NetworkPolicies + ResourceQuota + LimitRange from source, runsterraform applyagainst fresh state, optionally re-creates the namespace, restores data from a specified pre-incident backup snapshot.
Verification
# 1. Emergency NetworkPolicy removed
kubectl get networkpolicy -n agent-claude-code-prod emergency-deny-everything 2>&1 | grep -i "not found"
# 2. Operational NetworkPolicies in effect
kubectl get networkpolicy -n agent-claude-code-prod
# expected: default-deny-all + claude-code-egress
# 3. ResourceQuota in effect
kubectl get resourcequota -n agent-claude-code-prod
# 4. terraform plan is clean
terraform plan -no-color | grep -E "Plan: 0 to add"
# 5. Data restored to expected pre-incident state
# (data-store-specific verification; check row counts / object inventory against pre-incident snapshot)
Common failure modes
- - Backup taken after contamination point. Always pick a backup from before the earliest indicator of compromise from Sentinels, not the most recent backup.
- - Backups stored in same account as production. Immutable backups (S3 Object Lock, GCP Bucket Retention) reduce but do not eliminate this risk if the credentials that wrote them were compromised.
- - Emergency NetworkPolicy not removed. Workloads come back up but cannot communicate.
- -
terraform applyagainst drifted state, the recovery applies the state-of-the-incident, not the desired state. Alwaysterraform planfirst and review the diff.
Citation
NIST CSF 2.0 RC.RP-01, RC.IM-01; PR.IR-01 (recovery dimension); PR.DS-11 (backup restoration). NIST SP 800-34 Rev. 1 (contingency planning). NISTIR 8596. NIST AI RMF MANAGE 4.1.
Primary failure modes
Documented, not hypothetical. A control whose bypass is undocumented is worse than no control, because somebody trusted it.
- backup taken after contamination point
- backups stored in same account as production
- emergency NetworkPolicy not removed
Crosswalk
| NIST CSF 2 0 | RC.RP-01, RC.IM-01, PR.IR-01, PR.DS-11 |
|---|---|
| NIST AI RMF | MANAGE 4.1 |
| OWASP AGENTIC | ASI05, ASI08 |
| OTHER | NIST SP 800-34 Rev. 1, NISTIR 8596 |
Cite this cell:
https://agenticcovenants.com/recover/blast-radius/server-side/