Respond (RS) · Supply chain
Supply chain at the server side layer
external · Outside the agent entirely
How do I stop the bleeding now?
What this cell does
Remove poisoned image from registry, deploy emergency Kyverno deny rule on signature, block compromised registry/MCP domain at network/DNS, force redeploy with last-known-good image SHA pinned.
Artifacts (3)
agent-quarantine-supply-chain-serverview on GitHub#!/usr/bin/env bash
# ABOUTME: Server-side supply-chain quarantine runbook. crane delete, Kyverno deny on digest, Cilium FQDN deny, force redeploy.
# ABOUTME: Pre-stage cilium-deny-suspect-fqdns.yaml and last-known-good-image-sha.txt under /etc/agents/emergency/.
set -euo pipefail
if [[ $# -lt 2 ]]; then
echo "Usage: agent-quarantine-supply-chain-server <AGENT_NAME> <SUSPECT_IMAGE>" >&2
echo " SUSPECT_IMAGE format: registry/org/name@sha256:DIGEST" >&2
exit 64
fi
AGENT_NAME="$1"
SUSPECT_IMAGE="$2"
NAMESPACE="agent-${AGENT_NAME}"
INCIDENT_ID="$(uuidgen 2>/dev/null || python3 -c 'import uuid; print(uuid.uuid4())')"
EMERGENCY_DIR="${EMERGENCY_DIR:-/etc/agents/emergency}"
if [[ ! -r "$EMERGENCY_DIR/last-known-good-image-sha.txt" ]]; then
echo "REFUSING: $EMERGENCY_DIR/last-known-good-image-sha.txt not pre-staged" >&2
exit 1
fi
# Extract digest portion of the suspect image. Used for the Kyverno deny rule
# so an attacker re-tagging the same digest does not bypass.
DIGEST="${SUSPECT_IMAGE##*@}"
if [[ "$DIGEST" == "$SUSPECT_IMAGE" || ! "$DIGEST" =~ ^sha256:[0-9a-f]+$ ]]; then
echo "REFUSING: SUSPECT_IMAGE must be a digest reference (registry/name@sha256:...)" >&2
exit 64
fi
# 1. Remove poisoned image from registry. May fail if registry permissions
# do not allow delete; warn rather than abort so the rest of the runbook runs.
if command -v crane >/dev/null 2>&1; then
crane delete "$SUSPECT_IMAGE" 2>/dev/null || \
echo "WARN: image deletion failed; manual registry intervention may be needed" >&2
else
echo "WARN: crane not installed; skipping registry deletion" >&2
fi
# 2. Apply emergency Kyverno deny rule on the suspect digest.
SAFE_INCIDENT="${INCIDENT_ID//-/}"
kubectl apply -f - <<EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: emergency-deny-image-${SAFE_INCIDENT:0:16}
annotations:
incident: "true"
policies.kyverno.io/severity: critical
spec:
validationFailureAction: Enforce
background: false
rules:
- name: deny-suspect-digest
match:
any:
- resources:
kinds: [Pod]
validate:
message: "Image quarantined under incident ${INCIDENT_ID}"
pattern:
spec:
containers:
- image: "!*${DIGEST}*"
EOF
# 3. Block egress to compromised MCP domain. The pre-staged CNP must list the
# domain. For dynamic blocking, edit the pre-staged file before applying.
if [[ -r "$EMERGENCY_DIR/cilium-deny-suspect-fqdns.yaml" ]]; then
kubectl apply -f "$EMERGENCY_DIR/cilium-deny-suspect-fqdns.yaml" 2>/dev/null || \
echo "WARN: Cilium policy apply failed (Cilium may not be the CNI here)" >&2
fi
# 4. Force redeploy workloads with last-known-good image SHA pinned.
LAST_GOOD_SHA="$(tr -d '[:space:]' < "$EMERGENCY_DIR/last-known-good-image-sha.txt")"
LAST_GOOD_IMAGE="${SUSPECT_IMAGE%%@*}@$LAST_GOOD_SHA"
kubectl set image -n "$NAMESPACE" \
deployment/claude-code \
"claude=$LAST_GOOD_IMAGE" 2>/dev/null || \
echo "WARN: deployment image update failed (deployment may be named differently)" >&2
kubectl rollout restart -n "$NAMESPACE" deployment/claude-code 2>/dev/null || true
logger -t agent-incident -p user.warning \
"$(jq -n \
--arg event "supply_chain_quarantined_server" \
--arg agent "$AGENT_NAME" \
--arg suspect "$SUSPECT_IMAGE" \
--arg digest "$DIGEST" \
--arg incident "$INCIDENT_ID" \
--arg actor "$(whoami)" \
--arg ts "$(date -Iseconds)" \
'{event:$event, agent:$agent, suspect:$suspect, digest:$digest, incident:$incident, actor:$actor, ts:$ts}')"
echo "Server-side supply chain quarantined for $AGENT_NAME (suspect: $SUSPECT_IMAGE, incident: $INCIDENT_ID)"
cilium-deny-suspect-fqdns.yamlview on GitHub# ABOUTME: Pre-staged CiliumNetworkPolicy denying egress to known-suspect FQDNs. Edit before applying or pre-populate at incident start.
# ABOUTME: Requires Cilium with enable-l7-proxy: true. The toFQDNs list is the deny mechanism — DNS resolution to these names returns NXDOMAIN.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: emergency-deny-suspect-fqdns
namespace: kube-system
annotations:
incident: "true"
spec:
endpointSelector: {}
egressDeny:
- toFQDNs:
# Substitute these entries with the known-malicious FQDNs from the
# incident. The runbook expects the file to be edited (or the list to
# be maintained as a known-bad inventory) before apply.
- matchName: "suspect-mcp.example.com"
- matchName: "evil-registry.example.com"
- matchPattern: "*.attacker-tld.example"
last-known-good-image-sha.txtview on GitHub55 bytes. Too large to inline; open it on GitHub.
Cell notes
Interventions, Supply chain / Server-side
Trigger. Cosign verification failure, SBOM diff with unauthorized package, image registry pull of unsigned image, FQDN egress denial spike to a suspect MCP domain.
Authority. On-call plus security review (registry-level changes affect other workloads).
Speed target. Under 5 minutes.
Tooling
- -
crane(or registry-specific CLI) for image deletion. - -
kubectlwith permission to apply emergency ClusterPolicies and CiliumNetworkPolicies. - - Cilium with Hubble for FQDN-deny logging.
Files in this directory
- -
agent-quarantine-supply-chain-server, runbook script. Removes the poisoned image viacrane delete, deploys an emergency Kyverno deny rule on the bad digest, applies the pre-staged Cilium FQDN deny policy, force-rolls workloads to the last-known-good image SHA. - -
cilium-deny-suspect-fqdns.yaml, pre-staged CiliumNetworkPolicy denying egress to a list of suspect FQDNs. Substitute thematchNameentries during the incident or pre-stage with a known-malicious list. Pre-stage at/etc/agents/emergency/cilium-deny-suspect-fqdns.yaml. - -
last-known-good-image-sha.txt, pre-staged digest of the last-known-good agent image. Audit periodically (the pre-staged value should not itself be contaminated). Pre-stage at/etc/agents/emergency/last-known-good-image-sha.txt.
Verification
# 1. Image removed from registry
crane manifest "$SUSPECT_IMAGE" 2>&1 | grep -i "not found" || echo "FAIL: image still in registry"
# 2. Kyverno deny in effect
kubectl run test --image="$SUSPECT_IMAGE" -n agent-claude-code-prod 2>&1 | grep -i "quarantined" || echo "FAIL: deny did not fire"
# 3. Egress to suspect domain blocked
kubectl exec -n agent-claude-code-prod $(kubectl get pods -n agent-claude-code-prod -o name | head -1) -- \
curl -sS --max-time 3 https://suspect-mcp.example.com 2>&1 | grep -iE "couldn.t resolve|connection refused"
# 4. Workloads on last-known-good image
kubectl get deployment -n agent-claude-code-prod claude-code -o jsonpath='{.spec.template.spec.containers[0].image}'
# expected: matches last-known-good
Common mistakes
- - Image deletion does not affect already-pulled cached images on nodes. The force-rollout step is mandatory.
- - Cosign deny rule by tag instead of digest, attacker pushes a new tag pointing at the same digest.
- - DNS block applied at one level (cluster CNP) but not at corporate DNS, agents on operator hosts still resolve.
- - Last-known-good SHA file is stale or itself contaminated. Audit the pre-staged value periodically.
Citation
NIST CSF 2.0 RS.MI-01, RS.MI-02; ID.RA-09 (response dimension); GV.SC-07 (response dimension). NIST SP 800-218A. SLSA framework. OWASP ASI04. NIST AI RMF MANAGE 3.1.
Primary failure modes
Documented, not hypothetical. A control whose bypass is undocumented is worse than no control, because somebody trusted it.
- cached images on nodes keep running after registry deletion
- cosign deny by tag instead of digest
- DNS block at one level but not corporate resolver
- last-known-good SHA itself contaminated
Crosswalk
| NIST CSF 2 0 | RS.MI-01, RS.MI-02, ID.RA-09, GV.SC-07 |
|---|---|
| NIST AI RMF | MANAGE 3.1 |
| OWASP AGENTIC | ASI04 |
| OTHER | NIST SP 800-218A, SLSA framework |
Cite this cell:
https://agenticcovenants.com/respond/supply-chain/server-side/