Agentic Covenants

Respond (RS) · Supply chain

Supply chain at the server side layer

external · Outside the agent entirely

How do I stop the bleeding now?

What this cell does

Remove poisoned image from registry, deploy emergency Kyverno deny rule on signature, block compromised registry/MCP domain at network/DNS, force redeploy with last-known-good image SHA pinned.

Artifacts (3)

agent-quarantine-supply-chain-serverview on GitHub
#!/usr/bin/env bash
# ABOUTME: Server-side supply-chain quarantine runbook. crane delete, Kyverno deny on digest, Cilium FQDN deny, force redeploy.
# ABOUTME: Pre-stage cilium-deny-suspect-fqdns.yaml and last-known-good-image-sha.txt under /etc/agents/emergency/.

set -euo pipefail

if [[ $# -lt 2 ]]; then
  echo "Usage: agent-quarantine-supply-chain-server <AGENT_NAME> <SUSPECT_IMAGE>" >&2
  echo "  SUSPECT_IMAGE format: registry/org/name@sha256:DIGEST" >&2
  exit 64
fi

AGENT_NAME="$1"
SUSPECT_IMAGE="$2"
NAMESPACE="agent-${AGENT_NAME}"
INCIDENT_ID="$(uuidgen 2>/dev/null || python3 -c 'import uuid; print(uuid.uuid4())')"
EMERGENCY_DIR="${EMERGENCY_DIR:-/etc/agents/emergency}"

if [[ ! -r "$EMERGENCY_DIR/last-known-good-image-sha.txt" ]]; then
  echo "REFUSING: $EMERGENCY_DIR/last-known-good-image-sha.txt not pre-staged" >&2
  exit 1
fi

# Extract digest portion of the suspect image. Used for the Kyverno deny rule
# so an attacker re-tagging the same digest does not bypass.
DIGEST="${SUSPECT_IMAGE##*@}"
if [[ "$DIGEST" == "$SUSPECT_IMAGE" || ! "$DIGEST" =~ ^sha256:[0-9a-f]+$ ]]; then
  echo "REFUSING: SUSPECT_IMAGE must be a digest reference (registry/name@sha256:...)" >&2
  exit 64
fi

# 1. Remove poisoned image from registry. May fail if registry permissions
# do not allow delete; warn rather than abort so the rest of the runbook runs.
if command -v crane >/dev/null 2>&1; then
  crane delete "$SUSPECT_IMAGE" 2>/dev/null || \
    echo "WARN: image deletion failed; manual registry intervention may be needed" >&2
else
  echo "WARN: crane not installed; skipping registry deletion" >&2
fi

# 2. Apply emergency Kyverno deny rule on the suspect digest.
SAFE_INCIDENT="${INCIDENT_ID//-/}"
kubectl apply -f - <<EOF
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: emergency-deny-image-${SAFE_INCIDENT:0:16}
  annotations:
    incident: "true"
    policies.kyverno.io/severity: critical
spec:
  validationFailureAction: Enforce
  background: false
  rules:
  - name: deny-suspect-digest
    match:
      any:
      - resources:
          kinds: [Pod]
    validate:
      message: "Image quarantined under incident ${INCIDENT_ID}"
      pattern:
        spec:
          containers:
          - image: "!*${DIGEST}*"
EOF

# 3. Block egress to compromised MCP domain. The pre-staged CNP must list the
# domain. For dynamic blocking, edit the pre-staged file before applying.
if [[ -r "$EMERGENCY_DIR/cilium-deny-suspect-fqdns.yaml" ]]; then
  kubectl apply -f "$EMERGENCY_DIR/cilium-deny-suspect-fqdns.yaml" 2>/dev/null || \
    echo "WARN: Cilium policy apply failed (Cilium may not be the CNI here)" >&2
fi

# 4. Force redeploy workloads with last-known-good image SHA pinned.
LAST_GOOD_SHA="$(tr -d '[:space:]' < "$EMERGENCY_DIR/last-known-good-image-sha.txt")"
LAST_GOOD_IMAGE="${SUSPECT_IMAGE%%@*}@$LAST_GOOD_SHA"

kubectl set image -n "$NAMESPACE" \
  deployment/claude-code \
  "claude=$LAST_GOOD_IMAGE" 2>/dev/null || \
  echo "WARN: deployment image update failed (deployment may be named differently)" >&2

kubectl rollout restart -n "$NAMESPACE" deployment/claude-code 2>/dev/null || true

logger -t agent-incident -p user.warning \
  "$(jq -n \
      --arg event "supply_chain_quarantined_server" \
      --arg agent "$AGENT_NAME" \
      --arg suspect "$SUSPECT_IMAGE" \
      --arg digest "$DIGEST" \
      --arg incident "$INCIDENT_ID" \
      --arg actor "$(whoami)" \
      --arg ts "$(date -Iseconds)" \
      '{event:$event, agent:$agent, suspect:$suspect, digest:$digest, incident:$incident, actor:$actor, ts:$ts}')"

echo "Server-side supply chain quarantined for $AGENT_NAME (suspect: $SUSPECT_IMAGE, incident: $INCIDENT_ID)"
cilium-deny-suspect-fqdns.yamlview on GitHub
# ABOUTME: Pre-staged CiliumNetworkPolicy denying egress to known-suspect FQDNs. Edit before applying or pre-populate at incident start.
# ABOUTME: Requires Cilium with enable-l7-proxy: true. The toFQDNs list is the deny mechanism — DNS resolution to these names returns NXDOMAIN.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: emergency-deny-suspect-fqdns
  namespace: kube-system
  annotations:
    incident: "true"
spec:
  endpointSelector: {}
  egressDeny:
  - toFQDNs:
    # Substitute these entries with the known-malicious FQDNs from the
    # incident. The runbook expects the file to be edited (or the list to
    # be maintained as a known-bad inventory) before apply.
    - matchName: "suspect-mcp.example.com"
    - matchName: "evil-registry.example.com"
    - matchPattern: "*.attacker-tld.example"
last-known-good-image-sha.txtview on GitHub

55 bytes. Too large to inline; open it on GitHub.

Cell notes

Interventions, Supply chain / Server-side

Trigger. Cosign verification failure, SBOM diff with unauthorized package, image registry pull of unsigned image, FQDN egress denial spike to a suspect MCP domain.

Authority. On-call plus security review (registry-level changes affect other workloads).

Speed target. Under 5 minutes.

Tooling

  • - crane (or registry-specific CLI) for image deletion.
  • - kubectl with permission to apply emergency ClusterPolicies and CiliumNetworkPolicies.
  • - Cilium with Hubble for FQDN-deny logging.

Files in this directory

  • - agent-quarantine-supply-chain-server, runbook script. Removes the poisoned image via crane delete, deploys an emergency Kyverno deny rule on the bad digest, applies the pre-staged Cilium FQDN deny policy, force-rolls workloads to the last-known-good image SHA.
  • - cilium-deny-suspect-fqdns.yaml, pre-staged CiliumNetworkPolicy denying egress to a list of suspect FQDNs. Substitute the matchName entries during the incident or pre-stage with a known-malicious list. Pre-stage at /etc/agents/emergency/cilium-deny-suspect-fqdns.yaml.
  • - last-known-good-image-sha.txt, pre-staged digest of the last-known-good agent image. Audit periodically (the pre-staged value should not itself be contaminated). Pre-stage at /etc/agents/emergency/last-known-good-image-sha.txt.

Verification


# 1. Image removed from registry
crane manifest "$SUSPECT_IMAGE" 2>&1 | grep -i "not found" || echo "FAIL: image still in registry"

# 2. Kyverno deny in effect
kubectl run test --image="$SUSPECT_IMAGE" -n agent-claude-code-prod 2>&1 | grep -i "quarantined" || echo "FAIL: deny did not fire"

# 3. Egress to suspect domain blocked
kubectl exec -n agent-claude-code-prod $(kubectl get pods -n agent-claude-code-prod -o name | head -1) -- \
  curl -sS --max-time 3 https://suspect-mcp.example.com 2>&1 | grep -iE "couldn.t resolve|connection refused"

# 4. Workloads on last-known-good image
kubectl get deployment -n agent-claude-code-prod claude-code -o jsonpath='{.spec.template.spec.containers[0].image}'
# expected: matches last-known-good

Common mistakes

  • - Image deletion does not affect already-pulled cached images on nodes. The force-rollout step is mandatory.
  • - Cosign deny rule by tag instead of digest, attacker pushes a new tag pointing at the same digest.
  • - DNS block applied at one level (cluster CNP) but not at corporate DNS, agents on operator hosts still resolve.
  • - Last-known-good SHA file is stale or itself contaminated. Audit the pre-staged value periodically.

Citation

NIST CSF 2.0 RS.MI-01, RS.MI-02; ID.RA-09 (response dimension); GV.SC-07 (response dimension). NIST SP 800-218A. SLSA framework. OWASP ASI04. NIST AI RMF MANAGE 3.1.

Primary failure modes

Documented, not hypothetical. A control whose bypass is undocumented is worse than no control, because somebody trusted it.

  • cached images on nodes keep running after registry deletion
  • cosign deny by tag instead of digest
  • DNS block at one level but not corporate resolver
  • last-known-good SHA itself contaminated

Crosswalk

NIST CSF 2 0RS.MI-01, RS.MI-02, ID.RA-09, GV.SC-07
NIST AI RMFMANAGE 3.1
OWASP AGENTICASI04
OTHERNIST SP 800-218A, SLSA framework