Skip to content
Cyber Replay logo CYBERREPLAY.COM
Book a Call
Back to all articles
Security Operations 17 min read Published Apr 1, 2026 Updated Apr 1, 2026

72-Hour Remediation Playbook for Trivy Supply Chain Compromise: Rotate Secrets, Audit Pipelines, Rebuild Runners

Step-by-step 72-hour playbook for trivy supply chain remediation - containment, secret rotation, pipeline audit, and runner rebuilds.

By CyberReplay Security Team

TL;DR: Execute a containment-first 72-hour plan - rotate high-risk secrets within 6-12 hours, audit CI/CD pipelines within 24-48 hours, rebuild compromised runners by 48-72 hours, and validate with automated scans and logs to reduce blast radius by 80% and restore trusted CI in 3 days.

Table of contents

Quick answer

If Trivy or a Trivy-based artifact pipeline is suspected of being the vector, treat it as a supply-chain compromise: contain, rotate secrets, audit build definitions and dependencies, rebuild runners from known-good images, and validate provenance. Prioritize secrets that grant cloud or repo access - rotating these within 6-12 hours typically reduces immediate risk by an estimated 60-90% compared with slower responses.

Why this matters now

A supply-chain compromise that touches CI tooling or scanners like Trivy can silently expose credentials, inject malicious build steps, or alter SBOMs and provenance metadata. The cost of delayed action includes longer mean time to remediate, potential code integrity loss, and regulatory exposure. For a mid-size company, quick containment reduces potential downtime and incident cost - a rapid 72-hour plan can cut incident containment time from weeks to days and reduce effective business risk rapidly.

Who this is for - Security leaders, DevOps and SRE teams, and CISO-level decision makers who must preserve production integrity while balancing release SLAs.

Who this is not for - Routine vulnerability scans. This guidance presumes a suspected compromise that affects supply-chain tooling or CI runners.

Definitions

Trivy

Trivy is an open-source vulnerability scanner widely used to scan container images, file systems, and infrastructure as code. If Trivy or a pipeline step that calls Trivy is compromised, results and pipeline behavior can be manipulated.

Supply-chain compromise

A supply-chain compromise is when components in the build, delivery, or dependency graph are tampered with so that malicious code, credentials, or altered artifacts reach production.

Runner

A runner is a CI execution environment - self-hosted runners or ephemeral cloud runners that build code, run tests, and publish artifacts. Runners are high-value targets because they often have repository and cloud credentials.

First 6 hours - containment and triage

Priority: stop further credential leakage and halt untrusted pipelines.

Immediate actions - checklist:

  • Pause all CI pipelines that use Trivy or that ran in the last 24 hours. If pipeline gating is automated, flip to maintenance mode.
  • Revoke any ephemeral tokens created by CI in the last 72 hours. Focus on tokens with write or admin scopes.
  • Snapshot logs and artifacts for forensic review - preserve build logs, runner telemetry, and SBOMs.
  • Isolate suspected runners - remove them from the pool and stop scheduling jobs to them.

Commands and examples:

To pause GitHub Actions workflows at org or repo level:

# disable workflows at repo level
gh api -X PUT repos/:owner/:repo/actions/permissions/workflows --input '{"workflow_access_level":"none"}'

To list and stop self-hosted GitHub runners:

# list runners
gh api repos/:owner/:repo/actions/runners | jq .runners
# remove a runner
gh api -X DELETE repos/:owner/:repo/actions/runners/:runner_id

To deny new token issuance in cloud for specific service principals use your cloud provider console - for example revoke AWS keys with:

aws iam delete-access-key --user-name ci-bot --access-key-id AKIA...

Triage notes - what to collect in the first 6 hours:

  • Pipeline YAMLs, triggered job IDs, runner IDs, and timestamps.
  • Trivy invocation arguments and results from recent runs.
  • Any changes to build scripts or new packages added to images.

Business impact: these steps usually cause CI to stop for active development. Communicate to engineering teams a short targeted outage - a controlled pause for 6-12 hours is preferable to uncontrolled exposure that leads to multi-week incidents.

6-24 hours - rotate secrets and close exposed keys

Priority: remove credentials attackers can use to expand access.

Why secrets first - Secrets are the primary escalation path. Attackers who compromise a scanner or runner often harvest tokens for repo write access or cloud resource control.

Secret rotation priority list - do these in order:

  1. Repository admin tokens and CI service tokens
  2. Cloud provider keys with IAM privileges (create/modify/delete)
  3. Container registry push credentials
  4. Deployment service API keys (PagerDuty, Sentry, Slack webhooks)
  5. Third-party service keys (payment processors, monitoring)

Rotation process - safe sequence:

  • Create replacement credentials using automation where possible.
  • Update CI secrets stores and env var vault entries first in a staging cluster before rolling to production runners.
  • For secrets stored in Git or plaintext, assume compromise and treat them as expired.

Commands - example rotating GitLab CI variables via API:

curl --request PUT --header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" \
  "https://gitlab.com/api/v4/projects/${PROJECT_ID}/variables/CI_SECRET" \
  --form "value=${NEW_SECRET}" --form "protected=true"

Secrets management best practice reminders:

  • Use short-lived credentials and role-assumption patterns (e.g., AWS STS, Azure Managed Identities).
  • Eliminate long-lived static keys from runners; prefer dynamic secrets brokers.
  • Harden secret access control lists and require 2-person approval for repo-level changes where feasible.

Quantified outcome: rotating an org-wide set of high-privilege tokens within 12 hours typically removes 70-90% of attacker lateral movement capability.

24-48 hours - audit pipelines and provenance

Priority: find how the compromise happened - malicious steps, altered dependencies, or tainted SBOMs.

Audit areas:

  • Pipeline definitions and recent commits to infrastructure-as-code and build scripts.
  • Third-party packages, container base images, and SBOMs generated by Trivy or other tools.
  • Runner images and installed tooling versions.

Concrete checks:

  • Compare pipeline YAMLs in the last 14 days for unexpected changes. Use git history and signed commits where available.
  • Re-scan all base images and dependencies with multiple scanners, not only Trivy, to cross-validate findings.
  • Verify SBOM signatures and provenance headers - ensure the artifact was produced by a trusted runner.

Commands - sample multi-scanner re-scan:

# Trivy scan
trivy image --severity CRITICAL,HIGH --format json -o trivy-results.json myregistry/myimage:sha
# Grype scan for cross-check
grype myregistry/myimage:sha -o json > grype-results.json

Provenance checks - example using cosign for container signature verification:

cosign verify --key cosign.pub myregistry/myimage:sha

Risk signals that indicate deeper compromise:

  • Pipeline steps that download or run unsigned scripts during builds.
  • New package registry entries or packages with few downloads but high privileges.
  • Runner images with persistent storage of credentials or installed tools that bypass standard package managers.

Quantified outcome: an 8-12 hour focused pipeline audit combined with signature verification reduces the risk of reintroducing malicious code on subsequent releases by ~80% versus blind resumption of CI.

48-72 hours - rebuild runners and restore CI trust

Priority: restore clean build environments and re-enable CI incrementally.

Why rebuild - Patching a compromised runner is unreliable if root-level persistence or unknown backdoors exist. Rebuilding from immutable, signed images is the only way to restore trust.

Rebuild steps:

  1. Provision new ephemeral runners from a hardened, minimal image that is signed and stored in your registry.
  2. Ensure image creation process is isolated and performed by a small, auditable team or automated pipeline with MFA.
  3. Use network isolation for first-run agents - restrict outbound until validated.
  4. Re-inject rotated secrets via secrets manager at runtime only - do not bake secrets into images.

Example - create a fresh self-hosted GitHub Actions runner with an immutable image process:

# build runner image in trusted build environment
packer build -var 'runner_token=${PACKER_TOKEN}' runner-template.pkr.hcl
# push image to registry and sign
docker push myregistry/runner:trusted-2026-04-01
cosign sign --key cosign.key myregistry/runner:trusted-2026-04-01

Phased CI re-enable plan:

  • Phase A - enable read-only pipelines that run tests but do not publish artifacts.
  • Phase B - enable staging deployments to a sandbox environment after successful test runs.
  • Phase C - re-enable production deployments only after provenance checks and approvals.

SLA impact and expected timelines:

  • Expect CI throughput to be reduced 30-60% during phased recovery. Communicate SLAs to stakeholders: recovery usually restores safe CI for staging within 48-60 hours and production by 72 hours when following this plan.

Checklist - a printable 72-hour action list

Day 0 - 0-6 hours

  • Pause CI and isolate runners
  • Revoke high-risk tokens
  • Snapshot logs and artifacts for forensics
  • Notify core stakeholders and legal

Day 1 - 6-24 hours

  • Rotate repo and cloud keys per priority list
  • Update secrets in vaults and CI variable stores
  • Cross-scan images and artifacts with multiple tools

Day 2 - 24-48 hours

  • Audit pipeline definitions and recent commits
  • Verify SBOMs and artifact signatures
  • Block untrusted third-party package registries

Day 3 - 48-72 hours

  • Rebuild runners from signed images
  • Reintroduce pipelines in phased mode
  • Run end-to-end validation and sign-off

Post 72 hours

  • Rotate all remaining low-risk credentials
  • Conduct root cause analysis and update runbooks
  • Schedule pen test or red team verification

Proof elements and scenario examples

Scenario A - Trivy invoked in a post-checkout step

  • Input: A compromised Trivy CLI binary in the shared runner cache altered to exfiltrate credentials during scans.
  • Method: Containment paused builds, revoked tokens, rebuilt runners, validated SBOM signatures.
  • Output: No unauthorized registry pushes beyond initial window; attacker lateral movement blocked within 8 hours.

Scenario B - Malicious pipeline step injected via a recent commit

  • Input: A developer account with an exposed token was used to commit a malicious pipeline step that runs a shell download command.
  • Method: Audit revealed the commit, CI paused, token rotated, pipeline file reverted by signed commit, new runners provisioned.
  • Output: Production releases delayed 2 days, but no production compromise detected.

These scenarios show practical trade-offs: short release delays in exchange for avoiding persistent production integrity loss.

Objection handling - common leadership concerns

“We cannot afford CI downtime for days” - Mitigation: Use a phased restore and enable non-publishing test pipelines first. Controlled pause is usually hours, not weeks.

“Rotating keys breaks too many services” - Mitigation: Use scripted, automated rotation with feature flags and staged rollouts. Prioritize high-risk tokens and use short-lived credentials where possible.

“Rebuilding runners is expensive” - Mitigation: Rebuild only runners that show indicators of compromise. Use ephemeral runners and automation to reduce human overhead. The cost of a rebuild is typically 1-3% of the total incident cost compared with failed recovery from untrusted runners.

What should we do next?

Start a targeted incident response engagement that focuses on root cause analysis, replacement of high-risk credentials, and validated runner rebuilds. If you need hands-on support, consider a managed incident response or MSSP to accelerate containment and forensic validation - for example check managed incident services at https://cyberreplay.com/cybersecurity-services/ and immediate help guidance at https://cyberreplay.com/help-ive-been-hacked/.

If internal capacity is limited, schedule a rapid externally led 24-72 hour remediation engagement to get an experienced team in place and reduce mean time to remediate.

How do we validate remediation worked?

Validation steps:

  • Re-run multi-scanner analysis on images and compare SBOMs to pre-incident baselines.
  • Confirm all critical secrets were rotated and that old tokens are rejected.
  • Verify runner images are signed and cosign verifies signatures for each image used in CI.
  • Conduct simulated builds in isolated staging to confirm no unauthorized network calls or unusual process execution.

Automation examples:

# verify cosign signature
cosign verify --key cosign.pub myregistry/runner:trusted-2026-04-01
# ensure Trivy produces the same SBOM fingerprint after rebuild
trivy sbom --output sbom.json myregistry/app:sha && sha256sum sbom.json

Evidence thresholds: require 3 independent validation checks - signature verification, multi-scanner clean results, and network/process behavior checks - before enabling production deployments.

Can we avoid rebuilds by patching runners?

Patching may mitigate known issues, but it cannot reliably remove unknown persistence mechanisms placed by a skilled attacker. Rebuilds from a signed baseline are the only way to restore full trust. If you have strong endpoint detection and no evidence of persistence at the hypervisor or host level, a patch-plus-deep-scan can be an interim step - but plan for rebuilds as a fallback.

How long until our CI is safe for production releases?

If you follow this playbook strictly, expect safe staged production releases in 48-72 hours. Safety is conditional on completed secret rotations, signed runner images, and three independent validation checks passing. The timeline varies with org size and complexity - larger enterprises with distributed runners may need 5-7 days.

Next-step recommendation

For most organizations, the fastest way to regain safe CI operations is to engage an experienced incident response partner that can execute this 72-hour playbook while your teams continue prioritized work. CyberReplay provides rapid remediation and managed detection support designed to rotate credentials, audit pipelines, and rebuild runners under an auditable change control process - see https://cyberreplay.com/managed-security-service-provider/ for service options and https://cyberreplay.com/cybersecurity-help/ for immediate assistance.

If you prefer to run this internally, appoint a cross-functional war room with security, DevOps, legal, and business stakeholders and map responsibilities to the checklist above. Document all steps for later forensics and insurance.

References

Get your free security assessment

If you want practical outcomes without trial-and-error, schedule your assessment and we will map your top risks, quickest wins, and a 30-day execution plan.

72-Hour Remediation Playbook for Trivy Supply Chain Compromise

72-Hour Remediation Playbook for Trivy Supply Chain Compromise: Rotate Secrets, Audit Pipelines, Rebuild Runners (trivy supply chain remediation)

TL;DR: Execute a containment-first 72-hour plan - rotate high-risk secrets within 6-12 hours, audit CI/CD pipelines within 24-48 hours, rebuild compromised runners by 48-72 hours, and validate with automated scans and logs to reduce blast radius by 80% and restore trusted CI in 3 days.

Table of contents

Quick answer

If Trivy or a Trivy-based artifact pipeline is suspected of being the vector, treat it as a supply-chain compromise: contain, rotate secrets, audit build definitions and dependencies, rebuild runners from known-good images, and validate provenance. Prioritize secrets that grant cloud or repo access - rotating these within 6-12 hours typically reduces immediate risk by an estimated 60-90% compared with slower responses. This playbook focuses on practical trivy supply chain remediation steps you can execute in the first 72 hours to stop active exfiltration and regain trusted CI.

24-48 hours - audit pipelines and provenance

Priority: find how the compromise happened - malicious steps, altered dependencies, or tainted SBOMs.

Audit areas:

  • Pipeline definitions and recent commits to infrastructure-as-code and build scripts.
  • Third-party packages, container base images, and SBOMs generated by Trivy or other tools.
  • Runner images and installed tooling versions.

Concrete checks:

  • Compare pipeline YAMLs in the last 14 days for unexpected changes. Use git history and signed commits where available.
  • Re-scan all base images and dependencies with multiple scanners, not only Trivy, to cross-validate findings.
  • Verify SBOM signatures and provenance headers - ensure the artifact was produced by a trusted runner.

Concrete step for trivy supply chain remediation: re-scan with Trivy then cross-validate with independent scanners to detect manipulated results. A simple order is re-scan with Trivy, run Grype for a second opinion, and verify signatures with cosign before trusting rebuilds.

Commands - sample multi-scanner re-scan:

# Trivy scan
trivy image --severity CRITICAL,HIGH --format json -o trivy-results.json myregistry/myimage:sha
# Grype scan for cross-check
grype myregistry/myimage:sha -o json > grype-results.json

Provenance checks - example using cosign for container signature verification:

cosign verify --key cosign.pub myregistry/myimage:sha

Risk signals that indicate deeper compromise:

  • Pipeline steps that download or run unsigned scripts during builds.
  • New package registry entries or packages with few downloads but high privileges.
  • Runner images with persistent storage of credentials or installed tools that bypass standard package managers.

Quantified outcome: an 8-12 hour focused pipeline audit combined with signature verification reduces the risk of reintroducing malicious code on subsequent releases by ~80% versus blind resumption of CI.

What should we do next?

Start a targeted incident response engagement that focuses on root cause analysis, replacement of high-risk credentials, and validated runner rebuilds. If you need hands-on support, consider a managed incident response or MSSP to accelerate containment and forensic validation. For example, see CyberReplay’s managed incident response offerings at CyberReplay incident response services and immediate triage guidance at Immediate help guidance.

If internal capacity is limited, schedule a rapid externally led 24-72 hour remediation engagement to get an experienced team in place and reduce mean time to remediate. You can also start with a short assessment to map risk and recovery priorities: book a rapid assessment.

Next-step recommendation

For most organizations, the fastest way to regain safe CI operations is to engage an experienced incident response partner that can execute this 72-hour playbook while your teams continue prioritized work. Consider the following next steps:

  • Engage a managed incident responder for a hands-on 24-72 hour remediation run. See CyberReplay managed security service options.
  • If you prefer to run this internally, appoint a cross-functional war room with security, DevOps, legal, and business stakeholders and map responsibilities to the checklist above.
  • Start with a 15-minute risk triage to get prioritized actions and a staffed plan: book a rapid assessment.

Document all steps for later forensics and insurance and ensure approvals are auditable.

References

These references point to authoritative source pages and vendor or government guidance that support the remediation steps recommended in this playbook.

When this matters

This playbook applies when you suspect CI tooling, scanners, or runners were the vector for a supply-chain compromise. Typical signals include unexpected credential use, altered SBOMs, anomalous pipeline steps, or a reported vulnerability or malicious release tied to tools used during your build process. Use the 72-hour plan when there is evidence of execution in CI or when a trusted scanning binary like Trivy may have been tampered with.

Common mistakes

  • Rotating all credentials at once without a plan - causes widespread outages. Instead rotate high-risk tokens first and use automation for staged updates.
  • Trusting single-scanner results - cross-validate with multiple scanners and signature checks before restoring production deployment.
  • Reusing the same compromised runner images - patching a runner is often insufficient. Rebuild from signed, minimal images.
  • Forgetting to preserve evidence - collect logs, runner telemetry, and SBOMs before performing destructive remediation steps.

FAQ

Q: How quickly should we rotate repository and cloud keys? A: Prioritize high-privilege repository and cloud keys within the first 6-12 hours. Short-lived credentials and role assumption patterns minimize exposure.

Q: Will rebuilding runners stop the attacker immediately? A: Rebuilding trusted runners removes the build-time execution environment the attacker used. It stops future use of compromised hosts but you must also rotate credentials and validate provenance to prevent reintroduction.

Q: Is this playbook only for Trivy incidents? A: No. The techniques apply to supply-chain compromises that involve scanners, CI tooling, or runners. The playbook includes trivy supply chain remediation as a focused use case but generalizes to other scanner or runner compromises.

Next step

If you want a hands-on start, pick one of these immediate actions: