Thank you for joining our BlackHat session “The Snowball Effect - A Practical Playbook for Supply Chain Incident Response”!
Don’t wait for the next supply chain attack to hit your organization, follow these recommendations to tighten security and enhance visibility. You can also find useful the investigation playbook, summarizing the steps demonstrated in the session and based on our experience so far.
In case you are experiencing an incident, don’t hesitate to contact Wiz CIRT.
Recommendations
Configure Logging into a CDR
Stream audit logs for real-time monitoring and wider coverage.
Turn on and stream access logs to cloud secret vaults to a CDR where available.
Centralize all log sources in one place to get cross-platform detections and better context when investigating.
Manage Packages
Lock dependencies to prevent automated upgrades.
Pin package version for package distribution systems such as NPM and PyPI.
Pin third-party actions to a specific commit SHAs to block tag hijacking.
Fork and self-manage a copy of open-source projects.
Enforce package cooldown to lower risk of infection.
Use package stores with personalized configurations to enforce the desired restrictions on developers endpoints as well.
Limit Blast Radius
Enforce secret scanning of committed code and block pull requests that contain secrets.
Manage and limit usage of credentials in environment variables.
Centralize all secrets in secure vaults.
Harden GitHub Actions Secrets security - enforce environments for sensitive and production secrets.
A Little About GitHub Actions Environments
In the GitHub Actions ecosystem, environments are designed to manage deployment needs and can provide an extra layer of security to your workflows. Environments include deployment protection rules, and you can scope secrets to specific environments so they are protected by these same rules.
While these are commonly used in open-source projects to protect deployment secrets from external contributors, we also recommend them for internal and private projects. This helps isolate sensitive secrets, scoping them to environments that limit access and require further approval.
For example, a package manager token is typically only needed when publishing a new release. By configuring this token as an environment secret, you restrict its access to manually approved deployment pipelines, keeping it safe from unauthorized workflow runs.
We recommend reviewing the secrets used in your organization's workflows, and for production or cloud secrets which often do not require access from all branches or PRs, enforce environment restrictions.
Investigation Playbook
Disclaimer: the investigative steps outlined in this playbook are specifically derived from "The Snowball Effect" lecture. Please note that this is not an exhaustive list of all possible artifacts or techniques to search for during a supply chain attack. Rather, it serves as a focused guide highlighting the specific methodologies, real-world examples, and prioritized steps presented in the session.
Phase 1: Indicators of Compromise
Example anomalies to watch for in GitHub Audit Logs:
Anomalous Cloning: Look for mass cloning activity in your audit logs that deviates from a user's baseline behavior.
Suspicious PAT Usage: Monitor for unexpected activity originating from Personal Access Tokens (PATs).
Workflow Irregularities: Notice workflows executed for the very first time or workflow runs that are suddenly deleted.
Branch Protection Overrides: Check for
protected_branch.*events in GitHub audit logs to flag unauthorized code pushes.IP Discrepancies: Compare the IP addresses used for regular development or workflow activity against the IPs used for mass cloning (which are often VPN IPs).
Anomalous Runtime: Look for runtime anomalies that deviate from the usual process tree related to code package installation on action runners / workstations / cloud resources.
Phase 2: Immediate Containment of a GitHub User
If a GitHub user is identified as compromised, take these steps immediately:
Deauthorize the affected user from the organization.
Ask the user to revoke all PATs and SSH/GPG keys.
Validate the revocations by checking your audit logs (if using Enterprise Managed Users) or ask the user for the user’s audit logs.
Investigate the unauthorized activity of the user’s PAT.
If the compromise originated from a user’s machine, scan the machine for other hidden secrets and check for persistence mechanisms.
Finally, format the compromised workstation completely.
Phase 3: Code Investigation
When relevant:
Query the SBOM: Use Software Bill of Materials (SBOM) to search for usage of compromised packages across all resources.
Hunt for Malicious Commits: Search the codebase for relevant malware IOCs on every branch and across every repository.
Analyze Workflows: Review workflow code by using the
head_shavalue from theworkflows.*audit logs.Analyze Suspicious Push to Main: Review commits by using the
aftervalue from theprotected_branch.*audit logs.Identify Additional Affected Users: Map out which other users or endpoints might have been affected by the malicious code by checking
git.clonelogs for the infected repositories.
Phase 4: Map Blast Radius
Identify potential lateral movement and map blast radius of the attack by:
Determine if plain-text secrets or GitHub Action Secrets were exposed during the attack.
Map usage of the compromised secrets and widen the search for other accessed secrets.
Maintain strict documentation of the leaked secrets with searchable identifiers (e.g., Access Key IDs and hashed value of GitHub tokens).
Immediately rotate all exposed secrets, especially sensitive distribution system secrets and high privilege cloud credentials.
Search your cloud provider logs for unauthorized activity using those compromised searchable identifiers.
Additional Resources
[GitHub docs] Deployments and environments: Environment secrets
[GitHub docs] GitHub Events REST API
[Wiz Blog] Code to Cloud Attacks: From Github PAT to Cloud Control Plane
[fwd:cloudsec lecture] Not So Secret: The Hidden Risks of GitHub Actions Secrets
