Gartner® Emerging Tech: Top 4 Security Risks of GenAI
In this report, Gartner offers insights and recommendations for security and product leaders to gain a competitive edge by addressing the key transformation opportunities presented by these risks.
Download Report
The Top 7 AI security risks
Understanding the risks at each stage of the AI development process allows organizations to build more secure AI systems by proactively implementing the proper security measures. Below, we explore the different types of AI risks along with potential attack scenarios and mitigation recommendations.
1. Limited testing
Why it matters:
AI models can behave in unexpected ways in production, which can adversely affect user experience and open up the system to a variety of known and unknown threats.
Real-life attack scenarios:
Malicious actors might manipulate the model’s behavior by subtly altering the input data (evasion attack) or by strategically positioning/manipulating data during model training (data poisoning attack).
Mitigation:
To test datasets, introduce a wide variety of real-world examples and adversarial examples.
Establish a comprehensive testing framework that encompasses unit tests, integration tests, penetration tests, and adversarial tests.
Advocate for adversarial training during model development to enhance model resilience against input manipulations.
State of AI in the Cloud 2024
Did you know that over 70% of organizations are using managed AI services in their cloud environments? That rivals the popularity of managed Kubernetes services, which we see in over 80% of organizations! See what else our research team uncovered about AI in their analysis of 150,000 cloud accounts.
Download PDF
2. Lack of explainability
Why it matters:
AI models can behave in ways that are hard to understand and justify. Limited visibility into AI logic minimizes testing capabilities, leading to reduced trust and increased risk of exploitation.
Real-life attack scenarios:
An attack could attempt to reverse engineer the AI model to gain unauthorized access (model inversion attack). An attacker could also manipulate input data directly (content manipulation attack) to compromise your model.
Mitigation:
Advocate for the use of interpretable models and techniques during model development.
Implement post hoc explainability techniques to analyze and interpret the decisions made by the AI model after deployment.
Establish clear, documented guidelines that AI developers can use as a reference point to maintain transparency.
3. Data breaches
Why they matter:
The exposure of sensitive data can harm customers and cause business disruptions. Furthermore, data breaches often lead to wide-reaching legal consequences resulting from regulatory non-compliance.
Real-life attack scenarios:
An attacker might try to detect if a specific individual’s data was used to train an AI model (membership inference attack). Cybercriminals may also attempt to deduce sensitive data by analyzing a model’s output (attribute inference attack).
Generative AI applications, especially when built on large language models (LLMs), are particularly sensitive to these types of attacks. That’s why it’s especially important to keep an eye on gen AI risks.
Mitigation:
Implement robust encryption for data at rest and in transit.
Ensure differential privacy techniques are applied during model development.
Regularly audit and monitor access to sensitive data, following the principle of least privilege.
Adhere to data protection regulations, such as GDPR.
4. Adversarial attacks
Why they matter:
Adversarial attacks compromise the integrity of the AI models, resulting in incorrect or unwanted outputs, which undermine system reliability and the overall security posture.
Real-life attack scenarios:
Threat actors could aim to exploit the model's sensitivity to changes in input features by manipulating gradients during the training process (gradient-based attack). Threat actors can also reduce the model’s resistance to attacks by manipulating input features (model evasion through input manipulation).
Indirect prompt injections have also been discovered presenting a rather severe security risk to LLMs, whereby malicious prompts are created in requested content and either result in the redirection or misdirection of LLM activities. These methods can also be potentially used to gather sensitive user details for data exfiltration, execute malicious content, compromise the LLM model output or implement redirection of an unsuspecting LLM-enabled chat user to a malicious content.
Mitigation:
Implement a routine for updating model parameters to fortify the model against attacks.
Employ ensemble methods to combine predictions from multiple models.
Conduct ethical hacking and penetration testing to proactively identify and address vulnerabilities in the AI system.
Establish continuous monitoring mechanisms to detect unusual patterns or deviations in model behavior.
5. Partial control over outputs
Why it matters:
Even with extensive testing and extended explainability, AI models can still return unexpected outputs that could be biased, unfair, or incorrect. Model developers only have partial control over outputs—and users can also intentionally or unintentionally prompt AI in irregular ways.
Real-life attack scenarios:
An attacker could aim to create hyper-realistic fake content using your AI model to spread misinformation (deep fakes), or a malicious actor may try to inject bias in your model via input manipulation (content-bias injection).
Mitigation:
Conduct bias audits on training data and model outputs using tools like Fairness Indicators.
Advocate for the implementation of bias-correction techniques, such as re-weighting or re-sampling, during model training.
Define and implement ethical internal guidelines for data collection and model development.
Promote transparency by sharing ethical guidelines for AI usage with users.
LLMs, combined with chat interfaces and software automation, enable attackers to:
-> Increase productivity — GenAI can help attackers create more attacks
-> Improve plausibility — GenAI applications can help discover and curate content from multiple sources to increase the trustworthiness of a lure and other fraudulent content (e.g., brand impersonation).
-> Enhance impersonation — GenAI can create more realistic human voices/video (deepfakes) that appear to be from a trusted source and could undermine identity verification and voice biometrics.
-> Introduce attack and malware polymorphism — Generative AI can be used to develop varied attacks, harder to detect than repacking polymorphism.
-> Enhance autonomy — LLMs can enable a higher level of autonomous local action decision or more automated command and control interactions, allowing malicious applications to operate an end-to-end attack life cycle until an attack goal is achieved.
-> Enhance future novel attack types — The worst possible security threat from GenAI would be the large-scale discovery of entirely new attack classes.
6. Supply chain risks
Why they matter:
AI is heavily based on open-source datasets, models, and pipeline tools for which security controls can only be partially implemented. Vulnerabilities exploited in the supply chain can not only compromise the AI system but also extend to other production components.
Real-life attack scenarios:
An attack could aim to tamper with/substitute model functionalities (model subversion) or attempt to introduce compromised datasets filled with adversarial data (tainted dataset injection).
Mitigation:
Vet and validate datasets, models, and third-party AI integrations to ensure their security and integrity.
Implement secure communication channels and encryption for data exchange in the supply chain.
Establish clear contracts and agreements with suppliers that explicitly define AI security standards and expectations.
7. Shadow AI
Why it matters:
The presence of unauthorized or unnoticed AI systems, commonly referred to as shadow AI, introduces undetectable vulnerabilities that don’t have corresponding mitigation strategies.
Real-life attack scenarios:
If an employee uses ChatGPT from their browser without adjusting privacy settings, sensitive or proprietary data could be used for model training by OpenAI. Employees may also use AI solutions that lack minimum security guarantees, introducing significant risks.
Mitigation:
Create standardized operations for AI support within your organization to streamline the deployment and monitoring of AI systems.
Institute protocols for swiftly responding to and addressing any unauthorized AI deployment.
Conduct comprehensive education and training programs to ensure personnel are well-informed about the safe and authorized use of AI.
Protecting your AI applications with Wiz
As a key part of our mission to help organizations create secure cloud environments that accelerate their businesses, Wiz is the first cloud native application protection platform (CNAPP) to introduce a native and fully-integrated AI security offering.
AI Security Posture Assessment Sample Report
Take a peek behind the curtain to see what insights you’ll gain from Wiz AI Security Posture Management (AI-SPM) capabilities. In this Sample Assessment Report, you’ll get a view inside Wiz AI-SPM including the types of AI risks AI-SPM detects.
Download Sample Report
AI security posture management (AI-SPM) solution offers you a variety of automated security functionalities, including:
Management of an AI bill of materials (AI-BOM): The AI-BOM gives you full visibility over every AI service, technology, library, and SKD in your environment. Use it to discover your AI pipelines and detect shadow AI as soon as it’s introduced.
Assessment of AI pipeline risk: By testing your AI pipelines against known vulnerabilities, exposures, and other risks, AI-SPM allows you to uncover attack paths to your AI services with a focus on pipeline misconfigurations and detection of instances where sensitive data is used in training sets.
Access to an AI security dashboard: Navigate your AI security posture through a dashboard that offers a consolidated view of security risks. Our dashboard provides a prioritized queue of contextualized risks for your AI pipelines, and it also lists vulnerabilities found in the most popular AI storage solutions and AI SDKs, such as OpenAI and Hugging Face.
Wiz’s innovative approach to security provides end-to-end protection for your hybrid IT infrastructure, including robust safeguards for your AI systems. You can learn more by visiting the Wiz for AI webpage. If you prefer a live demo, we would love to connect with you.
Learn what makes Wiz the platform to enable your cloud security operation
