Public cloud storage buckets remain one of the most persistent attack surfaces in cloud security. Bucket names are globally guessable, misconfigurations are common, and aggregators index hundreds of thousands of open buckets daily. Once a bucket is exposed, the data inside it is one URL away from exfiltration.
Data security posture management (DSPM) is core to how Wiz protects customer sensitive data. But knowing that a file “contains names” is not enough. Security teams need to know what the actual file is, what kind of data it holds, who it belongs to, and how severe the exposure actually is. They need context, not just detection.
Here’s how we built that, an AI-powered data classification system that started as a public bucket scanner and became a context engine running across every data surface Wiz covers from IaaS to SaaS.
What Traditional Scanners Miss
Traditional data scanners rely on predefined regex patterns and rule engines. They work for structured, well-known formats: a CSV with a column named ssn, a .env file with AWS_SECRET_ACCESS_KEY. But real-world cloud storage is messy. Files are exported from dozens of systems, named inconsistently, stored in unexpected formats, and written in multiple languages.
Scanned documents make it worse. PDFs of passport photos, ID cards, and handwritten forms routinely defeat OCR, and the ones that don’t produce noisy text, that regex will happily match and provide some inaccurate results.
Rigid patterns also cut the other way. They fire on test data, placeholder values, and templates, which erodes trust. A traditional scanner tells you a file contains an email address. It can’t tell you that the file is a customer export from a production e-commerce system, that it contains 10,000 identity profiles combining names with government IDs and financial data, that it is almost certainly real, and that the exposure is critical.
Starting With a Concept
We built an AI-based prototype composed of multiple specialized sub-agents, each responsible for a different stage of the analysis pipeline and paired with the model tier best suited for the job. Lightweight models handle high-volume triage and file selection. Larger models handle deep content analysis where nuanced reasoning is critical. The agents communicate findings between stages, building a progressively richer understanding of the bucket and its contents.
A key design choice was making the pipeline iterative:
The system gets smarter as it goes. If the analysis stage discovers that customer IDs follow a specific format, or that the bucket contains documents in a particular language, those learned patterns feed back to discovery, which finds additional files matching the newly understood schema.
We ran the concept without constraints: no cost ceilings, no structured output, no runtime limits. It was slow and expensive. But it surfaced sensitive data that our existing scanners missed entirely: identity verification documents in a vacation rental platform, customer databases under non-obvious filenames, credential files buried in nested directory structures. It proved that contextual reasoning could catch what regex alone could not.
Taking It to Production
Making the concept production-ready introduced a different set of challenges. Cost matters, runtime matters, and every output must feed into the systems that security teams rely on daily.
Full Visibility, Not Just Alerts
Production required a structured output that gives complete context on every file analyzed. For each file, the system produces: what the file is, what its purpose appears to be, the data categories it contains (PII, PHI, credentials, financial, and more) broken down into granular sub-categories and specific attributes, a sensitivity classification, severity assessment, and anonymized evidence. Severity is inferred from the full context of the scan: content signals, file metadata, data volume and recency, and an internal verification step that identifies synthetic data, test data, and sandbox artifacts. This is what allows the system to distinguish a real customer database from a staging fixture with identical field names.
Security teams get a complete picture of every finding. This output feeds directly into the Wiz Security Graph, triggers alerts, and populates dashboards.
Optimizing for Scale
We applied optimizations across the pipeline: matching each stage to the minimum model capability required, batching work to reduce overhead, tracking resource consumption and halting gracefully when budgets are reached, and compressing context between stages. Deterministic logic was moved out of the AI layer entirely, reserving AI for tasks where contextual understanding is genuinely needed.
The production system runs in an isolated sandbox, with purpose-built tools for handling the variety of file formats in real-world cloud storage: archives, compressed files, virtual disk images, PDFs, and various text encodings, including nested files to a configurable depth.
Through continuous iteration, we accumulated extensive pattern libraries: hundreds of path-based patterns for file discovery, content-detection patterns covering national identifier formats from dozens of countries, and a large library of false-positive indicators that prevent the system from reporting noise.
Purpose-Built Orchestration
Even with these optimizations, the initial architecture carried overhead from its general-purpose foundations. We rebuilt the orchestration layer from scratch for our specific workflow. One innovation was algorithmically grouping structurally similar files and selecting representative samples from each group, ensuring broad coverage with minimal redundancy even in buckets containing tens of thousands of files. Candidate selection also incorporates additional signals, including file classification, clustering, and context from Wiz’s classification engine, allowing the system to prioritize the most informative files while preserving broad coverage. Together, these optimizations reduced both cost and runtime substantially without compromising detection quality and breadth.
The Feedback Loop That Made It Work
Building the system was half the challenge. Making it accurate required relentless iteration.
We deployed in preview mode with access to the original files, letting us verify every finding against ground truth. Each review cycle fed improvements back at multiple levels: new detection patterns for regional formats we hadn’t anticipated, tuning of how signals are weighed when assessing severity, and calibration of confidence thresholds.
The result: false-positive rates dropped to near zero while recall remained high across diverse bucket types, data categories, cloud providers, and languages. This continuous feedback loop is what separates a good scanner from one that security teams actually trust.
What We Saw in One Week
To pressure-test the system at scale, we continuously analyzed publicly accessible cloud storage over the course of a single week. It produced more than 2,700 sensitive data findings across hundreds of unique public buckets, spanning nearly 2,000 unique files. The exposed data ranged from internal business documents and customer exports to credentials, financial records, and identity documents.
The findings that mattered most weren’t driven by a single sensitive value. They were the ones where the engine correlated related files across a bucket (a schema definition, an export, and a credential file two directories over) and figured out what the collection was. It required understanding what the data was, how it is related, and why it mattered.
From Scanner to Context Engine
As the system matured, we realized what we had built was not a scanner. It was a context engine. Given any file, from any source, it answers: what is this file, what sensitive data does it contain, how should it be classified, and how confident are we?
That capability turned out to be far more broadly applicable than public buckets. The same engine now powers data classification across Wiz:
Public and private cloud storage. The original use case, now extended to authenticated environments.
Cloud drives. Shared drives and collaborative storage surfaces.
Endpoint data. Data-at-rest classification on endpoints.
Attack simulation. When Wiz’s Red Agent identifies a data exposure during a simulated attack path, it passes the content to the classification engine for structured, high-quality data findings.
The engine is context-aware but context-agnostic: the same core logic serves every use case, with only the input framing adjusted. Every improvement benefits all contexts simultaneously. Every new deployment surface generates patterns and edge cases that make the engine stronger everywhere else.
What We Learned
Start unconstrained, then optimize. The concept was expensive and slow. It was also essential. Running without constraints showed us what the system could find, and those discoveries defined the production requirements.
Context beats patterns. The biggest improvement in detection quality came from giving the system enough context to reason about what it was looking at, not just match regexes against content.
False-positive prevention earns trust. We invested as much effort in preventing false positives as in detecting sensitive data. A scanner that cries wolf loses its audience fast.
The feedback loop is the product. Accuracy after months of continuous feedback was dramatically better than at launch. The ability to verify findings, track metrics, and feed improvements back into the system is the core differentiator.
Build for reuse early. By structuring the output as a well-defined schema, we created something naturally portable. When other Wiz teams needed data classification, the integration was straightforward.
What’s Next
Data classification is a foundational capability for DSPM, and we’re extending the engine to additional contexts. What started as “can we find sensitive files in public buckets?” became something broader: a system that doesn’t just detect data, but understands it.
Discover how Wiz discovers, classifies, and contextualizes sensitive data risks across IaaS, SaaS, and virtual drives. Request a demo or explore Wiz DSPM.
