What makes up the total cost of running Kubernetes?
Kubernetes clusters consist of the control plane, worker nodes, and cloud provider add-ons, with each of these billed separately by the cloud or vendor. However, your final bill for running Kubernetes will also include hidden costs, such as cluster expansion, SRE time, networking expenses, and cardinality growth in observability systems.
Kubernetes Security Best Practices [Cheat Sheet]
Apply advanced Kubernetes security techniques across clusters, workloads, and infrastructure with this guide.
Control plane and worker nodes
Your approach to cluster management will determine what you pay for a Kubernetes cluster’s control plane and worker nodes.
Managed Kubernetes-as-a-service (EKS/GKE/AKS)
If you take the managed route, your cloud provider will operate the control plane and lifecycle management components, while you retain responsibility for worker nodes—unless you select managed node groups.
Control plane costs in managed services (EKS charges ~$0.10/hour or ~$73/month per cluster; GKE and AKS include control plane costs in node pricing) remain relatively fixed. Your total bill variability comes primarily from worker node choices: instance families (general-purpose vs. compute-optimized), pricing models (Spot instances offer 60%-90% savings vs. On-Demand), and topology (multi-AZ deployments triple control plane costs but provide high availability).
Self-managed/on‑premises (including cloud IaaS)
Organizations may run self-managed Kubernetes on cloud IaaS (using tools like kubeadm or kOps on EC2, Compute Engine, or Azure VMs), where they operate both the control plane and worker nodes. This differs from managed services (EKS, GKE, AKS), where the cloud provider operates the control plane. Self-managed clusters require factoring in SRE personnel costs—typically 1-2 FTEs per 5-10 clusters for patching, upgrades, and incident response.
Note: With self-managed/on-premises, your Kubernetes cluster price model needs to include the actual SRE personnel costs.
Serverless flavors (Fargate, EKS Auto Mode, GKE Autopilot)
Pricing is based on pod resource time—vCPU-seconds and GB-seconds of memory consumed during execution. These platforms enforce strict pod constraints: DaemonSets are not supported (EKS Fargate, GKE Autopilot), privileged containers are blocked, and host-level access (hostNetwork, hostPath) is restricted to maintain the serverless security model.
Add-on costs to plan for
The adoption of Kubernetes leads to large-scale clusters and add-ons, which is where spend really piles up. Fees for storage, observability, CI/CD, and security features all tend to exceed control-plane expenses.
Storage
Organize storage volumes according to actual usage requirements; avoid creating excessive capacity for potential future needs.
Select the most affordable storage tier (SSD or HDD tiers) that fulfills your performance requirements, and check your monthly costs.
Enable lifecycle policies for logs, backups, and snapshots, and move cold data to object storage for storage optimization.
Automatically detect and clean up unused PersistentVolumeClaims (PVCs) and their underlying PersistentVolumes (PVs) when namespaces or Helm releases are removed.
Note: Helm does not delete PVCs by default; PV deletion/retention is controlled by the StorageClass reclaim policy (Delete vs. Retain).
Observability
Set metric label/cardinality budgets, and implement a scrape-time process to remove high-cardinality labels.
Use single-replica setups for monitoring development and testing environments, as they do not require high availability.
Implement standardized Prometheus relabeling and logging filters to prevent massive ingestion spikes.
CI/CD
Maintain cached base layers and language/package stores while using version pins to prevent complete rebuilds.
Limit build operations per repository and team to a specific number before implementing queuing rather than running them all at once.
Configure registry retention policies in Amazon ECR, Google Artifact Registry, or Azure Container Registry to automatically delete old images (e.g., keep last 10 tags per branch for 30 days). Also, enable layer pruning to remove unreferenced layers—the actual storage objects—since multiple tags can share layers. For example, a registry with 500 images might have 2,000 layers; pruning unreferenced layers can reduce storage by 40%-60%, cutting costs from $50/month to $20/month.
Security tooling
Kubernetes Security Posture Management (KSPM) and a Cloud‑Native Application Protection Platform (CNAPP) should be your first choices because they help minimize both breach impact and unexpected spend.
Initial security measures should focus on cost protection using egress policies, image signing, and least privilege access.
The system should begin scanning critical namespaces first and only expand its scope to other areas after demonstrating its value.
See Wiz Cloud in Action
In your 10 minute interactive guided tour, you will:
Get instant access to the Wiz platform walkthrough
Experience how Wiz prioritizes critical risks
See the remediation steps involved with specific examples
How scaling impacts your bill
Autoscalers right‑size capacity to current demand, which reduces waste from idle nodes; however, they do not guarantee lower spend on their own if pod requests/limits are oversized or the workload is inefficient.
To actually cut your bill, combine autoscaling with request/limit tuning, rightsizing, performance optimizations, and the use of cost‑visibility tools.
| Scaler | What it does | Cost risks |
|---|---|---|
| Horizontal Pod Autoscaler (HPA) | Scales replicas based on CPU/memory utilization by default; supports custom metrics (e.g., request latency, queue depth) via Metrics Server adapters | Can scale out pods when requests/limits are oversized, which may trigger node scale‑ups and mask code performance issues instead of fixing them |
| Vertical Pod Autoscaler (VPA) | Analyzes historical data to generate recommendations for CPU and memory settings; can either suggest or apply these automatically | Following VPA recommendations restarts pods and can reduce bin‑packing efficiency; ignoring recommendations leads to over‑provisioning and higher spend |
| Cluster Autoscaler/Karpenter | Adds/removes nodes just-in-time when pods can't schedule due to resource constraints, affinities, or taints; Cluster Autoscaler scales node groups, Karpenter provisions individual nodes | Creates additional nodes (additional cost) when requests are high; very large nodes concentrate many pods, so disruptions or scale‑downs can cause wide evictions and cascading reschedules |
| Serverless scaling | Manages capacity automatically; bills per pod based on vCPU-seconds and GB-seconds of memory during execution (not pod count) | Cold starts and image pulls add paid time where the pod is not yet doing useful work, which shows up as a separate billable duration |
Monitoring and cost allocation
Linking platform telemetry data from scheduling and autoscalers to finance data for namespaces and teams delivers the best foundation for Kubernetes cost monitoring and management.
Metrics to watch (daily)
Review these high‑signal metrics every day to identify waste and reliability problems at an early stage:
CPU and memory metrics to compare request amounts against actual usage levels
Reliability indicators like pod restarts, queue levels, and networking saturation levels
Scheduling metrics, e.g., pending pods and scheduling latency
Events to alert on (in real time)
The following conditions require immediate attention:
HPA thrash (scaling up/down every 1-2 minutes) or VPA recommendation changes >50% (e.g., 1 GB → 1.5 GB memory) that trigger pod evictions and restarts, causing service disruption
Cluster Autoscaler/Karpenter system entering continuous loops between unschedulable pods and a 'no matching provisioner' condition (no Karpenter provisioner configuration matches the pod's constraints)
High egress traffic volumes that lead to network address translation (NAT) connection exhaustion (running out of ports/flows on the NAT gateway, causing failed connections, retries, and potential egress‑cost spikes)
High numbers of OOMKilled and CrashLoopBackOff errors
Allocation & tagging
The first step to achieving cost accountability is establishing clear ownership and standardized tagging systems. All namespaces and workloads need to receive owner labels that include team, service, and environment designations.
Your environment should also export data to OpenCost or Kubecost to attribute actual dollars to Kubernetes owners (team/namespace/service) and resources (pods, deployments, labels). These tools use K8s metrics to show who spent what, provide spending dashboards, and enable weekly reviews with each team.
How Wiz helps
The Wiz Security Graph correlates cost hotspots with security risk context and ownership. It then ranks issues by savings and risk reduction and then routes this data to the relevant team, so waste is removed without disrupting essential workflows.
Kubernetes cost management in the AI/GPU era
GPUs, which power machine learning training and inference, are billed at a high per‑hour rate. A single job that over‑requests GPUs, runs for long durations, or sits under‑utilized can rapidly accumulate large costs—erasing months of prior savings.
Amazon EKS now allows clusters to reach 100,000 worker nodes. That means, keeping your spend under control is all about governance.
Today, GPU access is often unregulated. The fix is to regulate it with a shared framework—namespace/team‑level quotas, limits, and admission policies—that controls who can request GPUs, how many, and which types. This doesn’t eliminate the need for ongoing governance; however, it does prevent sprawl and surprise costs by enforcing caps and approvals throughout the cluster.
GPU cost patterns to watch
You need to be on the lookout for hidden costs that end up inflating your bill!
Keep an eye on the following usage and architectural patterns:
Underutilized accelerators: The majority of accelerators operate at low utilization rates because jobs allocate full GPUs for tasks that only need 20% to 40% of the available resources. Time‑slicing using multi-instance GPU (MIG), node sharing, and queue‑based schedulers should be used instead of job reservations for GPU resources.
Data‑gravity taxes: The cost of data transfer between different AZs and regions becomes the main expense factor for training ops. Deploying data storage near computing resources and object-store caching functions is an essential optimization step.
Preemption without checkpoints: The lack of checkpoint functionality in spot instances leads to complete job loss when preemption occurs (i.e., when the cloud provider reclaims a node and stops your job without notice). If you lose your work progress, you can wave goodbye to any initial cost savings gained. So if you use spot instances, make sure to run checkpoints at regular intervals.
Oversized images: Deployment time and egress can be slashed through pre-pull and slim layers when ML image sizes exceed 10 GB.
The following example training job requests one GPU with explicit CPU/memory, tolerates Spot nodes, and mounts a checkpoint PVC so preemptions don’t lose progress. Adjust the node selector or MIG profile to match your cluster:
apiVersion: batch/v1
kind: Job
metadata:
name: mnist-train-ckpt
spec:
template:
spec:
restartPolicy: OnFailure
nodeSelector:
node.kubernetes.io/instance-type: g5.xlarge # choose a cost‑effective GPU node
tolerations:
- key: "k8s.amazonaws.com/spot"
operator: "Exists"
effect: "NoSchedule"
containers:
- name: trainer
image: ghcr.io/acme/ml-train:sha-abc123 # keep image slim; avoid >10 GB
imagePullPolicy: IfNotPresent
resources:
requests:
cpu: "2"
memory: "8Gi"
nvidia.com/gpu: "1" # or nvidia.com/mig-1g.5gb on MIG clusters
limits:
cpu: "2"
memory: "8Gi"
nvidia.com/gpu: "1"
env:
- name: CHECKPOINT_DIR
value: /chkpt
volumeMounts:
- name: chkpt
mountPath: /chkpt
volumes:
- name: chkpt
persistentVolumeClaim:
claimName: training-checkpoints
Cost anomalies & security: Two sides of the same coin
Unexpected expenses often signal security incidents—cryptomining, data exfiltration, or misconfigured autoscalers. Treat cost spikes as incident signals in your IR workflow: When spend jumps >30% day-over-day, open an IR ticket with owner/namespace context and investigate for abuse or misconfiguration.
Unified anomaly-to-owner context: Platforms that correlate spend spikes with resource ownership, exposure status, and vulnerability context enable security and platform teams to triage cost vs. incident signals together. For example, a $2,000 spike in a namespace with an internet-exposed pod running an outdated image suggests cryptomining, not legitimate load.
Common anomalies
The following list includes common system anomalies that can negatively impact both system security and operational costs:
Exposed control planes: Attackers use exposed control planes and dashboards to launch cryptomining operations that scale up. They then use cluster scheduling to deploy miners for their operations.
Data‑transfer shocks: The absence of egress restrictions leads to unexpected data transfer expenses, i.e., sudden spikes in costs for cross-region and NAT traffic.
Image bloat: The growth of images in storage causes both longer pull times and higher storage costs. And if you suffer from both large base images and cache misses, you’ll end up with deployment delays and higher registry expenses.
Advanced Container Security Best Practices [Cheat Sheet]
Strengthen container security across build, deploy, and runtime stages using battle-tested techniques.
Policy management as a cost & security lever
With admission policies (OPA Gatekeeper, Kyverno), enforce ownership tags, resource limits, instance type restrictions, and GPU class allowlists. Start in audit mode to report violations without blocking, then switch to enforce mode after validating rules.
Single policy engine across stages: Use one set of guardrails from CI/CD (pre-commit hooks, pipeline gates) to cluster admission (runtime enforcement) to ensure consistency. For example, the same policy that blocks >2 GB container images in CI also blocks them at pod creation, preventing expensive image pulls and registry bloat across the entire software delivery lifecycle.
Guardrails to implement
Use the following guardrails to lower your bill:
All namespaces and workloads need to display owner labels that show team, service, and environment information.
All requests and limits must be mandatory, and namespace CPU and memory quotas must be enforced.
The system should only allow GPU access through approved namespaces and deny access to all other namespaces by default.
Application pods in user namespaces must avoid privileged mode and hostNetwork access while using signed images (e.g., Cosign verification). Exempt only essential system components (CNI plugins, node monitoring agents, CSI drivers) deployed in system namespaces like kube-system, with explicit approval and documentation.
The system should enforce an image‑size policy in CI/CD; if the built container image is larger than 2 GB, the CI job fails the build and blocks the push/deploy.
What is Cloud Cost Management?
Cloud cost management, also known as cloud spend management, is the process of monitoring, controlling, and optimizing cloud spend across an organization’s cloud environments.
Read more
Wiz: Security, context, and cost analysis in one platform
Now you know how to calculate the complete cost of a Kubernetes cluster from start to finish. You’ve seen how HPA/VPA/Karpenter and serverless technologies affect K8s spend. You can even set up a Kubernetes cost monitoring system with allocation capabilities that use policy-based guardrails to protect GPU resources.
But effectively managing Kubernetes costs requires different teams to work together using monitoring systems, proper resource allocation, and proactive measures to prevent unnecessary expenses.
Wiz correlates spend with risks and owners to surface secure savings opportunities. This is how Wiz fits into your Kubernetes cost management solutions stack:
Agentless multi-cloud discovery of clusters, nodes, namespaces, images, and cloud assets within minutes
KSPM and CNAPP to determine the most critical security fixes through code change analysis with runtime data
Security Graph for performing a query for "expensive & exposed & critical CVEs" to produce a list of secure savings options
Identification of abuse activities, e.g., cryptomining operations, and security recommendations that follow the principle of least privilege
Wiz helps companies cut waste by identifying version drift and EKS extended-support exposure. It also enriches OpenCost/Kubecost cost data with owner/team/namespace metadata to attribute and track savings across clusters and accounts over time.
Ready to cut spend without adding risk? Request a demo of Wiz to see secure savings opportunities mapped to your owners and services—prioritized by dollar impact and risk reduction.
See Wiz in action
See how Wiz connects the dots between misconfigurations, identities, data exposure, and vulnerabilities—across all environments, including Kubernetes. No agents, just full context.
