In the first part of our series, we explained why the introduction and use of cloud-native environments is a challenge for today's security tools. That's why solutions such as eBPF have emerged and become standards for addressing these issues. 

In this second part, we use eBPF's capabilities to demonstrate why this solution is so effective and how it can help us answer the following questions: 

• Was an attempt to escape from a container performed by a malicious actor?  

• Was a process executed with non-standard socket input, representing a Remote Code Execution (RCE) attack? 

• Was there an attempt to access `/etc/shadow`? 

Let's delve deeper into the topic of containers by examining their interaction with the host kernel. Containers communicate directly with the host kernel, but the layers of LSMs, capabilities, and namespaces prevent them from having complete access to the host kernel. On the other hand, VMs utilize a guest kernel (a dedicated kernel running in a hypervisor) instead of sharing a kernel. This implies that if the guest kernel of a VM is compromised, more effort is needed to escape from the hypervisor and gain access to the host. 

 The eBPF technology enables the safe execution of programs within the Linux kernel through sandboxed programs. To use eBPF, one needs the Linux kernel version 3.18 or higher and can refer to the bpf(2) syscall and man pages for more information. eBPF functions as a virtual machine with a 64-bit RISC instruction set. With eBPF, you can connect to kernel code paths and capture real-time security observability events without causing any disruption to the application. These events can be forwarded to userspace and saved for future analysis, incident investigation, threat hunting, or security policy creation.   

As cloud computing continues to thrive, Kubernetes has become the leading operating system. However, as more applications are moved to the cloud and containerized, Kubernetes has become a popular target for cyber attackers. Unfortunately, Kubernetes does not have a default security configuration, and is unable to detect whether your pods or cluster have been compromised. As a result, it is crucial to take proactive measures to secure your infrastructure.  

So how can you detect the executables being launched within a container? One technology for doing this is eBPF. 

Scenario 1: Privileged Container

When performing pen testing on a Kubernetes system and after gaining access to the Cluster, red teams will focus on finding containers that run with elevated privileges and have direct access to the host system in the environment. Let’s consider this Nginx container as an example. Under normal circumstances, the only processes you expect to see inside such a container are Nginx processes.  

Pretending that a hacker has access to the Cluster, as a next step of the attack, how can they get to the host file system? By deploying a POD with security Context privileges set to true, it deceives all the sandboxing mechanisms inside of the POD, which is enabled by Kubernetes by default, meaning that the pod can access the host’s resources and kernel capabilities (including CAP_SYS_ADMIN: true and CAP_NET_RAW: true = root).  

apiVersion: v1
kind: Pod
metadata:
  name: privileged-pod
spec:
  hostPID: true
  hostNetwork: true
  containers:
  - name: privileged-pod
    image: nginx:latest
    ports:
    - containerPort: 80
    securityContext:
      privileged: true

Executing and breaking 

Once the pod is created, just executing the kubectl command like the following will allow you to escape from the pod: 

➜ ~ kubectl get pods

➜ ~ kubectl exec -it privileged-pod -- nsenter -t 1 -m -u -n -i -p bash  

nsenter --target 1 allows us to enter the host namespace. It also mounts the host volume and accesses the host networking and processes and then it runs as bash, breaking out from the container in every possible way. 

Using eBPF, we can monitor the container runtime and the host itself for real-time system activities. 

Scenario 2: Containerized Web app

Let's look at another scenario, from a Pen tester perspective, to see how eBPF can help to detect such attacks.  

Attack Reconnaissance 

In this next scenario, we will show you how a vulnerability in a web application can be exploited to gain access to a pod hosted in Kubernetes. In the end, you will be familiar with common attack patterns and learn how to use Container Runtime Protection and eBPF to identify sophisticated attacks. 

Before launching an attack, red teams usually conduct reconnaissance to gather information about the target. The purpose is to understand the technology behind the target website, which will help determine the most effective attack strategies. Security testing tools are commonly used to scan targets in such situations. Even though we have different intentions as Penetration Testers, we use these same tools to test our applications. 

Nmap is a tool commonly used by hackers in their kill chain. Attackers typically scan large network ranges to locate exposed IPs and ports. In this case, we will utilize Nmap to determine which ports are open and vulnerable to an attack on our target. 

As expected, port 80 is open, which is inherently a security risk in many cases. It is configured without the proper configurations in place, which attackers can easily use to access systems and data.  

After accessing the page in the unencrypted port, we see that we landed in an internal custom application that takes input and provides base64 code.  

By checking the source code of the application, we see it is missing an input validation, so the application will execute any commands from the users.  Now that we know that the target application is susceptible to RCE, we will exploit this vulnerability to get access to the host.  

We tested using the ls command and got a return value.  

Exploiting the vulnerability

The first step is to set up a shell listener using the netcat command in our pen testing machine and execute the crafted payload, which executes a remote shell (RCE) into the application. Here’s an example to set up the shell listener: 

nc - lvp 4444 

The payload is constructed to run a Bash reverse shell command on the victim’s host to connect to our attacker machine, which would initiate the interactive shell and run /bin/sh. The screen below shows that we got access.  

By doing an initial recon on this host, we can see that we got access to a Kubernetes node.

Escalating privileges

To interact with the Cluster, we will download kubectl from our attacker Kali Linux machine, and by running some simple enumeration commands, we can see that we have full access to the cluster and the ability to list and create Pods. 

We can read the service account secret Token on the recently hacked cluster. 

To do so, we used this command: 

./kubectl get secret admin-service-account-token –o yaml  

Now, we can create new pods using the service account token defined as an environment variable.

We run: 

./kubectl get pods –A –token=${TOKEN} 

How to Prevent Kubernetes Runtime Attacks  

 
It's important to consider how security breaches like container escapes and RCE attacks could have been prevented in the first place. Here are some measures that can help: 

• Network policies: setting network policies can limit communication between containers over the network. For instance, you can create a policy that disallows communication from an external host to a pod running an internal custom web application. 

• Security Control: using security policies and guardrails like the Wiz Admission Controller can prevent the creation of privileged pods, making it impossible for an attacker to escape to the host. The request would be blocked by the webhook. 

• Runtime observability and security: Using eBPF with Runtime protection capabilities and having complete visibility of activities in the cluster can help us detect and stop an attacker early in the attack. 

Wiz has released the Runtime Sensor which utilizes the eBPF program to monitor Kubernetes and containerized environments in real-time. The Sensor gathers targeted eBPF-captured system calls and sends them to Wiz's cloud-based SaaS solution. Using this information, Wiz can identify malicious files, IPs, domains, malicious behavior, and anomaly detection. The Process tree allows Security operators to see connections between process activities, user sessions, and network connections across containers and systems. Wiz also automatically detects workload behaviors, making it possible to detect new behaviors in real-time. In one example, the Sensor identified a non-standard input socket to a remote network (reverse shell) during an attack.

This was based on a process that began a non-standard socket connection to a remote network address, indicating the presence of a threat actor using a reverse shell script to execute commands: 

Wiz Sensor detected a Network tool being used; in this case, the netcat command with the purpose of getting a shell: 

When we deployed our malicious POD, we also detected the execution of create Pods and reading /etc/shadow file: 

Summary 

In this blog, we demonstrated attack vectors in Kubernetes and how It is highly likely that your Kubernetes clusters are the primary target for hackers. We have shown that leveraging eBPF for runtime security and observability is important to minimize the attack surface area in Kubernetes clusters and how the Wiz Sensor detects malicious techniques such as container escape, remote code execution, lateral movement, and persistence. To learn more about the Runtime Sensor, check out the release blog post.