GameOver(lay): Easy-to-exploit local privilege escalation vulnerabilities in Ubuntu Linux affect 40% of Ubuntu cloud workloads

CVE-2023-2640 and CVE-2023-32629 were found in the OverlayFS module in Ubuntu, which is a widely used Linux filesystem that became highly popular with the rise of containers as its features enable the deployment of dynamic filesystems based on pre-built images. OverlayFS serves as an attractive attack surface as it has a history of numerous logical vulnerabilities that were easy to exploit. This makes the new discovered vulnerabilities especially risky given the exploits for the past OverlayFS vulnerabilities work out of the box without any changes. 

 The two vulnerabilities are exclusive to Ubuntu because Ubuntu introduced several changes to the OverlayFS module in 2018. These modifications did not pose any risks at the time. In 2020, a security vulnerability was discovered and patched in the Linux kernel, however due to Ubuntu’s modifications, an additional vulnerable flow was never fixed in Ubuntu. This shows the complex relationship between Linux kernel and distro versions, when both are updating the kernel for different use cases. This complexity poses hard-to-predict risks. 

Ubuntu fixed the vulnerabilities on July 24th, 2023. Users should update their kernels to the latest version.

Who is vulnerable? 

As there are many releases available for Ubuntu, determining the affected versions can be challenging. Our research determined that the following were impacted:

This matrix is a work in progress, and we will update it as we obtain new information. In the meantime, Ubuntu has released official security bulletins for CVE-2023-32629 and CVE-2023-2640, and Wiz customers can check their impacted resources in the Wiz portal.

What is GameOver(lay)? 

OverlayFS in Linux is a union filesystem that lays one filesystem on top of another, enabling file modifications without altering the base. Commonly used in scenarios like Docker containers, it keeps the base image intact while storing changes in a separate layer. However, this flexibility makes OverlayFS a potential security concern and an interesting attack surface. It allows users to mask other filesystems and perform unintended operations against them (like ignoring mount options such as nodev or nosuid when accessing USB/FUSE filesystems). Moreover, it lets users copy files from the “lower” to the “upper” directory while maintaining critical file metadata (e.g. sticky-bit), which should not be possible in Linux.  

Historically, OverlayFS has suffered from multiple vulnerabilities, many of which have been strikingly similar in that they leverage the primitives mentioned above to bypass Linux’s basic security restrictions. These vulnerabilities are so alike that their exploits often function in much the same way. 

Our team has discovered significant flaws in Ubuntu's modifications to OverlayFS. These flaws allow the creation of specialized executables, which, upon execution, grant the ability to escalate privileges to root on the affected machine. Linux has a feature called "file capabilities" that grants elevated privileges to executables while they're executed. This feature is reserved for the root user, while lower-privileged users cannot create such files. However, we discovered that it's possible to craft an executable file with “scoped” file capabilities and trick the Ubuntu kernel into copying it to a different location with “unscoped” capabilities, granting anyone who executes it root-like privileges.   

The general flow of the vulnerability is the same as a previously disclosed Linux kernel vulnerability from 2020, CVE-2021-3493, whose exploits work for the discovered vulnerability without any changes. This raises the question: how did Ubuntu become vulnerable to an issue that was already addressed two years ago?

How did Ubuntu become vulnerable? 

Ubuntu is widely recognized as one of the most popular Linux distributions. Ubuntu often modifies the Linux kernel to optimize hardware compatibility and introduce new features it sees fit. Ubuntu pioneered the accessibility of OverlayFS to non-privileged users, unlike the Linux project and other distributions. In adopting this feature in the early stage, Ubuntu encouraged its success and advanced the container ecosystem. At the same time, it also bet on its security maturity and the potential risk it could pose.ﾠ

Ubuntu introduced several changes to the OverlayFS module, which did not pose any immediate risks. But later, the situation changed due to further modifications made by the Linux project:

• 29 Jan, 2018: Ubuntu modified its kernel to use an internal implementation of the function responsible for setting extended attributes (__vfs_setxattr_noperm instead of vfs_setxattr).

• 14 Dec, 2020: Linux discovered and patched a new OverlayFS vulnerability. When Linux pushed the fix in 2020, it added protections to the function responsible for setting extended attributes (vfs_setxattr). But since Ubuntu used an internal implementation (__vfs_setxattr_noperm), it still had a vulnerable flow.

• 28 April, 2022: Due to additional OverlayFS modifications by Linux, a second vulnerable flow was introduced that shares the same root cause.

Mitigation & detection 

Taking advantage of these vulnerabilities necessitates the establishment of a user namespace and an OverlayFS mount. This implies that an attacker would need local code execution capabilities on the targeted system, making remote exploitation seem improbable. 

Ubuntu released fixed versions for the impacted kernels (see the list above). It's crucial for Ubuntu users who are affected to upgrade to these newer versions. For those unable to upgrade their kernel, there's an alternative solution: restrict the user namespace usage to users with limited privileges to stop potential exploitation.

To enforce this, execute the following command:

sudo sysctl -w kernel.unprivileged_userns_clone=0

For those wanting this change to persist after the system restarts, use this command:

echo kernel.unprivileged_userns_clone=0 | \
  sudo tee /etc/sysctl.d/99-disable-unpriv-userns.conf


Wiz customers can use the pre-built query and advisory in the Wiz Threat Center to search for vulnerable instances in their environment. In addition, the Wiz Runtime Sensor can detect exploitation of the GameOver(lay) kernel vulnerability, as well as the exploitation of other Local Privilege Escalation (LPE) vulnerabilities using generic behavioral detections.

Conclusions 

#GameOver(lay) represents two easy-to-exploit privilege escalation vulnerabilities that affect ~40% of Ubuntu workloads. Furthermore, due to the nature of the issues, the same publicly available exploit that works for prior OverlayFS vulnerabilities can be used to exploit these issues. Security teams should patch their workloads or restrict OverlayFS to root users only. 

The vulnerabilities we discovered also highlight the risks involved in modifying complex open-source projects. Initially, Ubuntu's kernel modifications seemed harmless. After subsequent changes made to the Linux kernel, which would naturally seem reasonable to any developer, vulnerabilities were inadvertently introduced.

The research story 

In early 2023, a patch was added to the Linux kernel for CVE-2023-0386, an earlier vulnerability in OverlayFS. Essentially, a threat actor could abuse OverlayFS to manipulate the kernel to copy files from attacker-controlled mounts to regular directories. To exploit the flaw, the attacker could copy a SUID file from a specially crafted FUSE mount to a regular directory. When executed from a regular directory, this would grant them root privileges. 

When we reviewed the changes made to the Linux kernel to fix CVE-2023-0386 (out of curiosity), we observed that Linux restricted file creation to users within the user namespace. By preventing the creation of files as the original root, the patch effectively blocks the generation of SUID files.

However, there are other ways to grant special security features to files besides making them an SUID owned by root, such as using Linux file capabilities. With these capabilities, the owner of the file can remain a low-privileged user but still be granted super-user capabilities upon the file’s execution, thereby potentially bypassing this patch. Our team not only wanted to explore what would happen with that method, but also check whether there were other mitigations in place to prevent this evasion. Ultimately, this led us to discover #GameOver(lay).

Vulnerability #1: CVE-2023-2640 (ovl_copy_xattr)

To test our theory, we first entered a new user namespace. The user namespace is crucial, because we can only set file capabilities and mount the OverlayFS filesystem in a user namespace.ﾠ

unshare -rm bash 

The second step was to add powerful file capabilities to an arbitrary executable in a directory that later would be used as the “lower” layer for OverlayFS. 

setcap cap_sys_admin +eip lower/suid 

The third step was to mount the OverlayFS filesystem. By specifying the lowerdir and upperdir mount options, we set the base (lower) layer to the directory with the executable and the upper to the writable directory.ﾠﾠ

mount -t overlay overlay -o rw,lowerdir=lower,upperdir=upper,workdir=workdir mountﾠ

Finally, we had to modify the executable’s metadata so it would be copied to the upper directory by the kernel and hopefully keep its file capabilities. So we gave it one last touch: 

touch mount/suid

This was a somewhat naïve test, but the existence of such an easily exploitable vulnerability as CVE-2023-0386 made us believe it just might work. 

When examining the executable’s capabilities after exiting the user namespace, we were extremely surprised to discover the file capabilities had indeed remained!

sagi@localhost:/home/sagi$ getcap lower/suid 
lower/suid = cap_dac_override,cap_sys_admin+eip 
sagi@localhost:/home/sagi$ getfattr -d -m - lower/suid 
# file: lower/suid 
security.capability=0sAQAAAgIAIAACACAAAAAAAAAAAAA= 

Only a few minutes later we discovered that the vulnerability does not work on all Linux distributions and is actually specific to Ubuntu. Luckily, as Ubuntu fans, we ran the experiment on our own Ubuntu instance, otherwise we might have missed it. 

To understand the root cause, we delved into Ubuntu’s OverlayFS implementation to learn how Ubuntu’s kernel copies extended attributes of files upon their modification. 

The copy occurs within the function ovl_copy_xattr by calling ovl_do_setxattr. In certain Ubuntu kernel versions, ovl_do_setxattr calls __vfs_setxattr_noperm directly to set the extended attributes rather than calling vfs_setxattr:

int ovl_copy_xattr(struct super_block *sb, struct path *oldpath, struct dentry *new)
{
  // ...
  for (name = buf; list_size; name += slen) {
    // ...
    error = ovl_do_setxattr(OVL_FS(sb), new, name, value, size, 0); // <--- call to ovl_do_setxattr from ovl_copy_xattr
    if (error) {
      if (error != -EOPNOTSUPP || ovl_must_copy_xattr(name))
        break;

      /* Ignore failure to copy unknown xattrs */
      error = 0;
    }
  }
  // ...
}

static inline int ovl_do_setxattr(struct ovl_fs *ofs, struct dentry *dentry, const char *name, const void *value, size_t size, int flags)
{
  // ...
  err = __vfs_setxattr_noperm(ovl_upper_mnt_userns(ofs), dentry, name, value, size, flags); // <--- call to __vfs_setxattr_noperm
  // ...
}

One of the key differences between vfs_setxattr and __vfs_setxattr_noperm is that whereas the former converts file security capabilities to limit them to a user namespace before copying them to the upper directory via cap_convert_nscap, the latter copies them directly without the conversion. 

To better illustrate this, here is a comparison of different outputs when running getfattr on a “capable” file where the extended attributes are set via __vfs_setxattr_noperm in a vulnerable kernel version and vfs_setxattr in a secure version:

user@ubuntu:~$ getcap capable_file 
capable_file = cap_dac_override,cap_setuid,cap_sys_admin+eip 
user@ubuntu:~$ getfattr -d -m – capable_file 
# file : capable_file 
security.capability=0sAQAAAoIAIACCACAAAAAAAAAAAAA= <--- vulnerable version  

user@ubuntu:~$ getcap capable_file 
capable_file = cap_dac_override,cap_setuid,cap_sys_admin+eip 
user@ubuntu:~$ getfattr -d –m - capable_file 
# file: capable_file 
security.capability=0sAQAAA4IAIACCACAAAAAAAAAAAADoAwAA <--- non-vulnerable version

Vulnerability #2: CVE-2023-32629 (ovl_copy_up_meta_inode_data) 

To find CVE-2023-32629 we went the other way around and searched for code flows for the unsafe ovl_do_setxattr function.  

When mounting OverlayFS with metacopy=on, Linux does not copy the entire file if only the file’s metadata has changed. Instead, it copies only the metadata, which includes file capabilities.  

This operation is performed by the ovl_copy_up_meta_inode_data function, which uses the ovl_do_setxattr function to copy file capability information.  

By mounting OverlayFS with the metacopy flag and modifying a file’s metadata, we were able to reach this code flow and prepare an empty skeleton file with the desired file capabilities in the upper directory. Then by giving it a little touch on the mount point, the kernel copied the file’s content into the skeleton file and yielded a fully functional executable with high-privilege file capabilities outside the user namespace.

Responsible disclosure timeline:

We disclosed the vulnerabilities to Ubuntu in June 2023. Ubuntu promptly investigated and addressed the issues while keeping us updated. 

• June 23, 2023 – Wiz Research reported GameOver(lay) to Ubuntu. 

• June 29, 2023 – Ubuntu confirmed the existence of GameOver(lay), mentioning that it was already familiar with one of the vulnerabilities (CVE-2023-2640) but unaware of the other (CVE-2023-32629). 

• June 30, 2023 – Ubuntu informed Wiz Research that it had successfully reproduced the second CVE and was working on a patch.  

• July 10, 2023 – Ubuntu informed Wiz Research that it planned on releasing the fixes to its users by the end of the month. 

• July 25, 2023 – Ubuntu informed Wiz Research that the patches were available. 

References:

Overlayfs Over Fuse Local Root Privilege Escalation (CVE-2016-1576) 

OverlayFS - Potential Privilege Escalation using overlays copy_up (CVE-2021-3847) 

Ubuntu overlayfs privilege escalation vulnerability analysis (CVE-2021-3493) 

The OverlayFS vulnerability CVE-2023-0386: Overview, detection, and remediation (CVE-2023-0386)