CVE-2024-40950: 
Linux Kernel vulnerability analysis and mitigation

CVE-2024-40950 addresses a vulnerability in the Linux kernel's memory management subsystem, specifically related to the misuse of mappinglargefolio_support() function for anonymous folios. The issue was discovered when a large folios split test triggered a warning message "Cannot split file folio to non-0 order", despite the test cases being only for anonymous folios (Kernel Git).

The vulnerability stems from a missing foliotestanon() check in the splithugepagetolisttoorder() function, where the folio passed to mappinglargefoliosupport() could be an anonymous folio. The mappinglargefoliosupport() function is only intended for page cache folios, not anonymous folios. This resulted in failed splits of anonymous THP (Transparent Huge Pages). The issue also affects shmemmapping(), though the shmemmapping() in _splithuge_page() is not impacted because for anonymous folios, the end parameter is set to -1 (Kernel Git).

When CONFIGTRANSPARENTHUGEPAGE is enabled, khugepaged attempts to collapse read-only file-backed pages to THP, even when the filesystem doesn't properly support large folios. This can lead to improper handling of memory pages and potential system instability (Kernel Git).

The issue has been resolved by adding proper checks for anonymous folios and implementing a VMWARNONONCE() warning in mappinglargefoliosupport() for anon mapping to detect incorrect usage. The fix ensures that large anon THP is successfully split without triggering warnings (Kernel Git).