CVE-2023-50981: 
Linux Debian vulnerability analysis and mitigation

ModularSquareRoot in Crypto++ (aka cryptopp) through 8.9.0 contains a vulnerability that allows attackers to cause a denial of service through infinite loop exploitation. The vulnerability was discovered and disclosed in December 2023, affecting all versions of Crypto++ up to and including version 8.9.0. The issue specifically occurs when processing crafted DER public-key data associated with squared odd numbers (NVD).

The vulnerability exists in the ModularSquareRoot function, which is used in ECP::DecodePoint. The issue occurs in the second while loop where the function attempts to find a value n starting from 2 and incrementing by one until Jacobi(n, p) = -1. The flaw manifests when p is in the form of m^2 where m is an odd number, causing Jacobi(n, p) to only return values of 1 or 0, never -1, resulting in an infinite loop. The vulnerability has been assigned a CVSS v3.1 base score of 7.5 (HIGH) with vector CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H (NVD, GitHub Issue).

When exploited, this vulnerability can lead to a denial of service condition. Any user or server attempting to read a maliciously crafted DER public key file during processes such as ECDSA verification can be forced into an infinite loop, effectively causing the system to become unresponsive (GitHub Issue).

A proposed fix involves checking whether p is a prime number at the beginning of the ModularSquareRoot function. If p is not prime, the function should reject further calculations as any results would be incorrect in this case (GitHub Issue).