The OpenSSL Project has fixed a vulnerability in several versions of the software that could enable an attacker to cause a DoS condition on machines that are targetted.
The bug exist due to an odd condition in the way that OpenSSL handles some elliptic curve parameters in certificates. Thr function can go into an infinite loop when parsing those certificates in some cases.
“The BN_mod_sqrt() function, which computes a modular square root, contains a bug that can cause it to loop forever for non-prime moduli. Internally this function is used when parsing certificates that contain elliptic curve public keys in compressed form or explicit elliptic curve parameters with a base point encoded in compressed form,”OpenSSL Statement
The possibility to trigger the infinite loop by crafting a certificate that has invalid explicit curve parameters is likely. Since certificate parsing happens prior to verification of the certificate signature, any process that parses an externally supplied certificate may thus be subject to a DoS attack. The infinite loop can be reached when parsing crafted private keys as they can contain explicit elliptic curve parameters.
There are several different situations in which an OpenSSL implementation can be vulnerable to this bug, including TLS clients that are processing client or server certificates, hosting providers that are taking certificates or private keys from customers, or certificate authorities that are parsing certification requests from subscribers.
With OpenSSL 1.0.2 version the public key is not parsed during initial parsing of the certificate which makes it slightly harder to trigger the infinite loop. However any operation which requires the public key from the certificate will trigger the infinite loop. The attacker can use a self-signed certificate to trigger the loop during verification of the certificate signature.
The vulnerability affects versions 1.0.2, 1.1.1, and 3.0, and is fixed in 1.1.1n and 3.0.2, which were released earlier this week.