In Bouncy Castle for Java versions prior to 1.86, the MLS implementation (org.bouncycastle.mls) incorrectly stores the RFC 9420 uint32 leaf_index as a signed int. When the wire value has the top bit set, it decodes to a negative number, which is a valid encoding but mishandled by membership checks (GroupKeySet.SecretTree.hasLeaf and Group.validateRemove). These checks compare the signed decoded value directly against the tree's leaf count, allowing out-of-range senders to pass. This flaw enables a crafted unprotected PrivateMessage to cause LeafIndex.directPath to perform NodeIndex.parent() arithmetic that never reaches the tree root, causing unbounded growth of the node list and eventual JVM heap exhaustion, resulting in denial of service to all group members. The fix involves interpreting the leaf index as unsigned using Integer.toUnsignedLong, thereby rejecting out-of-range senders without affecting well-formed leaf indices.

An attacker who is a current member of an MLS group can send a specially crafted small message that triggers unbounded memory growth in the JVM of all other group members, causing a denial of service. This affects availability but does not indicate compromise of confidentiality or integrity. No known exploits in the wild have been reported.

A fix is available in Bouncy Castle for Java version 1.86, which correctly interprets the leaf_index as unsigned and rejects out-of-range sender values. Users should upgrade to version 1.86 or later to remediate this vulnerability. No other mitigation is required as the issue is fixed in the official release.