The vulnerability in Bouncy Castle's Java MLS implementation arises because the uint32 leaf_index defined in RFC 9420 is stored in a signed int. When the top bit is set, the decoded value becomes negative, which is legitimate encoding but mishandled by membership checks in GroupKeySet.SecretTree.hasLeaf and Group.validateRemove. These checks use signed comparisons, allowing out-of-range senders to pass membership validation. This leads to unbounded growth in node lists during PrivateMessage processing, exhausting JVM heap memory and causing denial of service. The fix involves using Integer.toUnsignedLong to treat the leaf_index as unsigned, correctly rejecting invalid senders without affecting valid indices.

An attacker who is a member of a current group can send a specially crafted small message that triggers unbounded memory allocation in the JVM of other group members, causing denial of service. This impacts availability of the MLS group communication. There is no indication of privilege escalation, data disclosure, or code execution. No known exploits are reported in the wild.

A fix is available in Bouncy Castle Java library version 1.86 and later, which correctly interprets the leaf_index as unsigned and rejects out-of-range senders. Users should upgrade to version 1.86 or newer to remediate this vulnerability. No other mitigation is indicated.