A denial-of-service vulnerability in ruby-saml occurs because the SAML response is validated for Base64 format before checking the message size, allowing large inputs to cause resource exhaustion. This can lead to excessive memory and CPU usage, application slowdown, or crashes. The issue affects ruby-saml versions prior to 1.18.1 and is fixed in version 1.18.1.