Check that email address length is valid on the original email address string since callers may continue to use that string

Previously, we checked that the ASCII email address (with IDNA ASCII) and the normalized email address satisfied the whole-address length limit. However, callers may use the original input string. Since Unicode NFC normalization typically reduces string length (if it changes the string), this can cause the post-normalization check to pass when the pre-normalization length is not valid. So we should additionally check that the original input also meets the maximum length requirement. Callers might also construct an address that has an internationalized local part and ASCII domain, maybe? So that's now checked too.

The whole-address length test is revised to test each possible address format, first the original email address string (with any display name removed) so that exception messages correspond to the input string where possible. Then the normalized address is checked, since we encourage callers to use it. Then the ASCII address is checked since callers who send email without a SMTPUTF8-enabled stack will use this, or the normalized internationalized local part (there won't be an ASCII local part in this case) combined with the ASCII domain.

Some length tests are added with a Unicode character whose NFC normalization is actually a decomposition: U+FB2C (Hebrew Letter Shin With Dagesh And Shin Dot) is unusual in that its NFC normalization actually expands it to multiple code points (https://www.unicode.org/faq/normalization.html). In these cases, the address will be valid before normalization but not valid after.

See #142.

Author: JoshData

CHANGELOG.md

@@ -2,6 +2,7 @@ In Development
 --------------
 
 * Email addresses with internationalized local parts could, with rare Unicode characters, be returned as valid but actually be invalid in their normalized form (returned in the `normalized` field). Local parts now re-validated after Unicode NFC normalization to ensure that invalid characters cannot be injected into the normalized address and that characters with length-increasing NFC normalizations cannot cause a local part to exceed the maximum length after normalization.
+* The length check for email addresses with internationalized local parts is now also applied to the original address string prior to Unicode NFC normalization, which may be longer and could exceed the maximum email address length, to protect callers who do not use the returned normalized address.
 * A new option to parse `My Name <address@domain>` strings, i.e. a display name plus an email address in angle brackets, is now available. It is off by default.
 
 2.1.2 (June 16, 2024)
@@ -10,7 +11,7 @@ In Development
 * The domain name length limit is corrected from 255 to 253 IDNA ASCII characters. I misread the RFCs.
 * When a domain name has no MX record but does have an A or AAAA record, if none of the IP addresses in the response are globally reachable (i.e. not Private-Use, Loopback, etc.), the response is treated as if there was no A/AAAA response and the email address will fail the deliverability check.
 * When a domain name has no MX record but does have an A or AAAA record, the mx field in the object returned by validate_email incorrectly held the IP addresses rather than the domain itself.
-* Fixes in tests.
+* Fixes in tests. Some additional tests added.
 
 2.1.1 (February 26, 2024)
 -------------------------

README.md

@@ -300,13 +300,6 @@ they are unnecessary. For IPv6 domain literals, the IPv6 address is
 normalized to condensed form. [RFC 2142](https://datatracker.ietf.org/doc/html/rfc2142)
 also requires lowercase normalization for some specific mailbox names like `postmaster@`.
 
-### Length checks
-
-This library checks that the length of the email address is not longer than
-the maximum length. The check is performed on the normalized form of the
-address, which might be different from a string provided by a user. If you
-send email to the original string and not the normalized address, the email
-might be rejected because the original address could be too long.
 
 Examples
 --------

email_validator/syntax.py

@@ -176,12 +176,11 @@ def unquote_quoted_string(text: str) -> Tuple[str, bool]:
     return display_name, local_part, domain_part, is_quoted_local_part
 
 
-def get_length_reason(addr: str, utf8: bool = False, limit: int = EMAIL_MAX_LENGTH) -> str:
+def get_length_reason(addr: str, limit: int) -> str:
     """Helper function to return an error message related to invalid length."""
     diff = len(addr) - limit
-    prefix = "at least " if utf8 else ""
     suffix = "s" if diff > 1 else ""
-    return f"({prefix}{diff} character{suffix} too many)"
+    return f"({diff} character{suffix} too many)"
 
 
 def safe_character_display(c: str) -> str:
@@ -609,44 +608,66 @@ def validate_email_domain_name(domain: str, test_environment: bool = False, glob
 
 
 def validate_email_length(addrinfo: ValidatedEmail) -> None:
-    # If the email address has an ASCII representation, then we assume it may be
-    # transmitted in ASCII (we can't assume SMTPUTF8 will be used on all hops to
-    # the destination) and the length limit applies to ASCII characters (which is
-    # the same as octets). The number of characters in the internationalized form
-    # may be many fewer (because IDNA ASCII is verbose) and could be less than 254
-    # Unicode characters, and of course the number of octets over the limit may
-    # not be the number of characters over the limit, so if the email address is
-    # internationalized, we can't give any simple information about why the address
-    # is too long.
-    if addrinfo.ascii_email and len(addrinfo.ascii_email) > EMAIL_MAX_LENGTH:
-        if addrinfo.ascii_email == addrinfo.normalized:
-            reason = get_length_reason(addrinfo.ascii_email)
-        elif len(addrinfo.normalized) > EMAIL_MAX_LENGTH:
-            # If there are more than 254 characters, then the ASCII
-            # form is definitely going to be too long.
-            reason = get_length_reason(addrinfo.normalized, utf8=True)
-        else:
-            reason = "(when converted to IDNA ASCII)"
-        raise EmailSyntaxError(f"The email address is too long {reason}.")
-
-    # In addition, check that the UTF-8 encoding (i.e. not IDNA ASCII and not
-    # Unicode characters) is at most 254 octets. If the addres is transmitted using
-    # SMTPUTF8, then the length limit probably applies to the UTF-8 encoded octets.
-    # If the email address has an ASCII form that differs from its internationalized
-    # form, I don't think the internationalized form can be longer, and so the ASCII
-    # form length check would be sufficient. If there is no ASCII form, then we have
-    # to check the UTF-8 encoding. The UTF-8 encoding could be up to about four times
-    # longer than the number of characters.
+    # There are three forms of the email address whose length must be checked:
     #
-    # See the length checks on the local part and the domain.
-    if len(addrinfo.normalized.encode("utf8")) > EMAIL_MAX_LENGTH:
-        if len(addrinfo.normalized) > EMAIL_MAX_LENGTH:
-            # If there are more than 254 characters, then the UTF-8
-            # encoding is definitely going to be too long.
-            reason = get_length_reason(addrinfo.normalized, utf8=True)
-        else:
-            reason = "(when encoded in bytes)"
-        raise EmailSyntaxError(f"The email address is too long {reason}.")
+    # 1) The original email address string. Since callers may continue to use
+    #    this string, even though we recommend using the normalized form, we
+    #    should not pass validation when the original input is not valid. This
+    #    form is checked first because it is the original input.
+    # 2) The normalized email address. We perform Unicode NFC normalization of
+    #    the local part, we normalize the domain to internationalized characters
+    #    (if originaly IDNA ASCII) which also includes Unicode normalization,
+    #    and we may remove quotes in quoted local parts. We recommend that
+    #    callers use this string, so it must be valid.
+    # 3) The email address with the IDNA ASCII representation of the domain
+    #    name, since this string may be used with email stacks that don't
+    #    support UTF-8. Since this is the least likely to be used by callers,
+    #    it is checked last. Note that ascii_email will only be set if the
+    #    local part is ASCII, but conceivably the caller may combine a
+    #    internationalized local part with an ASCII domain, so we check this
+    #    on that combination also. Since we only return the normalized local
+    #    part, we use that (and not the unnormalized local part).
+    #
+    # In all cases, the length is checked in UTF-8 because the SMTPUTF8
+    # extension to SMTP validates the length in bytes.
+
+    addresses_to_check = [
+        (addrinfo.original, None),
+        (addrinfo.normalized, "after normalization"),
+        ((addrinfo.ascii_local_part or addrinfo.local_part or "") + "@" + addrinfo.ascii_domain, "when the part after the @-sign is converted to IDNA ASCII"),
+    ]
+
+    for addr, reason in addresses_to_check:
+        addr_len = len(addr)
+        addr_utf8_len = len(addr.encode("utf8"))
+        diff = addr_utf8_len - EMAIL_MAX_LENGTH
+        if diff > 0:
+            if reason is None and addr_len == addr_utf8_len:
+                # If there is no normalization or transcoding,
+                # we can give a simple count of the number of
+                # characters over the limit.
+                reason = get_length_reason(addr, limit=EMAIL_MAX_LENGTH)
+            elif reason is None:
+                # If there is no normalization but there is
+                # some transcoding to UTF-8, we can compute
+                # the minimum number of characters over the
+                # limit by dividing the number of bytes over
+                # the limit by the maximum number of bytes
+                # per character.
+                mbpc = max(len(c.encode("utf8")) for c in addr)
+                mchars = max(1, diff // mbpc)
+                suffix = "s" if diff > 1 else ""
+                if mchars == diff:
+                    reason = f"({diff} character{suffix} too many)"
+                else:
+                    reason = f"({mchars}-{diff} character{suffix} too many)"
+            else:
+                # Since there is normalization, the number of
+                # characters in the input that need to change is
+                # impossible to know.
+                suffix = "s" if diff > 1 else ""
+                reason += f" ({diff} byte{suffix} too many)"
+            raise EmailSyntaxError(f"The email address is too long {reason}.")
 
 
 class DomainLiteralValidationResult(TypedDict):

email_validator/validate_email.py

@@ -72,7 +72,9 @@ def validate_email(
 
     # Collect return values in this instance.
     ret = ValidatedEmail()
-    ret.original = email
+    ret.original = ((local_part if not is_quoted_local_part
+                    else ('"' + local_part + '"'))
+                    + "@" + domain_part)  # drop the display name, if any, for email length tests at the end
     ret.display_name = display_name
 
     # Validate the email address's local part syntax and get a normalized form.

tests/test_syntax.py

@@ -409,10 +409,15 @@ def test_domain_literal() -> None:
         ('me@中1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444.com', 'The email address is too long after the @-sign.'),
         ('meme@1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.com', 'The email address is too long (4 characters too many).'),
         ('my.long.address@1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.11111111112222222222333333333344444.info', 'The email address is too long (2 characters too many).'),
-        ('my.long.address@λ111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.11111111112222222222333333.info', 'The email address is too long (when converted to IDNA ASCII).'),
-        ('my.long.address@λ111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444.info', 'The email address is too long (at least 1 character too many).'),
-        ('my.λong.address@1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.111111111122222222223333333333444.info', 'The email address is too long (when encoded in bytes).'),
-        ('my.λong.address@1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444.info', 'The email address is too long (at least 1 character too many).'),
+        ('my.long.address@λ111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444.info', 'The email address is too long (1-2 characters too many).'),
+        ('my.long.address@\uFB2C111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444.info', 'The email address is too long (1-3 characters too many).'),
+        ('my.λong.address@1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.111111111122222222223333333333444.info', 'The email address is too long (1 character too many).'),
+        ('my.λong.address@1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444.info', 'The email address is too long (1-2 characters too many).'),
+        ('my.\u0073\u0323\u0307.address@1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444.info', 'The email address is too long (1-2 characters too many).'),
+        ('my.\uFB2C.address@1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.11111111112222222222333333333344444.info', 'The email address is too long (1 character too many).'),
+        ('my.\uFB2C.address@1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.11111111112222222222333333333344.info', 'The email address is too long after normalization (1 byte too many).'),
+        ('my.long.address@λ111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.11111111112222222222333333.info', 'The email address is too long when the part after the @-sign is converted to IDNA ASCII (1 byte too many).'),
+        ('my.λong.address@λ111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.1111111111222222222233333333334444444444555555555.6666666666777777777788888888889999999999000000000.11111111112222222222333333.info', 'The email address is too long when the part after the @-sign is converted to IDNA ASCII (2 bytes too many).'),
         ('me@bad-tld-1', 'The part after the @-sign is not valid. It should have a period.'),
         ('me@bad.tld-2', 'The part after the @-sign is not valid. It is not within a valid top-level domain.'),
         ('me@xn--0.tld', 'The part after the @-sign is not valid IDNA (Invalid A-label).'),