header encoding

[graingert]

see #1236

Some related URLs

https://dzone.com/articles/utf-8-in-http-headers
#1163
https://bugzilla.mozilla.org/show_bug.cgi?id=601933#c2

previously the recommendation was:

   The TEXT rule is only used for descriptive field contents and values
   that are not intended to be interpreted by the message parser. Words
   of *TEXT MAY contain characters from character sets other than ISO-
   8859-1 [22] only when encoded according to the rules of RFC 2047

but it has since been updated to
https://www.rfc-editor.org/rfc/rfc7230.html#page-26

   Historically, HTTP has allowed field content with text in the
   ISO-8859-1 charset [ISO-8859-1], supporting other charsets only
   through use of [RFC2047] encoding.  In practice, most HTTP header
   field values use only a subset of the US-ASCII charset [USASCII].
   Newly defined header fields SHOULD limit their field values to
   US-ASCII octets.  A recipient SHOULD treat other octets in field
   content (obs-text) as opaque data.

so it seems like they're saying

def decode(v: bytes) -> bytes | str:
    try:
        return v.decode("ascii")
    except UnicodeDecodeError:
        return v

but we need to maintain an allowlist of "newly" and "historically" defined header fields

[Cajuteq]

Hi, I ended here after I tried to use utf-8 in content-disposition: filename there is rfc 8187 which described use of encoding in such situation. I think starlette should support the following :

{
    "Content-Disposition": "attachment; 
        filename=foobar.txt; 
        filename*=UTF-8''" + filename
}

an explanation (in french) of the rfc8187 uses a similar example here :

<Files "foobar.txt">
    Header set Content-Disposition "attachment; filename=foobar.txt; filename*=utf-8''f%%C3%%B6b%%C3%%A0r.txt"
</Files>

Also I think #1098 should link here as well

Edit : Also this has already been done in the FileResponse Class of Starlette, I'm here talking of the Response Class that does not support the starred versions of headers