Auto-linking with a schema that uses % as the schema identifier

[black-puppydog]

I'm trying to implement some custom extensions to markdown for the scuttlebutt markdown flavour as implemented in ssb-markdown which is the JS implementation and relies on markdown-it. Hence it makes sense to me to make the re-implementation using markdown-it-py. 🙂

One of the key features of ssb is that messages are referenced by ids like this: %9eJYIT1HDNhWOeLK0EhhiHJTPwvDGZWGd/E6CBCG5XY=.sha256
(feeds have an @ identifier, and blobs a & so they may have similar issues, but let's talk about message ids only for the sake of this discussions)

Anyway, so these message ids should be linked to urls like this:

 <a href="#/msg/%259eJYIT1HDNhWOeLK0EhhiHJTPwvDGZWGd%2FE6CBCG5XY%3D.sha256">
  %9eJYI...
 </a>

Note that the link text is an abbreviated version of the full id.

So I have this regex to match message IDs:

MESSAGE_SIGIL_REGEX = r'[a-zA-Z0-9+/=]{44}\.sha256'

To automatically linkify these ids I set the % character up as a schema:

# this is in the main rendering method
md.linkify.add("%", {"validate": message_regex, "normalize": normalize_message_sigil})

def normalize_message_sigil(obj, match):
  old_url = match.url
  match.url = urllib.parse.quote(old_url, safe="")
  match.text = f"{old_url[:6]}..."

The problem I have with this is that once matched by linkify, the link text that results is actually interpreted as a url-encoded string, i.e. the %9e gets decoded to a (non-displayable) character.
The resulting link isn't exactly what I had hoped for:

  <a href="#/msg/%259eJYIT1HDNhWOeLK0EhhiHJTPwvDGZWGd%2FE6CBCG5XY%3D.sha256">
   �JYI...
  </a>

I've stepped through this a while now and I haven't figured out yet whether this is a bug or just me holding this wrong...
The resulting text gets put through state.md.normalizeLinkText here:

https://github.com/executablebooks/markdown-it-py/blob/b1a74b460d6bcc24e38209ea784164485abe2a1a/markdown_it/rules_core/linkify.py#L104

That function in turn passes the whole thing through mdurl.decode(mdurl.format(parsed), mdurl.DECODE_DEFAULT_CHARS + "%"):

https://github.com/executablebooks/markdown-it-py/blob/bb6cf6ecc545cf587357794cd340cc2207aa9688/markdown_it/common/normalize_url.py#L63

And I don't see any way to prevent it from doing so...

Now, I realize I could just generate the text to escape the % into something like %, but the result is then that the & sign is escaped into %9eJYI... which is also not quite what I want...

So... is this an issue of usage? Is there something obvious I'm missing?