Mojo::UserAgent and Latin-1 encoded query parameters

[sanko]

• Mojolicious version: 9.38
• Perl version: v5.38.2
• Operating system: Windows 11 Pro

I'm at the absolute very beginning of rewriting the very first module I ever pushed to CPAN to try and fall in love with Perl again and of course I'm bumping my head on something.

To scrape a BitTorrent tracker, you must include a SHA-1 hash as a query parameter. Trackers expect a properly escaped, ISO-8859-1/Latin-1 encoded string. Of course, Mojo::UserAgent, by default, encodes query params to UTF-8 which causes my escaped infohash look like %1B%C3%90%C2%88%C3%AE%C2%91f%C2%A0b%C3%8FJ%C3%B0%C2%9C%C3%B9%C2%97+%C3%BAn%1A13 rather than the expected
%1B%D0%88%EE%91f%A0b%CFJ%F0%9C%F9%97+%FAn%1A13.

This should be overridable but, for HEAD and GET, Mojo::UserAgent::Transactor::_form(...) silently ignores the user defined charset when merging query parameters. I hesitate to report this as an issue so I'm posting it here; I'm sure there's a solid reason it functions this way (I'm not an HTTP standards expert) but I can't find a git blame or any other discussion around it and no other HTTP client is making this choice by default. Am I missing something? Is setting the charset manually with $tx->req->url->query->charset(undef); my only/best option?

Code Example

Here's a minimal example if such a thing is needed. This scrapes the Debian project's tracker for debian-12.7.0-amd64-netinst.iso.torrent so it shouldn't set off alarms at most rational ISPs but you could always just comment out the start(...) call.

use v5.38;
BEGIN { $ENV{MOJO_CLIENT_DEBUG} = 1; }
use Mojo::UserAgent;
$|++;
#
my $ua        = Mojo::UserAgent->new;
my $info_hash = pack( 'H*', '1bd088ee9166a062cf4af09cf99720fa6e1a3133' );
#
{
    my $tx = $ua->build_tx(
        GET     => 'http://bttracker.debian.org:6969/announce',
        form    => { info_hash => $info_hash, peer_id => 'aaaaaaaaaabbbbbbbbbb', left => 0, downloaded => 0, port => 8080, uploaded => 0 },
        charset => undef    # This does not work
    );
    say '** Wrong: ' . $tx->req->url->to_string;
    say $ua->start($tx)->res->code // 'Error!';
}
say '=' x 20;
{
    my $tx = $ua->build_tx(
        GET     => 'http://bttracker.debian.org:6969/announce',
        form    => { info_hash => $info_hash, peer_id => 'aaaaaaaaaabbbbbbbbbb', left => 0, downloaded => 0, port => 8080, uploaded => 0 },
        charset => undef    # This does not work
    );
    $tx->req->url->query->charset(undef);    # I have to do this?
    say '** Right: ' . $tx->req->url->to_string;
    say $ua->start($tx)->res->code // 'Error!';
}

[Grinnz]

Technically what you want is to interpret as bytes not latin-1 - latin-1 is only relevant in that latin-1 characters map to the bytes equivalent to the same unicode codepoints.

Since all you're doing is constructing a URL you can do that rather than using the "form" generator, try something like this:

my $url = Mojo::URL->new('http://bttracker.debian.org:6969/announce');
$url->query->charset(undef)->append(%params);
my $tx = $ua->build_tx(GET => $url);

[Grinnz]

And of course, this will break if you pass any text that isn't either ASCII or a byte string to the query, since it no longer encodes text to bytes.

[sanko]

Technically what you want is to interpret as bytes not latin-1 - latin-1 is only relevant in that latin-1 characters map to the bytes equivalent to the same unicode codepoints.

Yep, raw bytes. You're right and I know this... It became latin-1 when I started fleshing out my original question to put here. It was going to just be "Why does Mojo::UserAgent ignore charset for URL queries?" and my code example on PerlMonks but then I came here and started editorializing.

Well, anyway, I guess I'm already doing the right/only thing.