Cannot access a very big url

[Udaymakxenia]

Hi Phil I was trying to download an mp3 from an AWS server and could not do it as it kept giving me HTTP 404 Bad Request error. I checked the url and it works with the normal HTTP client libary. but when I use the url.begin("URL") it throws an error 404. Other shorter urls work perfectly fine. The URL that I have is 1500 characters long. Any idea on solving this.
17:00:49.606 -> [I] URLStream.h : 89 - virtual bool audio_tools::URLStream::begin(const char*, const char*, MethodID, const char*, const char*): https://s3.ap-south-1.amazonaws.com/socfiles.geomitra.in/Anganwadi_AUDIO_Instructions/Audio_Instructions_ADVANCED/Advanced_Act_01_Empathy_in_Nature.m

17:00:49.606 -> [I] Url.h : 74 - Url::parse

17:00:49.606 -> [I] Url.h : 110 - url->https://s3.ap-south-1.amazonaws.com/socfiles.geomitra.in/Anganwadi_AUDIO_Instructions/Audio_Instructions_ADVANCED/Advanced_Act_01_Empathy_in_Nature.mp3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAYP5VRGTBGGWWGJFR%2F20231008%2Fap-south-1%2Fs

17:00:49.653 -> [I] Url.h : 112 - host->s3.ap-south-1.amazonaws.com

17:00:49.653 -> [I] Url.h : 113 - protocol->https

17:00:49.653 -> [I] Url.h : 114 - path->/socfiles.geomitra.in/Anganwadi_AUDIO_Instructions/Audio_Instructions_ADVANCED/Advanced_Act_01_Empathy_in_Nature.mp3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAYP5VRGTBGGWWGJFR%2F20231008%2Fap-south-1%2Fs3%2Faws4_request&X-Amz-Date=202310

17:00:49.653 -> [I] Url.h : 115 - port->443

17:00:49.653 -> [I] URLStream.h : 293 - WiFiClientSecure

17:00:49.653 -> [I] URLStream.h : 293 - WiFiClientSecure

17:00:49.700 -> [I] HttpRequest.h : 227 - process connecting to host s3.ap-south-1.amazonaws.com port 443

17:00:50.582 -> [I] HttpRequest.h : 213 - connected 1 timeout 60000

17:00:50.582 -> [I] HttpRequest.h : 238 - Free heap: 161176

17:00:50.582 -> [I] HttpHeader.h : 361 - HttpRequestHeader::setValues - path: /socfiles.geomitra.in/Anganwadi_AUDIO_Instructions/Audio_Instructions_ADVANCED/Advanced_Act_01_Empathy_in_Nature.mp3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAYP5VRGTBGGWWGJFR%2F20231008%2Fap-south-1%2Fs3%2

17:00:50.582 -> [I] HttpHeader.h : 127 - Content-Length 0

17:00:50.582 -> [I] HttpHeader.h : 258 - HttpHeader::write

17:00:50.582 -> [I] HttpHeader.h : 370 - HttpRequestHeader::write1stLine

17:00:50.629 -> [I] HttpHeader.h : 385 - -> GET /socfiles.geomitra.in/Anganwadi_AUDIO_Instructions/Audio_Instructions_ADVANCED/Advanced_Act_01_Empathy_in_Nature.mp3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAYP5VRGTBGGWWGJFR%2F202

17:00:50.629 -> [I] HttpHeader.h : 193 - -> Content-Length: 0 17:00:50.629 -> [I] HttpHeader.h : 193 - -> Host: s3.ap-south-1.amazonaws.com 17:00:50.629 -> [I] HttpHeader.h : 193 - -> Connection: close 17:00:50.629 -> [I] HttpHeader.h : 193 - -> Accept: */* 17:00:50.629 -> [I] HttpHeader.h : 301 - -> <CR LF> 17:00:50.629 -> [I] HttpRequest.h : 275 - Request written ... waiting for reply 17:00:50.676 -> [I] HttpHeader.h : 226 - HttpHeader::read 17:00:50.676 -> [W] HttpHeader.h : 233 - Waiting for data...

17:00:51.148 -> [I] HttpHeader.h : 163 - HttpHeader::readLine -> HTTP/1.1 400 Bad Request 17:00:51.148 -> [I] HttpHeader.h : 163 - HttpHeader::readLine -> Date: Sun, 08 Oct 23 11:30:49 GMT 17:00:51.148 -> [I] HttpHeader.h : 163 - HttpHeader::readLine -> Connection: close 17:00:51.195 -> [I] HttpHeader.h : 163 - HttpHeader::readLine -> x-amz-request-id: 8A716F8841670FCB 17:00:51.195 -> [I] HttpHeader.h : 163 - HttpHeader::readLine -> Content-Length: 0 17:00:51.195 -> [I] HttpHeader.h : 163 - HttpHeader::readLine -> 17:00:53.212 -> [W] HttpLineReader.h : 27 - HttpLineReader readlnInternal->no Data 17:00:53.212 -> [I] HttpHeader.h : 163 - HttpHeader::readLine -> 17:00:53.212 -> [W] HttpRequest.h : 174 - no CONTENT_LENGTH found in reply 17:00:53.212 -> [I] URLStream.h : 110 - size: 0

17:00:53.212 -> [I] URLStream.h : 341 - virtual bool audio_tools::URLStream::waitForData() 17:00:53.212 -> [I] URLStream.h : 343 - Request written ... waiting for reply 17:00:53.212 -> [E] URLStream.h : 347 - Error code recieved ... stop waiting for reply 17:00:53.212 -> [I] URLStream.h : 353 - available: 0
The displayed in 17:00:50.582 -> [I] HttpHeader.h : 361 - HttpRequestHeader::setValues - path: /socfiles.geomitra.in/Anganwadi_AUDIO_Instructions/Audio_Instructions_ADVANCED/Advanced_Act_01_Empathy_in_Nature.mp3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAYP5VRGTBGGWWGJFR%2F20231008%2Fap-south-1%2Fs3%2`
does not show the full url. I have a doubt that there is a size limit to the url in the library. Any way of changing it?

[pschatzmann]

The MAX_HTTP_REQ_LEN is currently defined in HttpHeader.h.
Maybe it make sense to move this to AudioConfig.h

I am not sure if your long size is still ok or if you would need to increase the stack size..

[Udaymakxenia]

Ok Will check it out. Thanks for the quick reply.

[Udaymakxenia]

I did try increasing the MAX_HTTP_REQ_LEN value but it still takes in only the same number of characters in the url as before.

[pschatzmann]

How did you check this: you can't trust the logger unless you increase the corresponding size as well....
Also note that the value needs to be bigger then just the URL because we need to build the full 1st http request line

[Udaymakxenia]

I tried 4096, 2000 ,1600 all returned same results.
I will try changing the log buffer and update you

[Udaymakxenia]

Tried it still the same, it returns 199 characters in the first GET line

[pschatzmann]

Oh, you don't talk about the URL itself but parameters that you receive ?
That would be MAX_HTTP_HEADER_LINE_LENGTH

[Udaymakxenia]

Tried that too did not wok.

        void write1stLine(Client &out){
            LOGI("HttpRequestHeader::write1stLine");
            char msg[2000];
            Str msg_str(msg,2000);
            const char* method_str = methods[this->method_id];
            msg_str = method_str;
            msg_str += " ";
            msg_str += this->url_path.c_str();
            msg_str += " ";
            msg_str += this->protocol_str.c_str();
            msg_str += CRLF;
            out.print(msg);
            int len = strlen(msg);
            msg[len-2]=0;
            LOGI("-> %s", msg);
        }

I changed the size of char msg[2000]. It was 200 before. It works now. This can be found at line 368 in HTTPHeader.h file

[pschatzmann]

That's definitly a bug!
I have committed the related correction...

[pschatzmann]

I finally decided to move this data from the stack to the heap and increase the default size to 1K