Request without proxy when proxy is set

[galaczi]

I am running concurrently with experimentalContainers: true. Everything seems to work fine but after a while I am seeing Request without proxy console logs. Example:

Request without proxy 127.0.0.36 example.com:443
Request without proxy 127.0.0.36 fonts.gstatic.com:443

Edit: Actually I think it's coming from experimentalContainers and it's not about my proxies.

[szmarczak]

Yes, that's definitely experimental containers. I guess you're running Chromium. Can you try Firefox (launchContext.launcher)? It shouldn't have that issue.

The issue with Chromium its Proxy API is async, while the Firefox one is a dedicated one.

[galaczi]

I don't think they are the same. Example from a couple secs ago:

Request without proxy 127.0.0.22 hcaptcha.com:443
Request without proxy 127.0.0.23 hcaptcha.com:443

[szmarczak]

Thanks for the reply, I'll try to reproduce this.

[galaczi]

Any news on this? I can burn through 1000 proxies in minutes due to this issue. My proxy IPs get banned if a (sub)request is sent without proxy.

[galaczi]

I can reproduce this, but this happens very rarely (I think there might be a race condition, the browser schedules a request before the tab is closed, but the request is actually run after the tab has been closed). Can you confirm if the 127.X.X.X IP addresses in Request without proxy messages are all the same for a single crawl?

I believe this is the reason behind the issue, even though the IPs are not the same. With about 60 concurrency pages frequently close.

Edit: Even though this happens fairly rarely, in my case it snowballs. One request is made without proxy, then that session is banned, page is closed. Now another request is made without proxy, gets banned, closed, slowly most sessions get banned and I am retrying in vain, retiring sessions because they got banned.

Edit 2: Handling such a race condition is probably not simple, if possible at all. The other tab can always close, even mid- request. As a simple solution wouldn't it make more sense to use the new session's assigned proxy if the other tab is closed, instead of not using a proxy at all?

[galaczi]

@szmarczak Since you have been able to reproduce it, what do you think?

[galaczi]

Any feedback on this? Should I open an issue?

[B4nan]

Unfortunately, Szymon is no longer working with us and this experimental feature what hit own invention, I am afraid we don't have anybody who could look into this as of now. We are aware that the chrome support here started failing recently in our CI too, probably because of newer restrictions for extensions made by chrome, so we might be at a dead end there. The firefox version should be still working, I know you said you don't want to use that because of the lack of blockRequests support...

[galaczi]

That's sad news. It would be such a useful feature. I would use 10th the bandwidth I am using with incognitoPages.

I tried Firefox. It's not only the blockRequests that prevents me from using it. My target is a weird blazor website and in Firefox nothing works. A simple js popup takes 30+ secs so my selectors time out. In normal Firefox it works flawlessly.