Do I need useIncognitoPages with useSessionPool?

[galaczi]

I need to run my script concurrently on a website, without sharing any data between instances. Do I need useIncognitoPages with useSessionPool?

Edit: After some testing it seems that setting

    useSessionPool: true,
    persistCookiesPerSession: true,
    proxyConfiguration,

does not really create separate sessions. It used the same proxy config for every request even though I had 50 proxies.

Setting

    launchContext: {
        launcher: PuppeteerExtra,
        useIncognitoPages: true
    }

correctly used a new proxy for each session.

I am still curious if that is the intended behavior?

[metalwarrior665]

Could you share a little more of your config, please? Using SessionPool means that a new session is tied with each browser instance. Crawlee keeps some binding of sessions to proxyUrls so reproduction that we can debug would help.

[metalwarrior665]

No, it should fill the SessionPool up to maxPoolSize (1000 default) (https://crawlee.dev/api/core/interface/SessionPoolOptions) and rotate those sessions. Each session should map to one proxy.

I don't see proxyConfiguration in your example, that would help. How many proxies are there?

[metalwarrior665]

But for browser based crawlers, the session is not rotated for each request but for each browser https://crawlee.dev/api/browser-pool/interface/BrowserPoolOptions#retireBrowserAfterPageCount (100 default).

You can force the rotation by doing session.retire() in the requestHandler (or setting retireBrowserAfterPageCount to 1) so you can test it with tihs

[galaczi]

Ok, that's the solution. Thanks!

[galaczi]

Sorry for reviving an old thread, but I would like to know if there is a way to use different proxy for each page (concurrently, not after retiring session) without using experimental containers or incognito pages.

Experimental containers are not stable, incognito pages don't have cache.

[B4nan]

Nope, I believe that's not possible, not inside one browser instance, not with the same cache.