Does it handle cloudflare?

[therepo90]

Can it beat cloudflare in 2021? In comparison to pupeteer-stealth?

[mnmkng]

Yes, but not out of the box. Custom fingerprints are needed. We will hopefully add this feature before the end of this year.

[LeMoussel]

What browser do you use with PlaywrightCrawler?
Rem: It seems that using Firefox, the detection is more difficult.

@luvsic3 Can you post a complete example of how you are blocked with prompthero.com/newest

[mnmkng]

@luvsic3 I'm now getting a captcha on all tries even on my regular browser, so it just might be a setting of the website. You can set up Cloudflare leniently or strictly. When the settings are strict, it's very difficult to pass without custom modifications. It could also be a proxy country setting. I'm connecting from CZ now, maybe US would work better. Also LeMoussel correctly suggests that trying Firefox might help.

[rick-yo]

@LeMoussel @mnmkng
Thank you for your help, I'm using chromium, but Also get blocked when using firefox. I guess prompthero.com might set its Cloudflare setting very strictly.
I don't know if it is helpful, but I can crawl prompthero.com when using https://github.com/FlareSolverr/FlareSolverr
Thank you again~

[LeMoussel]

Do you test the Apify fingerprint generator?
It generate a fingerprint (you can store it) and then assign it as launchContext: { fingerprint }

[barjin]

@LeMoussel To set things straight, Crawlee already uses fingerprint-* tools under its hood - you can pass options to the underlying generator instance by doing this:

import { PlaywrightCrawler } from 'crawlee';
import { BrowserName } from '@crawlee/browser-pool';

(async () => {
    const crawler = new PlaywrightCrawler({
        async requestHandler({ page }) {
            // your code here
        },
        browserPoolOptions: {
            fingerprintOptions: {
                fingerprintGeneratorOptions: {
                    browsers: [BrowserName.firefox],
                    screen: {
                        minWidth: 1920,
                        minHeight: 1080,
                    }
                }
            }
        }
    });

    await crawler.run(['https://crawlee.dev']);
})();

If you provide your own manually generated fingerprint as you said, it might get overwritten by the internal fingerprint - to be sure, you can set browserPoolOptions.useFingerprints to false, which turns the automatic fingerprint generation off.