docs/introduction/crawling

[giscus[bot]]

docs/introduction/crawling

Your first steps into the world of scraping with Crawlee

https://crawlee.dev/docs/introduction/crawling

[Abhaygupta9]

nice

[hussainb]

@mnmkng, Is there any way to set the browser window size when using Playwright as shown in the above examples?
Playwright allows you do that by calling page.setViewportSize(), but we can only do that in the requestHandler. i.e after the page loads.
I was looking to set the viewport / browser size before the browser is navigated to the URLs, maybe some configuration can be done through crawlee.
Thanks!!

[mnmkng]

There are several ways of doing this.

For all browsers in the crawler:

Crawlee launches a persistent context by default, so you can pass the viewport size directly to launch options.

const crawler = new PlaywrightCrawler({
    launchContext: {
        launchOptions: {
            viewport: {
                width: 1920,
                height: 1080,
            }
        }
    }
})

For some browsers in the crawler:

This method allows you to dynamically change which browsers will start with which viewport. See BrowserPool docs.

const crawler = new PlaywrightCrawler({
    browserPoolOptions: {
        preLaunchHooks: [(pageId, launchContext) => {
            launchContext.launchOptions.viewport =  {
                width: 1920,
                height: 1080,
            }
        }],
    },
})

For individual pages:

There are multiple ways of doing this on a page level, but the easiest way is using crawler preNavigationHooks:

const crawler = new PlaywrightCrawler({
    preNavigationHooks: [
        async (crawlingContext, gotoOptions) => {
            const { page } = crawlingContext;
            await page.setViewPortSize({
                width: 1920,
                height: 1080,
            })
        },
    ]
})

Hope this helps :)

[hussainb]

Thank you so much for a prompt and a detailed response!

[mnmkng]

No problem. Note that by switching the viewport like this, you're modifying the browser fingerprint injected by Crawlee, so you might see increased block rates on some heavily protected sites. But if blocking is not an issue, then those are perfectly valid solutions.

You can also change the screen parameters using the fingerprint options, but note that you can't set exact numbers this way, only ranges. And the ability to generate fingerprints based on the provided values depends on whether we've seen screens like this in the wild.

browserPoolOptions: {
            fingerprintOptions: {
                fingerprintGeneratorOptions: {
                    screen: {
                        maxHeight: 2160,
                        maxWidth: 3840,
                        minHeight: 1080,
                        minWidth: 1920,
                    }
                },
            },
        },

[LeMoussel]

Do you mean that passing the viewport size directly to launch options modifying the browser fingerprint injected by Crawlee and that as a result, there may be an increase in blocking rates on some heavily protected sites?

[mnmkng]

@LeMoussel Yes, because there are more parameters than just the viewport tied to the screen size in the fingerprint and if those de-sync, anti-scraping software will most likely flag that as a discrepancy and lower the overall score of the device.

For example, if you have a 1920x1080 screen, then a 1920x900 viewport is ok, but 1920x1080 viewport is suspicious. A viewport like this would have to cover the whole screen and would not leave any room for the browser UI elements, which is not common on real user devices. Extending the viewport beyond the screen resolution is even more unlikely in the real world.

So if Crawlee generates a full HD screen, but you then set a 4k viewport, it could have impact on your bot scores and increase block rates. There are other implications of chaning viewport as well, like CSS media queries, keeping all the browser JS APIs in sync.

If blocking is an issue, I recommend using the fingerprintOptions to choose the right screen sizes.

[mjstelly]

Following the above code, I get
ERROR PlaywrightCrawler: Request failed and reached maximum retries. page.waitForSelector: Timeout 30000ms exceeded.
The whole crawl process failed. I know nothing about this tool, so I have no idea what this error is telling me.

INFO  PlaywrightCrawler: Error analysis: {"totalErrors":1,"uniqueErrors":1,"mostCommonErrors":["1x: page.waitForSelector: Timeout 30000ms exceeded. (file:///<my-path>/projects/my-crawler/src/main.ts:8:20)"]}
INFO  PlaywrightCrawler: Terminal status message: Finished! Total 1 requests: 0 succeeded, 1 failed.

I also could find the example selector ActorStorePagination-pages > a.

[mjstelly]

err, "could not find the example", that is.

[mnmkng]

Hi, yes you're right and thank you for letting us know. It seems the layout of the page we're crawling has changed. We will fix it as soon as possible. The general idea of the tutorial is still the same. Just the selectors might be wrong.