docs/guides/scaling-crawlers

[giscus[bot]]

docs/guides/scaling-crawlers

https://crawlee.dev/docs/guides/scaling-crawlers

[fireindark707]

The description of "maxRequestsPerMinute" and "maxTasksPerMinute" are the same. What's their difference?

[B4nan]

It is the same thing, just on different level, this is noted in the maxTasksPerMinute section:

This option can be set by specifying maxRequestsPerMinute in your crawler options too, as it is a shortcut for visibility and ease of access.

maxRequestsPerMinute is a BasicCrawler option, maxTasksPerMinute is an AutoscaledPool option - the pool is what is used under the hood.

[fireindark707]

Does Crawlee has option like DOWNLOAD_DELAY in the scrapy? I want to control the interval between each requests very accurately. For example, a real-world scenario might be that the site returns null values for visits within a 3-second interval.

[mnmkng]

No, not yet. You can get similar behavior by setting maxConcurrency to 1 and then await setTimeout(3000) in your requestHandler.

Or you can try setting maxRequestsPerMinute to a fraction, like maxRequestsPerMinute: 60/3, but even I'm not sure how this will actually work 😄 Maybe we could make it work like this @vladfrangu ? I think this would still allow it to fire 20 requests pretty fast and then wait for the rest of the minute to start again.

[B4nan]

I believe it should be based on a time window, so it should start doing things once "in last 60 seconds there were less than N requests", not wait for the full minute mark.

[vladfrangu]

What @B4nan said is how maxRequestsPerMinute works as it makes the crawling not have bursts of crawls then periods of waiting doing nothing. What you probably want is what @mnmkng said above (maxConcurrency set to 1 and adding a manual timeout at the end of each crawl). Maybe we can add an option for this in the future 👀

[tteguayco]

If we wanted to horizontally scale to being able to tackle a very big site by running the same scraper but in different servers (but avoiding that two different servers scrape the same content), would Crawlee incorporate functionality to help in such case?

[B4nan]

We have plans to implement locking support for request queue later this year, which will make this much easier. You can do a parallel scrape now too, not 100% sure what is the best approach, maybe @metalwarrior665 can suggest something?

[tteguayco]

Hmm. So based on what you mentioned, two scrapers running in different machines could have access to the same request queue?

[B4nan]

Hmm, right, depends on where you run it, it should be doable on apify platform, but with the memory storage you'd have to have a shared disk space, or implement your own storage that would use some shared database. That could be based on the https://github.com/apify/apify-storage-local-js, which is based on sqlite.

[B4nan]

But you could have multiple queues (e.g. for some URL prefix), and one master queue that would just distribute the requests instead of processing them.

[tteguayco]

Very interesting, I'll keep researching on this. Thanks a lot!