Skip to content

Allow distinct timeouts for differently labeled handlers in the router #1485

@damianr13

Description

@damianr13

Describe the feature
I want to be able to specify different timeouts for handlers with different labels. If I expect the website I am crawling to have two or more types of pages, I am doing different things on each of those types, so the time it takes to process a page is different.

Motivation
I am trying to crawl a category page with "infinite scroll" + "load more button" instead of pagination. Similarly to the example in the tutorial (https://crawlee.dev/docs/introduction/scraping), I have 2 types of pages: LIST and DETAIL.

Currently I am facing the problem of hitting a timeout before being able to load all the elements on the LIST page. I looked it up and I found the parameter requestHandlerTimeoutSecs that could be passed to the crawler to increase the timeout limit. My understanding is that this limit applies to all the requests regardless of their types, but I would still like to keep the limit for an individual DETAIL page lower than the high timeout value I need to specify for the LIST page.

Metadata

Metadata

Assignees

Labels

featureIssues that represent new features or improvements to existing features.t-toolingIssues with this label are in the ownership of the tooling team.

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions