-
Notifications
You must be signed in to change notification settings - Fork 1k
Open
Labels
t-toolingIssues with this label are in the ownership of the tooling team.Issues with this label are in the ownership of the tooling team.
Milestone
Description
- The current implementation (added in feat: add support for
sameDomainDelay
#2003) essentially holds requests to recently accessed domains in memory and re-enqueues them after a given timeout ⚠️ this may result in lots of unnecessary request queue writes⚠️ there may be unexpected results if this is used in conjunction with request locking- but resolving this for multiple request queue consumers would be quite an endeavor
- we cannot simply use
maxRequestsPerMinute
- we don't know the request URL (and thus the domain) before receiving it from the queue
Regardless of the resolution, the "final" version should be ported over to crawlee-python.
Metadata
Metadata
Assignees
Labels
t-toolingIssues with this label are in the ownership of the tooling team.Issues with this label are in the ownership of the tooling team.