Crawler causing SYN floods

[apienk]

Yesterday at 18:17 CEST we noted a SYN flood caused by the project crawler. Please implement request limits.

[jordangarrison]

Also experiencing this issue. We've had to block this project.

[kalebdf]

We experienced a sudden flood of requests as well. We have currently blocked this project (429 Too Many Request). All the best!

[tb0hdan]

Hi guys,

I apologize for this unintended behavior. DomainsProject crawler (https://github.com/tb0hdan/idun) doesn't have any kind of port scanning functionality and uses plain "net/http" library for connections. I am already working on additional limits (on top of existing robots.txt handling):

• delay/sleep between requests
• decreased number of connections to single site
• HTTP 429 code handling

Thank you very much for reporting this. Issue will remain open for historical purposes after the fix.

[Bmess1]

Thank you so much for the quick response!

…

On Jun 30, 2022, at 05:36, Bohdan Turkynewych ***@***.***> wrote:  Hi guys, I apologize for this unintended behavior. DomainsProject crawler (https://github.com/tb0hdan/idun) doesn't have any kind of port scanning functionality and uses plain "net/http" library for connections. I am already working on additional limits (on top of existing robots.txt handling): delay/sleep between requests decreased number of connections to single site HTTP 429 code handling Thank you very much for reporting this. Issue will remain open for historical purposes after the fix. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.

[apienk]

Thanks for the response. We will not blacklist your crawler for now.

[tb0hdan]

• delay/sleep between requests is set to 1 seconds + CrawlDelay as specified by robots.txt
• HTTP 429 code handling was added by use of hashicorp/go-retryablehttp