DDoS attack on a category path

[addison74]

Recently I noticed that the daily number of unique visitors exploded in a shop I am managing, From 800 to 20,000. It was clear to me that the website was under attack and I analyzed the webserver logs.

I noticed that a path to a specific category (for example ID:190) was being visited intensely. Here all the addresses started using the available filters. As the filters in the category were around 24, one IP address started to combine them, hence a record number of daily visits over 200,000.

For the moment I had no other idea than to ban filtering in that category and everything calmed down (for the moment). I mention that all the IP addresses did not have the signs of a bot, the fingerprint in the logs seemed like a real visit. All coming from Amazon, Huawei and US owners.

We will certainly face AI in the near future by attacking websites.

[toejam34]

Do you put anything in front of your server? I used Cloudflare as its the only way I can manage the bots

[colinmollenhour]

I think some crawlers see the layered nav links like new pages to visit and then it creates massive number of "unique" links b eventually hitting every possible combination of attributes. I coded an "anti-abuse" module long ago for a client and it basically set rates limits for things like layered nav visits, number of products added to the cart, add to compare, advanced search, etc. Basically anything that was potentially resource intensive and could be abused. But, the simplest protection would be something like nginx rate limit with a reasonable "sustained" limit. For example, 1 r/s with a burst of 100 to avoid accidental trips. You gotta be careful that static assets don't count towards this, though.

[colinmollenhour]

If it's not actually a malicious bot you can set Disallow rules in robots.txt to prevent crawlers from travelling down the layered nav links:

Disallow: /*color=*
Disallow: /*price=*
...

[addison74]

This solution decreased by 15% the number of total accessing. Most likely we are facing different types of website attacks. We have implemented a multi-level solution, the provider's firewall has been reconfigured, on the servers we have reconfigured HAProxy, we continue to block IP ranges. Compared to the initial one we managed to go down by almost 70%. Fortunately the webserver and database resources are more than enough and can cope. The only problem is that it loads our logs too much with garbage.

[sreichel]

Have you tried "logrotate" to only keep current logs?

[addison74]

We use logrotate. All logs are archived automatically. It's not a space issue, but those days are compromised if anyone want to analyze visits. Most likely a bot checks the path category by adding IDs or the sitemap. Those that exist it reads the available filters and then starts creating links to them for visiting.

[addison74]

The service provider assured us with a high-performance physically firewall, but I discussed the problem with him and he told me that these IPs behave like human visitors, they are not crawlers or bots. They make a few accesses to the filters in a category, adding to comparison and that's it. The accesses are random in time. We identified 11,200 IP addresses that participated in this action. We have blocked most of them and for the moment there are no problems.

[colinmollenhour]

Ahh yeah, sounds like a well-distributed botnet. Just FYI, I came across this service recently; am going to try it out for a site, although not a public one. I could see using it for OpenMage if you were having major issues with bots: https://arcjet.com/
It's a REST API so you'd need to write code to integrate it.

[addison74]

I'm back with new information. Clearly, the website was harvested before so that all available filters were known. Then the visits started, an IP address only makes one visit and that's it. There are tens of thousands from Saudi Arabia, Indonesia, the Philippines, China, Brazil, the United States. The fact that it comes directly to the website, without any other intermediate page helped us observe the behavior and block all these addresses, but it didn't end there. We went further and blocked the entire range of an IP address to be more sure. It decreased in intensity. It generally uses 2 - 3 filters in the URL. We'll think about other solutions after we manage to solve it definitively. For now we use Varnish that serves all requests of this type from the cache.

[addison74]

I solved the problem by making a combination of conditions in .htaccess and fail2ban+ipset. After analyzing the behavior of these bots I found some common elements, so a condition was created in .htaccess that serves a 403 error that is recorded in a log file. From the log fail2ban collects all these IP addresses and bans 16 other IP addresses around it (A.B.C.D/28). ipset takes care of keeping the addresses in a set. In two hours 2,700 IP addresses were collected.

[colinmollenhour]

Nice work mitigating a DDoS! Sounds like a big PITA. Was there some reason you can't use Cloudflare in this case?

I came across this gem in case you're looking for something more, although it sounds like you got it under control. https://github.com/topics/ddos-protection

[addison74]

We are using a high-performance solution provided by the company that supplied our dedicated server. The issue with these attacks is that each one originates from a single IP address. At first glance, it appears to be a human visit. The same IP randomly returns in the next days. Several IPs attack simultaneously. With the help of ChatGPT, we analyzed their behavior and were gradually provided with solutions until we reached the final version. Today has been quiet, but we are now filtering thousands of IPs, which we block for an extended period. Blocking IP ranges manually worked at first, until we got overwhelmed. Many thanks for your ideas.

[addison74]

In 3 days we collected 68.270 different IPs having the same behavior. At this moment the number is increasing with 5 IPs per minute coming from Brazil, India, US, Indonesia, Vietnam. China, Philippine.

[kiatng]

I asked copilot for the cost of attack:

Summary: Orchestrating such an attack could cost as little as a few hundred dollars using cheap proxies, or several thousand dollars for high-quality, persistent, and harder-to-block traffic from diverse global sources. The actual cost depends on the attack scale, sophistication, and quality of IP rotation.

[addison74]

There are much better ways to block such attacks (Cloudflare), but blocking those IP addresses in the firewall for a long time seems to me to be a quick and safe solution. I have deleted the jail with addresses several times to make sure that only the persistent ones will remain. Approximately 25,000 were captured, from individual addresses to ranges. When it calms down I will publish the list, maybe it will help others to protect themselves from these attacks.