Bots blocker

[addison74]

On my servers every day there is a massive collection of content by bots. Some of them use "User Agent" header but others don't. I consider OpenMage needs a better protection at least for known bots. For a few years I used for crawlers robots.txt but it is useless. My personal opinion is this file should be remove from OpenMage root.

But we can come up with a solution. We can add in .htaccess condition for bots whitelists/blacklists and for forbidden paths. This file content cannot be seen like robots.txt.

For those bots without "User Agent" header and which are hiding as a normal browser the only chance in by identifying the IP and block it with Fail2Ban/IPSet.

[luigifab]

We are using: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker