AI robots.txt, AI scrapers block ai scrapers
-
Updated
Aug 3, 2025
AI robots.txt, AI scrapers block ai scrapers
Protects websites & webapps from unwanted scraping by bots and AI by automating the creation of custom protection files such as robots.txt, ai.txt, and server config files such as .htaccess.
Add a description, image, and links to the block-bots topic page so that developers can more easily learn about it.
To associate your repository with the block-bots topic, visit your repo's landing page and select "manage topics."