ai.robots.txt
Version without Internet Archive? #74
TheZeldaZone
started this conversation in
Ideas
Replies: 1 comment
-
|
The Internet Archive crawler has user agent ia_archiver (see this question), which is not listed in robots.txt. Please could you say why you believe the Internet Archive is blocked? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
This list, while editable, has the same problem Cloudflare's AI blocker did, it includes Internet Archive for goodness knows why. I'm not sure why they're included at all but a branch without them would be handy at least.
Beta Was this translation helpful? Give feedback.
All reactions