Clarify policy of the kinds of agents we block

[glyn]

I'd like the README to be more explicit about the kinds of bots that we block.

This issue was raised while reviewing #126. In this particular case, I couldn't find a bot listed for training Mistral AI's LLM, so it's not the case that we only block bots associated with AI companies which have AI training bots that we also block.

The current statement:

This is an open list of web crawlers associated with AI companies and the training of LLMs to block.

(emphasis added)

seems a little too restrictive. But, depending on the language we end up with, we could block all bots from companies involved in AI, such as Google, and user website could end up not being indexed by the search engines of such companies. I think we want to steer clear of blocking search engine indexing bots, otherwise that will inhibit adoption of our robots.txt et al.

[cdransf]

I think that's a fair approach — I'd be happy to come down on the side of blocking training but not bots used for discovery. We'd need to adjust the list if that's the direction we choose to go as we currently include bots like Applebot, DuckAssistBot and OAI-SearchBot.

As much as I may personally dislike OpenAI, if OAI-SearchBot is purely for indexing, we should then allow that, for example.

Perhaps:

This is an open list of web crawlers associated with the training of LLMs to block.

With all that said I'd be ok having a supplemental list of these search crawlers for the sake of being thorough.

I would also be inclined to keep agents that act based on user prompts in the list as it's unclear whether that data is retained or how it's used in most cases. E.g. surfacing information vs. those that surface a link with a citation as one would get with a traditional search engine.

[glyn]

I'd be happy to come down on the side of blocking training but not bots used for discovery.

I'd be comfortable with that, but I was actually favouring our current direction and lumping in bots used in training AI with those used for discovery by AI systems.

This is an open list of web crawlers to block because they are associated with the training or operation of LLMs.

[cdransf]

I'd be happy to come down on the side of blocking training but not bots used for discovery.

I'd be comfortable with that, but I was actually favouring our current direction and lumping in bots used in training AI with those used for discovery by AI systems.

This is an open list of web crawlers to block because they are associated with the training or operation of LLMs.

Gotcha! I think that's perfectly reasonable as well — it may be easier to maintain.

Would you include AI search bots in the discovery category? If we have a single list we could allow the table to guide specific implementations.

[glyn]

Would you include AI search bots in the discovery category? If we have a single list we could allow the table to guide specific implementations.

Yes, but I'm not proposing a formal categorisation as we often need to read between the lines of crawler docs. So a bot is either in our list or not, but our list covers more than just LLM training bots.

[cdransf]

Would you include AI search bots in the discovery category? If we have a single list we could allow the table to guide specific implementations.

Yes, but I'm not proposing a formal categorisation as we often need to read between the lines of crawler docs. So a bot is either in our list or not, but our list covers more than just LLM training bots.

Understood — where does that leave us with cases like #126?

[glyn]

Once we've merged a README update for this issue, I'm happy to merge #126.

[cdransf]

Hm, perhaps something to the effect of "Our list contains AI-related crawlers of all types, regardless of purpose. Users should consult the table of bot metrics to guide the implementation of their list."

This may also pertain to and help resolve #53.

[cdransf]

Opened a PR here: #132

[cdransf]

Closing out with #132 merged.