AI agents - potential candidates for robots.txt

[not-not-the-imp]

I don't think the question between blocking AI robots that learn vs AI robots that act as a man in the middle, has been resolved.
Beside the climate impact [1], beside the silliness of rewriting a text that been painstakingly crafted, I'm firmly against a biased machine that will ignore some keywords and imply stuff that is not there [2].

If you end up creating another list of AI-agent bots, those are good candidates:

firecrawlAgent
AndiBot
ExaBot
PhindBot

While I don't expect to have an answer as to how these are classified, I think we could use this issue to compile a list until a decision is made

[1] https://www.scientificamerican.com/article/what-do-googles-ai-answers-cost-the-environment/
[2] https://surevine.com/resource-centre/deepseek-and-chatgpt-censorship-in-llms
[3] https://www.firecrawl.dev/
[4] https://andisearch.com/
[5] https://exabot.viallance.com/
[6] https://www.phind.com/

[glyn]

Please could you explain what you mean by a bot acting as a man in the middle?

[not-not-the-imp]

Take the example of andisearch where it scrape the internet, and provide a Summarize Results which is the one prominent button they want you to click.

That is effectively taking the content of a website relevant to the question, summarize it, and give it back to the user.
That one is interesting because it shows the link it used, so you can compare its output and the actual content.

One definition of Man in the Middle I found is MITM attackers ... enables ... to make covert changes to web pages, manipulate transactions and spy on the user’s activity.

I'm not sure we are at the point were people are using their credit card in AI bots search, so lets forget about the manipulate transaction part.
The changes to web pages is clear, the summary is an overt change to the webpage information.
What is the point of TLS on a website to ensure user have tampered free access to the information, if the AI bot change the information in the end.

Those bots are not biased free, the data used to trained them is not biased free, therefore act as a Man In The Middle when then do not provide the exact data from the website they deemed having the right answer.

[cdransf]

@glyn should we revisit this? Given our updated README I'd be comfortable including the 4 listed.

[tinaponting]

https://trustdecision.com/resources/blog/how-to-protect-against-man-in-the-middle-attacks

[glyn]

https://trustdecision.com/resources/blog/how-to-protect-against-man-in-the-middle-attacks

That's not the MITM mentioned earlier. It's an (alleged) application of AI to detect conventional MITM attacks. But unless we know the User Agent, we can't add any corresponding AI bot to our list.

[glyn]

@glyn should we revisit this? Given our updated README I'd be comfortable including the 4 listed.

Yes. So would I. PR anyone?

[not-not-the-imp]

So firecrawlAgent has been added already.
I could not find enough info on ExaBot from their website to include it here and feel confident, so I omitted it.

PR #134 has AndiBot PhindBot which I'm confident does AI scrapping and genAI it.

( I almost done an ignore previous instruction and here as one commenter does feel slightly sus)

[glyn]

@tinaponting Please could you keep your comments on topic in future. Posting just a link is rarely helpful. Best to explain why you think the link is relevant. Also, posting very brief, cryptic comments is unhelpful too.

If you persist in posting unhelpful comments, we will have no other alternative but to block you. If you find writing English language difficult, please use a translation service or avoid posting and lurk instead.

[tinaponting]

It was what "man in the middle is = Not Ai, i think:)

[glyn]

It was what "man in the middle is = Not Ai, i think:)

@not-not-the-imp was using the term "man in the middle" fairly loosely and generally, so best not to get hung up on the normal definition of the term in relation to security.