v1.34: adds Quillbot and Panscient crawlers

What's Changed

• chore(robots.json): adds Quillbot by @cdransf in #142
• chore(robots.json): adds Panscient by @cdransf in #143

Full Changelog: v1.33...v1.34

v1.33: adds SBIntuitionsBot

What's Changed

• Add SBIntuitionsBot by @Ivan-Chupin in #141

New Contributors

• @Ivan-Chupin made their first contribution in #141

Full Changelog: v1.32...v1.33

v1.32: Adds AndiBot, PhindBot, YandexAdditional + YandexAdditionalBot

What's Changed

• Add AndiBot and PhindBot by @not-not-the-imp in #134
• chore(robots.json): revert "adds imgproxy crawler" by @cdransf in #136
• chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard by @cdransf in #139
• chore(robots.json): adds YandexAdditional crawlers by @cdransf in #140

New Contributors

• @not-not-the-imp made their first contribution in #134

Full Changelog: v1.31...v1.32

v1.31: adds new bots, clarifies blocking methods for Bing, updates Claude bots

What's Changed

• chore(robots.json): adds Google-CloudVertexBot by @cdransf in #129
• Mention X-Robots-Tag header as alternative for bing by @Mihitoko in #131
• chore(README): updates the opening line of our README to clarify the types of agents we block by @cdransf in #132
• chore(robots.json): adds MistralAI-User/1.0 crawler by @cdransf in #126
• Updating Claude Bots by @kylebuckingham in #98
• chore(robots.json): adds wpbot by @cdransf in #133

New Contributors

• @Mihitoko made their first contribution in #131
• @kylebuckingham made their first contribution in #98

Full Changelog: v1.30...v1.31

v1.30: Adds HAProxy converter, Caddyfile, Bing AI opt-out instructions, Traefik plugin and QualifiedBot crawler

What's Changed

• Fixed meta-external* being titlecase, and removed period for consistency by @maiavixen in #112
• HAProxy converter added. by @rwijnen-um in #113
• Added a Caddyfile converter by @Crazyroostereye1 in #110
• Fix Python syntax error by @glyn in #115
• Bing AI opt-out instructions by @cdransf in #119
• Add Traefik plugin to the README.md file by @FlorentPoinsaut in #121
• chore(robots.json): adds QualifiedBot crawler by @cdransf in #122
• Fix JSON syntax error by @joehoyle in #123
• lint robots.json during pull requests by @holysoles in #125

New Contributors

• @maiavixen made their first contribution in #112
• @rwijnen-um made their first contribution in #113
• @Crazyroostereye1 made their first contribution in #110
• @FlorentPoinsaut made their first contribution in #121
• @joehoyle made their first contribution in #123
• @holysoles made their first contribution in #125

Full Changelog: v1.29...v1.30

v1.29: 5 bots added

This releases adds 5 new crawlers/bots.

aiHitBot
Cotoyogi
Factset_spyderbot
FirecrawlAgent
TikTokSpider

What's Changed

• Clarify our position on sponsorship by @glyn in #107
• AI bots to consider adding by @dennislee1 in #109

New Contributors

• @dennislee1 made their first contribution in #109

Full Changelog: v1.28...v1.29

v1.28: adds NovaAct and Operator

Adds NovaAct (Amazon) and Operator (OpenAI).

What's Changed

• Include "AI Agents" from Dark Visitors by @jsheard in #105

Full Changelog: v1.27...v1.28

v1.27: adds imgproxy crawler

Thanks to Brian at Skewray Research for highlighting this one.

What's Changed

• .htaccess: Allow robots access to /robots.txt by @sideeffect42 in #80
• Add run-tests workflow by @sideeffect42 in #84
• Document testing in README by @glyn in #83
• Update perplexity bots by @deyigifts in #91
• Implement Nginx configuration snippet export by @ThomasLeister in #94
• Fix html-mangled hyphen in 'Perplexity-Users' bot name by @fbartho in #100
• chore(robots.json): adds imgproxy crawler by @cdransf in #102

New Contributors

• @sideeffect42 made their first contribution in #80
• @deyigifts made their first contribution in #91
• @ThomasLeister made their first contribution in #94
• @fbartho made their first contribution in #100
• @cdransf made their first contribution in #102

Full Changelog: v1.26...v1.27

v1.26: Adds Brightbot 1.0

Adds Brightbot 1.0. Thanks to Brian at Skewray Research for calling this one out!

v1.25: update SemrushBot agents

What's Changed

Replaces SemrushBot with:

User-agent: SemrushBot-OCOB
User-agent: SemrushBot-SWA

• Actually block Semrush’s AI tools by @nisbet-hubbard in #73

Full Changelog: v1.24...v1.25