v1.31: adds new bots, clarifies blocking methods for Bing, updates Claude bots

cdransf released this 22 May 19:33

· 30 commits to main since this release

7bf7f91

What's Changed

chore(robots.json): adds Google-CloudVertexBot by @cdransf in #129
Mention X-Robots-Tag header as alternative for bing by @Mihitoko in #131
chore(README): updates the opening line of our README to clarify the types of agents we block by @cdransf in #132
chore(robots.json): adds MistralAI-User/1.0 crawler by @cdransf in #126
Updating Claude Bots by @kylebuckingham in #98
chore(robots.json): adds wpbot by @cdransf in #133

New Contributors

@Mihitoko made their first contribution in #131
@kylebuckingham made their first contribution in #98

Full Changelog: v1.30...v1.31

Contributors

cdransf, kylebuckingham, and Mihitoko

Assets 3