v1.14: adds Webzio-Extended

What's Changed

• Adds the Webzio-Extended bot
• only build on changes to robots.json by @jbowdre in #29
• Add FAQ by @glyn in #31
• Tracking Dark Visitors Automatically by @ChenghaoMou in #32
• Add links for reporting and FAQ to README.md by @nisbet-hubbard in #33

New Contributors

• @jbowdre made their first contribution in #29
• @ChenghaoMou made their first contribution in #32

Full Changelog: v1.13...v1.14

v1.13: corrects agent name

What's Changed

• Fix Imagesift user agent by @jsheard in #27
• Add Cloudflares first-party scraper blocking to FAQ by @jsheard in #28

New Contributors

• @jsheard made their first contribution in #27

Full Changelog: v1.12...v1.13

v1.12: adds ICC-Crawler

What's Changed

• Add last line of defence to FAQ by @nisbet-hubbard in #24
• Add ICC-Crawler by @mirium999 in #25

New Contributors

• @mirium999 made their first contribution in #25

Full Changelog: v1.11...v1.12

v1.11: adds PetalBot and facebookexternalhit

What's Changed

• Add PetalBot (and facebookexternalhit?) by @nisbet-hubbard in #22

Full Changelog: v1.10...v1.11

v1.10: adds Scrapy

Thank you to Brian at Skewray Research for calling this one out.

Full Changelog: v1.9...v1.10

v1.9: adds Timpibot

Full Changelog: v1.8...v1.9

v1.8: adds VelenPublicWebCrawler

Thank you to Brian at Skewray Research for calling this one out.

Full Changelog: v1.7...v1.8

v1.7: adds Meta-ExternalAgent

Many thanks to @nisbet-hubbard for raising this as well.

Full Changelog: v1.6...v1.7

v1.6: adds OAI-SearchBot to robots.txt

chore: add OAI-SearchBot

v1.5: drops AdsBot-Google; adds GoogleOther-Image/Video

Full Changelog: v1.4...v1.5