Update for RFC 9309 Robots Exclusion Protocol #35
Description
It seems Google got a standard published a few months ago. We should review it in detail, and make sure we follow it. We should also mention it in the README.
{{ message }}
It seems Google got a standard published a few months ago. We should review it in detail, and make sure we follow it. We should also mention it in the README.