robots: block semver-based URLs

jsha · syphar · commit 191bdcf5b312 · 2024-12-22T06:08:16.000+01:00
diff --git a/static/robots.txt b/static/robots.txt
@@ -1 +1,6 @@
 Sitemap: https://docs.rs/sitemap.xml
+# Semver-based URL are always redirects, and sometimes
+# confuse Google's duplicate detection, so we block crawling them.
+# https://docs.rs/about/redirections
+Disallow: */^
+Disallow: */~
