Figure out best approach for tuning indexing behavior of development docs

[sffc]

See #5

@robertbastian landed a robots.txt that blocks the ICU4X dev pages from crawlers. @lianghai suggested that this might not be the best practice. @markusicu chimed in, too. Let's figure out the best approach.

Also CC @glechner9147

[sffc]

I'll weigh in to say that it would be very nice to solve this problem with directory-level configurations. I don't want to get in the business of customizing our docs generation to put .html-specific meta tags, making sure that the tags get added in only certain situations but not others, etc.

[lianghai]

Note that with a hosting solution that’s not entirely static like GitHub Pages, you’ll be able to customize the HTTP headers. For example, see the documentation of Cloudflare Pages: https://developers.cloudflare.com/pages/configuration/headers/.

[sffc]

(We need a solution that works with GitHub Pages)