Is there a centralized place for adding a global noindex meta HTML tag?

[RemiBardon]

I deployed SPI on http://swift-package-index.clvrcld.net, but I don't want it to affect your search engine ranking because it finds a duplicate (see #2200 for a similar issue).

At first I thought about robots.txt, but Introduction to robots.txt · Google Documentation says it's not the solution as an external link pointing to the the page would redirect the crawler which would index the page anyway.

Block Search indexing with noindex · Google Documentation says I should add <meta name="robots" content="noindex"> to every page I don't want indexed.

I want to block all pages, so is there a centralized place I can add this tag? I can do it in a branch of my fork, and I'll create a PR with a flag for conditionally enabling it if you think it'd make sense (e.g. for staging/dev deployments).

I saw two places where it could make sense: DocumentationPageProcessor.swift and PublicPage.swift. Which one should I edit? Both?

[daveverwer]

First of all, thanks so much for your contribution. We're happy to see this project be useful to help test your support for Swift hosting. Thanks also for considering our search ranking, it's a tough problem to keep on top of!

You are correct that the best way to do this is with noindex. I even have some good news in that you won't need to make any code changes or further PRs, only an environment change. If you choose to keep the site up and hosted, we'd suggest running it in our dev/staging mode. This does two main things:

• Adds a banner to the top and bottom of the page showing clearly that this is a dev/staging site in case people happen across it.
• Puts a noindex tag at the top of every page. We did this because we don't want Google to index our staging site, either!

To enable this, set the ENV environment variable to dev, and you should see the changes take effect.

Let us know if this works for you

[RemiBardon]

Thank you for your quick and detailed answer 🙏🏻

I already deployed SPI in development mode using VAPOR_ENV, but I didn't know you had another custom environment variable. I will add it to my configuration then.

I just need to check if you introduced this env var before switching to Swift 5.8 because as I had mentioned we don't support Swift >5.7 yet.

[daveverwer]

I just need to check if you introduced this env var before switching to Swift 5.8 because as I had mentioned we don't support Swift >5.7 yet.

Yes, this has been there since very close to the beginning.

[RemiBardon]

Oh, perfect