Enhancement: automatically disallow robots in dev or test mode

[wernerkrauss]

When deploying sites to a staging server it'd be nice to disallow robots either by Environment dev or test or even by a .env setting.

An extension point to modify the per page settings would also be handy to tweak the settings dynamically.