Not clear what robots field does

[TimDaub]

Hi,

I'm having trouble understanding the robots field. A little description of what each value does would be really nice.

Let me know if I can help by e.g. sending a PR.