Build in respect for robots.txt

[mgifford]

Because Purple Hats uses apify we can leverage this:
apify/crawlee#229 (comment)

and use the https://github.com/samclarke/robots-parser

And then evaluate using:

getRobotsTxt(url)
isAllowedRobotsTxt(url, ua)
isDisallowedRobotsTxt(url, ua)
getCrawlDelayRobotsTxt(ua)