This repository was archived by the owner on Jul 23, 2019. It is now read-only.
This repository was archived by the owner on Jul 23, 2019. It is now read-only.
Add support for sitemap.xml or robots.txt in 'urls' option #18
Description
Since most sites requiring snapshots utilize a sitemap.xml or robots.txt listing all crawlable URLs already, it'd be nice to be able to utilize these files as the source for urls in the urls option, rather than a raw array of strings.