Finding sitemaps through robots.txt or sitemap index

[skttl]

#h5yr it's a great looking package!

I love that it can crawl urls from a sitemap, instead of only relying on the content cache. Does it work with sitemap index files or sitemaps referenced in robots.txt?

I think that would be a great feature!

[rickbutterfield]

Thanks @skttl!

Yes, index sitemaps are supported currently (in fact it runs exactly like this on my own site by checking against https://rickbutterfield.dev/sitemap-index.xml)

Discovering sitemaps from robots.txt isn't currently supported, so will get that on the roadmap!