This repository was archived by the owner on Sep 20, 2024. It is now read-only.
This repository was archived by the owner on Sep 20, 2024. It is now read-only.
Build on sitemap.xml file #21
Description
Our primary objective is to build to crawl http://www.va.gov/sitemap.xml
But obviously supporting Drupal's sitemap.xml file, but would be good - https://www.drupal.org/project/xmlsitemap
Basic format is https://www.sitemaps.org/protocol.html
Should be able to use something like this https://stackoverflow.com/questions/57579858/crawl-urls-from-sitemap-xml-using-apify-puppeteer-and-requestqueue