403 Error Pages #1
Description
While scraping, there are a handful of legit pages that return 403 errors for some reason. This appears to be a missing file or a misconfigured server on Epic's end and is not something that can easily be fixed.
However, the behavior is for the server to automatically serve the normal index.html page along with the 403 error and so it seems to navigate back to this page with no explanation of the error that has occurred.
I would like to replace all of these 403 pages with an actual 403 page so the user knows an error has happened. We could then include a link back to the root index.html so it still somewhat resembles the way the server behaves.
Proposed Change
- Have an extra step during each scrape script that will handle 403 pages
- It will need to scan the log for errors and parse out the file names
- Files should be replaced with a default 403 error page instead of the index.html it will have scraped.