Add HTTP resilience mode #176
Conversation
| self.session = requests.Session() | ||
|
|
||
| # IA specific thing | ||
| self.session.headers.update({"X-Requested-With": "XMLHttpRequest"}) |
There was a problem hiding this comment.
I don't know if it will have an unintended effect on requests in other jurisdictions. Maybe we should consider pulling it out of the base class.
There was a problem hiding this comment.
Seconded, pretty sure that'll trigger different behavior on a few sites.
There was a problem hiding this comment.
Oops, yeah that's a mistake in porting the code - thanks for catching!
| @@ -66,6 +71,22 @@ def clean_whitespace(obj): | |||
| return obj | |||
|
|
|||
|
|
|||
| def get_random_user_agent(): | |||
There was a problem hiding this comment.
I think this is a good approach for now, but in the future it might be annoying to have to update core if want to change this pool. An env var seems like it would be a monster mess. I kinda hate to go adding config files but maybe there's some better way to define in this in the future.
There was a problem hiding this comment.
I agree, this starting point is not optimal. Worth going as far as pulling a library in (https://pypi.org/project/fake-useragent/) ?
This is a WIP draft of porting the HTTP resilience features that @elseagle introduced (originally in the FL scraper, later I re-used in IA scraper) up into openstates-core. The goal is to turn these features into a toggleable behavior instead of something that needs to be grafted onto scrapers one-by-one.
This is not sufficiently tested yet. I'm running an IA scrape (with the grafted version removed from scraper code), and will want to do the same with FL. And then test with some other scrapers to make sure I haven't regressed something.
cc @showerst and @alexobaseki