How to run

$ scrapy runspider <spider> // wordhit_spider.py, for instance

Add proxy

References:

• https://github.com/TeamHG-Memex/scrapy-rotating-proxies
• https://github.com/aivarsk/scrapy-proxies
• https://www.buildersociety.com/threads/become-an-unforgiving-scrapist-the-scrapy-framework-for-python.1771/

Rotating user agent

References:

• https://github.com/alecxe/scrapy-fake-useragent
• https://github.com/cnu/scrapy-random-useragent
• https://gist.github.com/seagatesoft/e7de4e3878035726731d