Webscraping spiders
Scrapy is a free and open-source web crawling framework written in python. It is desigend for web scraping but can also be used for extracting data using the APIs or as a general purpose web crawler. It is currently maintained by Scrapinghub Ltd.
It is also known as web spider, automatic indexer or crawler. It is an internet bot that helps in web indexing. It collects information from a webpage and the links related to them. They crawl one page at a time until all the pages are indexed.
pip install scrapy
pip install scrapy-selenium