Scrapy Crawler Template

This repository must be used as template for new Scrapy crawler projects.

Installing

To install the dependencies for the project:

poetry install

For dependency managment, Poetry was trusted. So make sure you have it installed, then install the project dependencies.

Running

To run the crawler:

poetry shell
python crawler

Enter the Poetry shell to make sure you are using the correct Python version. Then invoke the crawler.

Documentation

Each project must build its documentation.