SeleniumScraper

Tool to speed up writing Selenium parsers

Requirements

• Python 3.10+
• Internet connection
• Chrome or Firefox browser installed on your machine

Installation

Installing with PyPI

...

Usage

Scraper initialization

You can choose between chrome or firefox browsers

from selenium_scraper import Scraper
scraper = Scraper.chrome()
scraper.get('https://github.com/nparamonov/SeleniumScraper')

Enable logging

SeleniumScraper uses the logging package. You can specify the logging level to see some entries

import logging
logging.basicConfig(level=logging.INFO)
# your code here

Example

...

Contributing

Installing

Clone the repository and navigate to the project directory

git clone https://github.com/nparamonov/SeleniumScraper.git
cd SeleniumScraper

Install Poetry (https://python-poetry.org/docs/#installation), e.g.:

pip install poetry

Install the dependencies:

poetry install

Activate the virtual environment:

poetry shell

Testing

Before PR, you should check the tests and add/edit them if necessary.

SeleniumScraper uses Pytest package.

pytest -v tests

Note that in order to successfully pass the tests, you must have Chrome and Firefox browsers installed on your machine.

You can also check pytest pipeline on GitHub Actions: pytest.yml, pytest workflow.

Coverage

Check code coverage

coverage run
coverage report
coverage html

License

This project is licensed under the terms of the Apache License 2.0.