Skip to content

This is a test app to help scrape a website with a login and specific elements.

Notifications You must be signed in to change notification settings

Compukid/web_scraper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

10 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Web Scraper

This project was created with chatgpt. The python script scrapes data from a specific URL app.smartoilgauge.com. The URL, Username and password are configurable with a .env file. The script creates an api on port 5000 that will trigger the scrap and be able to pull the data from. This can be run directly on a linux computer or as a docker image --> https://hub.docker.com/repository/docker/compukid/smartoilgauge-scraper/general

Setup

  1. Clone this repository:

    git clone https://github.com/yourusername/your-repository-name.git
  2. Install dependencies:

    pip install -r requirements.txt
  3. Set up a .env file with the following variables:

    WEBSITE_URL=https://url.com/page
    USER_NAME=your_username
    PASSWORD=your_password
    
  4. Run the scraper:

    python scraper.py

License

MIT

About

This is a test app to help scrape a website with a login and specific elements.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages