There are a few websites that provide custom solutions to scrape a bunch of job related apis. However, most of these sites/platforms require a subscription of some sorts. The matter becomes worse, when the subscription is paid in dollars and a lot of countries have individuals with credit cards that do not allow dollar payments.
I decided to build a job scraper pooling data from the following SOURCES
| Tool | Version |
|---|---|
| Golang | ^1.23.4 |
| MySQL | ^5.7 |
| Chromium-Browser | 133.0.6943.53 |
-
Install dependencies in go
go mod tidy -
Run database migrations in mysql
-
Log in to mysql
mysql -u <username> -p -
Enter your password in the empty screen
-
Run the command
create database job_scraper; -
Run the migrations file whilst in the mysql shell
SOURCE ./sql/job_migration.sql
- Run the application
-
Create your own .env file from the .env.example
cp .env.example .env -
Start the app
go run .
-
Run the test within the package
go test -
Run all test recursively
go test ./... -
Run all test recursively with coverage
go test -coverpkg=./... ./... -
Run test with the verbose flag
go test -v