-
Notifications
You must be signed in to change notification settings - Fork 194
Description
First of all, god bless you for this project. I was about to start my own scraper/crawler project with a similar approach as you, when my friend pointed me to this project. So heads up.
Context
I am running python 3.10.6
with pipenv shell
on Windows 10 22H2.
I ran the config wizard. I setup the usual suspects: Immowelt, WG-Gesucht, KA, etc... I even found and added the reese84
cookie without any issues.
My preferred approach would be to Dockerize it and run it as a background process on startup.
Before doing so I wanted to check if everything works fine with pytest
.
Here are some Issues I encountered, and I wanted to report as a good semeritan.
Issues
First it could not find the package exceptiongroup
eventhough this is listed as dependency in your Pipfile
and I of course ran the pip install -e .
command preemptively. I had to install this package on my own (might be something you want to look into).
Now the main issue: My testrun resulted in a buttload of failures. I tried to have a quick glance at the error log (which I attached here).
I can sum up these errors as follows: Permission Errors (I ran it from VS Code integrated, and Powershell in Admin mode), Operational Error (sql3 unable to find database file), No Windows Console found (I am running in PS), and a ValueError on Kleinanzeigen...
I have attached both the test-report from PS (without admin) and PS (with admin privileges). Both exhibit the same problems.
Theories and Recommendations
- Have a look at
exceptiongroup
dependencies (if any one else encounters this problem) - Am I missing some key aspect here, for the database problem? Should I setup the Docker container and run the testsuite from in there?
- I have not used Flask before.
- Might be useful to give a little disclaimer for Windows Users in the readme about the possibility of having permission errors, and to run it in admin mode instead.