How to enable proxy support?

[FOUNDATIONAIBASED]

How to enable in the docker version proxy tunnel support for socket it says: Missing dependencies for SOCKS support. and for http i always get a different error i used this list for proxies: https://geonode.com/free-proxy-list

[dosubot[bot]]

Hi @FOUNDATIONAIBASED! I'm Dosu and I’m helping the WaterCrawl team.

To enable SOCKS proxy support in the Docker version, you need to install the required Python packages for SOCKS proxies. Specifically, add requests[socks] and httpx[socks] to your requirements and rebuild your Docker image. This will install PySocks (for requests) and socksio (for httpx), resolving the "Missing dependencies for SOCKS support" error [source].

HTTP proxies should work out-of-the-box, but if you get errors, it may be due to unreliable or dead proxies from public lists like geonode.com. Make sure the proxies you use are active and support the required protocols.

Proxy configuration is managed in the WaterCrawl UI/backend and passed automatically to crawling jobs. If you need more help with HTTP proxy errors, please share the exact error message you're seeing.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[amirasaran]

Hey @FOUNDATIONAIBASED — thanks for starting the discussion!

It sounds like you're running into a dependency issue with SOCKS proxy support in the Docker setup. Could you please open a bug report with more details about your environment and the exact errors you're seeing (both for SOCKS and HTTP)? That’ll help us or the community investigate and work toward resolving it in the next release of Watercrawl.

Looking forward to your report!