should i bother trying "self-hosting my solution"?

[suffering-git]

the proxy solution ends up being kinda slow along with the added cost.

my script burned 928 units of quota for the youtube data api in an ~8 hour session, with the proxy active. It take like 5 seconds average to download one transcript. I had a 0.6s sleep between every ytt_api.fetch() call and every youtube data api call.

Seems like there's a reasonable chance I can ditch the proxy, self-host, and perhaps achieve free, faster rates?

I would of course still rate limit myself to probly nothing less than 2 requests/fetch calls per second.

are IP bans a permanent thing? kinda scary lol idk how any of it works though, im pretty out of place here, just hacking something together for market research😝bless the almighty power of ai code generation🙌

[jdepoix]

Unless you're self-hosting your proxy (which would require you to have access to a large pool of residential IPs), self-hosting is not going to solve the need for proxies for you. Your self-hosted solution will still have an IP address, which will be blocked by YouTube sooner or later. If you just want to decrease the proxy latency, you can check out this comment here, where I gave some advice on how to speed things up. However, there will always be additionally latency introduced by using proxies and there certainly is no way around the cost that comes with it. Operating proxies isn't free after all (even if you were to do it yourself) 🤷‍♂️

[suffering-git]

hmm can't async techniques solve speed quite easily? webshare does say they have a 500 concurrency default request limit or w/e, i would assume that's what that limit is for

[suffering-git]

crazy how hard it is to scale such a simple thing even just a little bit though😳respect to the professionals

[jdepoix]

Doing async requests will only make it easier to do multiple requests concurrently, than it would be using multithreading. But it won't make the individual requests any faster. However, this is completely unrelated to whether you self-host your solution or not.