So I am running a script for the last 18 hours (debug mode) that hits the API with webshare - any ideas of faster processing?

[saikat-crux]

Hi folks - I am so grateful for this API.
I am running a script that gets a video id from a valkey db, calls the youtube-transcript-api over a webshare rotating residential proxy, and calls yt-dlp for metadata, then creates a .json file and saves it on disk, and uploads to Google Drive (using gdrive API), if everything is successful it deletes the file on the local disk. I am running batches of 50 videos (with a 1 min delay between each batch run). This has been going for the last 18 hours or so and has downloaded transcripts for over 6000 videos. That is like 10.8s per video processed (including the delays etc.)

I was wondering if I can do something to speed up the Webshare proxy call. I think that is not the main performance drop (probably the disk i/o and gdrive upload are) area in this workflow, but still feels the proxy is slowing it down.

Any suggestions?

[jdepoix]

I left a comment here on how to speedup requests proxied by Webshare!