Add Asyncio and ProcessPoolExecutor

[zadamg]

I was thinking of making a pull request for this; I can add it myself, but it would be amazing to add asyncio and ProcessPoolExecutor. Anyone deep enough to use such an awesome module like this is probably going to be wanting it asynchronous and parallel.

Thoughts? Planned? Open to contribution?

thanks for this! ♥

[jamesturk]

I'm open to adding it. To be clear on what you're interested in, a way to call the GPT model async? I'm working on a related yet-to-be-released project right now that needs async/rate limiting so I might wind up adding some support there.

[zadamg]

Correct. I'd add asyncio to gather the html results and.... actually now that I'm awake and had coffee.. the ProcessPool isn't even relevant cuz we're not parsing anything ourselves; just calling apis.. so it's just asyncio I guess? Ultimately, we're gonna be limited heavily by OpenAI, so maybe this all moot. 🙃 I just want to go faster if, for example, I want to scrape 10 totally different sites at one time.

Maybe there's an py03 module that could be built that offers a first-pass parses the HTML down significantly before sending it to the API and it might be faster and cheaper.

[jamesturk]

Yep, the OpenAI limits are a big factor.

As for stripping down the HTML, That's definitely the most useful missing piece IMO, I am bike-shedding a bit here but I want a decent sized test corpus before tackling that aggressively since it'll be important to measure how aggressive we can be before losing quality.

If its interesting to you, definitely would love to collaborate!

[zadamg]

Parsing big HTML or XML strings with rust pattern matching is stupid-fast. Like 10000 RSS feeds worth of articles in 1-3 secs fast. You specify tags, it gets the content (simplifying), so seems like a module could be written that matches all possible known tags, strips content, and returns a json object to python of the tags and their content or something and it would probably get 90% of the way there. (exploring this myself to try to archive reddit pre-apocalypse, since everything is a .rss feed).