Async Api

[cillierburger]

Are there any plans to support an async version of the API, or some kind of multi-threaded access ?

[auxten]

Hi @cillierburger
chDB could run in multi-threaded env, but you need to reuse the Session.
Could describe more details of your needs?

[cillierburger]

Hi @auxten ,

Sure, we have a tool that builds and runs ETL pipelines on top of clickhouse. This tool potentially runs multiple different queries "concurrently".

I'm exploring changing the adapter to use chDb, at first for running test cases. After that we would like to considering slicing out parts of our pipeline and running them using chDb.

So the idea would be that something like this:

tasks.append(session.query('SELECT sleep(2)'))
tasks.append(session.query('SELECT sleep(2)'))
tasks.append(session.query('SELECT sleep(2)'))

await asyncio.gather(*tasks)

Would complete in ~ 2s

I found the "parralel" test in the repository, but this just seems to prove that the lock is being used.

Happy to provide a more complete example if useful.

Thanks! 🙂

https://github.com/chdb-io/chdb/blob/main/tests/test_parallel.py