Skip to content

Add streaming query #28

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 16, 2025
Merged

Add streaming query #28

merged 4 commits into from
May 16, 2025

Conversation

s0und0fs1lence
Copy link
Contributor

This pr is needed for adding streaming support for chdb (326)
I introduced a new driver type called PARQUET_STREAMING which can be used to load only the relevant chunks in memory.

I also changed the driverType interface for being able to support more streaming interfaces if needed

The stream result is just a Wrapper that will fetch the NextResult when needed.

@auxten
Copy link
Member

auxten commented Apr 28, 2025

@wudidapaopao pls help check the pipeline issue, seems related to the streaming API

@s0und0fs1lence
Copy link
Contributor Author

@wudidapaopao can confirm it's something related with the library itself. now just removing a comment and thus re-running the action, gives no error

@wudidapaopao
Copy link
Contributor

Hi @s0und0fs1lence, could you please advise on how to reproduce the failed case? I've tried running it multiple times locally but haven't been able to reproduce.

@s0und0fs1lence
Copy link
Contributor Author

Hi @wudidapaopao, I could not reproduce the issue in local neither myself, like you can see from the 3 actions run, it failed randomly on 2 different tests (that i didn't touch)
I Guess the problem was there even before the release, but we didn't caught it.

@wudidapaopao
Copy link
Contributor

Hi @s0und0fs1lence, the current libchdb.so appears to have a certain probability of crashing in the macOS 15 environment on GitHub Actions. To ensure the stability of our CI pipeline, I will temporarily roll back to the macOS 13 environment for now.
#29

@s0und0fs1lence
Copy link
Contributor Author

Hi @wudidapaopao, thanks for the help. Yeah i saw it even in local development, sometimes my mac with Sonoma got some strange crashes.

I think we can inspect the issue in a separate pr.

@wudidapaopao
Copy link
Contributor

wudidapaopao commented May 16, 2025

Hi @s0und0fs1lence, I created a separate issue and continue tracking it. I have added some additional information in the newly created issue. #30
For now, let's merge this PR.

@wudidapaopao wudidapaopao merged commit 08f247d into chdb-io:main May 16, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants