Ibis Table to pyspark df

[sonic-x11]

I have created an Ibis connection using the PySpark backend, created a table, and applied some filters. Now, I want to get the result as a PySpark DataFrame, but the available options for conversion are Polars, Arrow, and Pandas DataFrames. Thank you!

[gforsyth]

Hey @sonic-x11 -- that seems like a good feature request if you'd like to open an issue.

In the interim, here is a (hacky) workaround:

If you have your expression variable t which is the result of the table creation + applied filters, and you have an active connection object con, then you can do the following:

spark_df = con._session.sql(con.compile(t))

That spark_df will also be lazy -- I'm rusty on pyspark but I think you call collect on it to materialize.

[sonic-x11]

Thanks @gforsyth for the reply, I'll raise a feature request
Thank you