Ibis: Dataframe computing with pushdown operations to CrateDB

[amotl]

About

By adding an adapter to Ibis, people can use well-known dataframe-style computing code written in Python, but the mediation layer will leverage CrateDB operations instead of running it within Python/pandas/NumPy.

[amotl]

Superduper's SQL database interface is also based on Ibis, see also https://github.com/crate/zk/issues/15#issuecomment-2637820579.

• https://docs.superduper.io/docs/data_plugins/intro
• https://docs.superduper.io/docs/data_plugins/ibis/
• https://docs.superduper.io/docs/execute_api/vector_search_queries

[amotl]

xorq

When unlocking Ibis for CrateDB, we could possibly leverage quite a fragment of the DataFusion world fluently.

xorq, formerly LETSQL, is built on top of Ibis and DataFusion.

• https://github.com/xorq-labs/xorq

[amotl]

@shraik is also concerned about Ibis, and about CrateDB and ClickHouse too. ✨

[amotl]

SQLGlot

While previous versions of the PostgreSQL adapter for Ibis have been based on SQLAlchemy, it looks like contemporary versions use SQLGlot.

Thoughts

It looks like SQLGlot, SQLFrame, and SQLMesh are of the same friendly kind.

All Python dataframe libraries that are not Ibis (or SQLFrame) lock you into an execution engine.

References

• SQLMesh #176
• Aliasing column to "CHAR" does not work, while it works for all other reserved words crate#17108 (comment)
• SQLFrame: On the "all-types" validation, use char instead of character cratedb-examples#767
• SQLFrame: Add basic and all-types examples cratedb-examples#765