Can we remove the "dependencies" folder from the python package?

[debrando]

I see that since long time in the final package is included this:

[tool.maturin]
include = [
   (...)
   { path = "connectorx/dependencies/", format = "sdist"}
]

...which brings on the target environment, this folder:

 `/lib/python3.9/site-packages/connectorx/dependencies`

...which contains only a couple of java packages:

ls -R dependencies 
federated-rewriter.jar jassets

dependencies/jassets:
j4rs-0.15.3-jar-with-dependencies.jar

These:

• ...seems not being used at all, I tried manually deleting them and the read from table works.
• ...brings various HIGH CVEs on the final docker image.

So, I'm wondering: what's the purpose, and can we remove the above from the list?

[wangxiaoying]

Hi @debrando ,

The dependencies folder includes dependency jar files for query federation (e.g., issue a query that joins tables from multiple databases). Like this following example:

import connectorx as cx
db1 = "postgresql://username1:password1@server1:port1/database1"
db2 = "postgresql://username2:password2@server2:port2/database2"
cx.read_sql({"db1": db1, "db2": db2}, "SELECT * FROM db1.nation n, db2.region r where n.n_regionkey = r.r_regionkey")

This feature requires a query rewriter, which is a prototype implemented in Java (using Apache Calcite) and is currently experimental. I'm considering to decouple this part from the connectorx repo next, and reimplement it in rust in the future.