Improving integration with Java ecosystem

[westonpace]

Java has a mature and robust big data ecosystem. I'd like for Lance to have solid integrations with this ecosystem. We have done some work in Spark already and we are continuing to improve our Spark connector. Can we gather ideas of things we would like to see here and then we can fill out issues for these integration points (and maybe an epic tracking ticket in general)?

[lsyldliu]

Big +1, I think the integration with Java ecosystem is a great work, it will help many big data engine read and write Lance, make Lance more and more popularity. I can help to do some integration work.

[majin1102]

Some ideas in my mind. Will supplement as needed:

1. Catalog has become the de facto standard for metadata integration in big data compute engines. Beyond enabling compute engines to rapidly integrate with existing metadata systems, it also supports permission-based architectures and data sharing solutions. Moreover, enterprises can leverage Catalog to build service layers for data lake management, including asynchronous operations such as compaction, clean, branch lifecycle management, metrics.
2. Developers of big data always use a distributed engine like spark to manipulate data(mostly use sql). How to do an 'append column' or ‘merge column’ operation is quite confusing on a SQL-like interface(cause SQL is inherently row-oriented as well as DataFrame in Spark). There are some solutions using UDF. I don't know if we can figure out a more stardard way to do this.

[majin1102]

The missing features in Java Dataset API are(to be added):

• ​​Version control​​
  • usage
    • lake management service written in JAVA
    • metadata table like https://www.tabular.io/apache-iceberg-cookbook/basics-query-metadata/
  • current
    • lack Version and Tag definition
    • version, latestVersion returns version ID
  • new api intruduced in Dataset of JAVA(aligned with Python and Rust API)
    • listVersions: list all available version objects
    • listTags: list all available tag objects
    • checkout: return a dataset pointing to a certain version or tag
    • checkoutLatest: checkout latest version and do not return another dataset
    • restore: rollback to the current pointed version
• ​​configuration management​​
  • usage
    • DDL by Spark/Flink/Trino to update table properties
    • interface to handle field metadata
• ​​transaction support​

[majin1102]

Epic issue: #3950