Skip to content

Commit 870971d

Browse files
authored
feat!: remove Spark module in favor of the lance-spark repo (#3700)
All Spark connector code is now maintained in https://github.com/lancedb/lance-spark
1 parent 2b2fb3f commit 870971d

File tree

106 files changed

+6
-5090
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

106 files changed

+6
-5090
lines changed

java/README.md

Lines changed: 6 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -194,26 +194,22 @@ void dropColumns() {
194194
}
195195
```
196196

197-
## Integrations
197+
## JVM Engine Connectors
198198

199-
This section introduces the ecosystem integration with Lance format.
200-
With the integration, users are able to access lance dataset with other technology or tools.
199+
JVM engine connectors can be built using the Lance Java SDK. Here are some connectors maintained in lancedb Github organization:
201200

202-
### Spark connector
203-
204-
The [spark](https://github.com/lancedb/lance/tree/main/java/spark) module is a standard maven module.
205-
It is the implementation of spark-lance connector that allows Apache Spark to efficiently access datasets stored in Lance format.
206-
More details please see the [README](https://github.com/lancedb/lance/blob/main/java/spark/README.md) file.
201+
* [Spark Lance connector](https://github.com/lancedb/lance-spark)
202+
* [Flink Lance connector](https://github.com/lancedb/lance-flink)
203+
* [Trino Lance connector](https://github.com/lancedb/lance-trino)
207204

208205
## Contributing
209206

210207
From the codebase dimension, the lance project is a multiple-lang project. All Java-related code is located in the `java` directory.
211208
And the whole `java` dir is a standard maven project(named `lance-parent`) can be imported into any IDEs support java project.
212209

213-
Overall, it contains two Maven sub-modules:
210+
Overall, it contains one Maven sub-module:
214211

215212
* lance-core: the core module of Lance Java binding, including `lance-jni`.
216-
* lance-spark: the spark connector module.
217213

218214
To build the project, you can run the following command:
219215

java/pom.xml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,6 @@
5757

5858
<modules>
5959
<module>core</module>
60-
<module>spark</module>
6160
</modules>
6261

6362
<scm>

java/spark/README.md

Lines changed: 0 additions & 79 deletions
This file was deleted.

java/spark/src/main/java/com/lancedb/lance/spark/LanceCatalog.java

Lines changed: 0 additions & 93 deletions
This file was deleted.

java/spark/src/main/java/com/lancedb/lance/spark/LanceConfig.java

Lines changed: 0 additions & 137 deletions
This file was deleted.

java/spark/src/main/java/com/lancedb/lance/spark/LanceConstant.java

Lines changed: 0 additions & 19 deletions
This file was deleted.

0 commit comments

Comments
 (0)