Skip to content

[FEA] Build against $SPARK_HOME jars instead of Maven #12820

@gerashegalov

Description

@gerashegalov

Is your feature request related to a problem? Please describe.

Occasionally we need to re-build to debug the RAPIDS Plugin in environments without access to Maven repos but with an intact SPARK_HOME jars directory. This is similar to the situation with Databricks images

Describe the solution you'd like
This issue proposes to generalize the approach started in #12775 by adding the ability to build against the jars under $SPARK_HOME/jars such as /databricks/jars in the DBR case. It improves on the existing approach to install-file hand-picked set of jars into a local Maven repo (~/.m2/repository). Instead of invoking Maven to install-file first we can blindly register all jars in a shim-deps BOM as fake system-scope dependencies defined com.nvidia.spark_home:jar-1

Metadata

Metadata

Assignees

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions