feat: support DataFrameWriter `path` option

[keen85]

This works fine in PySpark but does not in Sail:
df.write.format("parquet").option("path", some_path).save()
Sail fails with error IllegalArgumentException: missing argument: save type

Workaround:
df.write.format("parquet").save(path=some_path)

path option is also important for creating external hive tables (where physical location deviates from spark.sql.warehouse.dir:
df.write.format("parquet").option("path", some_path).saveAsTable("foo")
When executing this in Sail, path option is ignored and table is saved with location spark.sql.warehouse.dir.

See

• https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrameWriter.html?highlight=dataframewriter#pyspark.sql.DataFrameWriter
• https://sparkbyexamples.com/pyspark/pyspark-save-dataframe-to-hive-table/

[linhr]

Thanks for reporting this!

Yeah the path option is not supported yet. I can see that this option would be valuable despite of the workaround. We'll figure it out to improve the user experience of writing to path/table.

[shehabgamin]

The path option is now supported in #833 (review).

Further work is needed to determine how conflicting locations should be handled when writing to an existing table.