add support for convert_to_deltalake from parquet

[djouallah]

it will be nice to support convert_to_deltalake from parquet, even a one off is good enough

https://delta-io.github.io/delta-rs/api/delta_writer/#convert-to-delta-tables

[linhr]

Thanks for the feature requests! We'll need to see how this operation fit into our extended DataFrame API or SQL syntax.

This seems a metadata-only operation. No data is copied, and only the transaction log is written. The Delta Python library can probably handle large datasets for this without scalability issues. Are there use cases that make you feel that Sail can offer an improvement for this conversion?

[djouallah]

nah, no case at all, just less dependency, not big deal :)

%%time
import pathlib
schema = f'CH{SF:04d}'
tpch_tables = ['nation','region','customer','supplier','lineitem','orders','partsupp','part']
Table_Path = '/lakehouse/default/Tables'
lineitem_parts = max(2, int(200 * (SF / 1000)))
orders_parts   = max(2, int(140 * (SF / 1000)))
partsupp_parts = max(1, int(80 * (SF / 1000)))
part_parts     = max(1, int(12 * (SF / 1000)))
customer_parts = max(1, int(20 * (SF / 1000)))
BASE_OUTPUT_DIR = f"/lakehouse/default/Tables/{schema}"

shell_script = f"""
set -e

SCALE_FACTOR={SF}
BASE_OUTPUT_DIR="{BASE_OUTPUT_DIR}"
declare -A TABLES_PARTS=(
    ["lineitem"]={lineitem_parts}
    ["orders"]={orders_parts}
    ["partsupp"]={partsupp_parts}
    ["part"]={part_parts}
    ["customer"]={customer_parts}
    ["nation"]=1
    ["region"]=1
    ["supplier"]=1
)
echo "--- Starting TPC-H Data Generation ---"
echo "Scale Factor: ${{SCALE_FACTOR}}"
echo "Base Output Directory: ${{BASE_OUTPUT_DIR}}"
echo ""

for table_name in "${{!TABLES_PARTS[@]}}"; do
    num_parts_to_generate="${{TABLES_PARTS[${{table_name}}]}}"
    echo "--- Generating data for table: ${{table_name}} (${{num_parts_to_generate}} parts) ---"
    echo "  -> Generating ${{table_name}} part ${{i}}/${{num_parts_to_generate}} "
        tpchgen-cli \\
            -s "${{SCALE_FACTOR}}" \\
            --tables "${{table_name}}" \\
            --output-dir "${{BASE_OUTPUT_DIR}}" \\
            --parts "${{num_parts_to_generate}}" \\
            --format="parquet" \\
            --parquet-row-group-bytes=268435456
    done
    echo "--- Finished generating data for table: ${{table_name}} ---"
echo "--- TPC-H Data Generation Complete! ---"
"""
if not pathlib.Path(f'{Table_Path}/{schema}').exists():
    !pip install -q tpchgen-cli
    !pip install deltalake
    from   deltalake  import convert_to_deltalake
    print('generate new data')
    get_ipython().system(shell_script)
    for tbl in tpch_tables:
      convert_to_deltalake(f'{BASE_OUTPUT_DIR}/{tbl}',storage_options={"allow_unsafe_rename":"true"})

[linhr]

For future reference, this SQL statement seems the user interface that we can follow: https://docs.databricks.com/aws/en/sql/language-manual/delta-convert-to-delta