Schema error: Invalid data type for Delta Lake: Null

[resulyrt93]

Environment

Delta-rs version: 0.14.0

Binding: Python

Environment:

• Cloud provider: AWS and Local
• OS: Linux and MacOS
• Other:

---

Bug

What happened: When i try to write my dataframe that has full null column, it raises error. If all rows are None in a column, it seems as null type in pyarrow schema and it causes error because i guess delta-rs doesn't recognize that type.

What you expected to happen: I would expect that create/write delta table.

How to reproduce it:

data = [['kevin', 10, None], ['angela', 12, None], ['oscar', 9, None]]
df = pd.DataFrame(data, columns=['Name', 'Age', 'Gender'])

write_deltalake('/my_table_path', df)

[ion-elgreco]

Null is not a valid primitive type https://github.com/delta-io/delta/blob/master/PROTOCOL.md#primitive-types, you can create the table first with DeltaTable.create("path", pa.schema)

[ion-elgreco]

Closing this since it's not a bug.

I suggest you create the table first with the specified schema where you make the gender column string or cast your data to the correct schema first.

[FrankPortman]

This is happening even when a table with the correct schema was created first. It cannot be coerced correctly for v1.1.0.