bug: support nullability in struct

[eddyxu]

Currently, lance does not allow null in struct. Be able to serialize null structs on disk in format v2.1

import lance
import pyarrow as pa

schema = pa.schema(
    [
        pa.field("id", pa.int64()),
        pa.field(
            "point",
            pa.struct(
                [
                    pa.field("x", pa.float64(), nullable=True),
                    pa.field("y", pa.float64(), nullable=True),
                    pa.field("z", pa.float64(), nullable=True),
                ],
            ),
            nullable=True,
        ),
    ]
)


tbl = pa.Table.from_pylist([{"id": 5, "point": None}], schema=schema)
print("origin: ", tbl.to_pydict())

lance.write_dataset(tbl, "empty.lance", schema=schema, data_storage_version="2.1")
ds = lance.dataset("empty.lance")
tbl = ds.to_table()
print(tbl.to_pydict())

Output

origin:  {'id': [5], 'point': [None]}
[2025-08-04T15:35:35Z WARN  lance_file::v2::writer] You have requested an unstable format version.  Files written with this format version may not be readable in the future!  This is a development feature and should only be used for experimentation and never for production data.
{'id': [5], 'point': [{'x': 0.0, 'y': 0.0, 'z': 0.0}]}

[westonpace]

This must be getting added in the fragment readers somewhere. If I just read the file directly I get null. For example, adding this to the repr script:

frag = ds.get_fragment(0)
path = frag.data_files()[0].path

from lance.file import LanceFileReader

reader = LanceFileReader(os.path.abspath(f"empty.lance/data/{path}"))
tbl = reader.read_all().to_table()

print(tbl.to_pydict())

[Xuanwo]

I've set up a test to reproduce the issue and will work on fixing it tomorrow.

[Xuanwo]

Fixed:

origin:  {'id': [5], 'point': [None]}
[2025-08-05T08:48:14Z WARN  lance_file::v2::writer] You have requested an unstable format version.  Files written with this format version may not be readable in the future!  This is a development feature and should only be used for experimentation and never for production data.
{'id': [5], 'point': [None]}

PR is coming