The manifest has recorded the WriterVersion, like this:

writer_version: Some(
        WriterVersion {
            library: "lance",
            version: "0.31.2",
        },
),

I would like to suggest checking the SDK compatibility. There are two cases I can share:

• We met an issue with our customer(there are different teams writing and reading Lance data independently). Once, the writing team used a higher version, but the reading team used a lower version, which caused an error due to perf: use size prefixing for zstd buffer compressor for better decompression performance #4028 .

• Another case, a customer's Python version is less than 3.9, and they ran pip install pylance, it would install 0.10.x. But it is not easy to find.

We can do two things:

• Check the reader SDK, if its version is less than the writer version recorded in the reading manifest. If yes, print a warning log.
• We could introduce a compatibility matrix if we allow a looser reader/writer compose strategy in the future. Considering an upgrade to the reader SDK is not a casual decision(it may bring new issues or bugs).