Lance Table and Column Level Governance

[jackye1995]

Governance is usually a catalog level feature, and typically comes with some sort of lock-in to use a specific catalog + specific engines that can respect the access policies in the catalog.

With the unique design of Lance, we can actually achieve table and column level governance even without a catalog. Because Lance can store different columns in different files, as long as we can set different encryption keys for different data files and metadata files, users can just configure access to those encryption keys to control access to different columns in the same table.

For example, consider a table with columns (c1, c2, c3, ssn). A user can configure:

• column groups g1 = [c1, c2, c3] and g2=[ssn] (see more details in the column group discussion: Column group with horizontal compaction and split #3995)
• encryption key k1 to encrypt g1 and encryption key k2 to encrypt g2

Technically, Parquet also have similar feature of column level encryption, but because all columns must be stored in the same Parquet file, this makes the Parquet spec quite complicated. Users have to upgrade to use it, and all the readers, writers, related table formats and engines have to be updated to support it, thus it becomes much harder to gain very wide ecosystem adoption (personal opinion :P). Compared to that, Lance just needs to add the ability for users to set the key, and ensure we pass the key information to the object storage layer (e.g. S3 SSE-KMS). There is no additional handling needed at the file format level.

[wjones127]

There is no additional handling needed at the file format level

That's a cool idea!