Why does the Parquet encoding list show both 'PLAIN' and 'BIT_PACKED'? Is BIT_PACKED actually used?

[whzruc]

Hi, I'm working with a Parquet file written via the Java Parquet library (v1.15.0). When I inspect the metadata using PyArrow, I see the following for a single INT32 column:

metadata <pyarrow._parquet.FileMetaData object at 0x7fdb18304860>
  created_by: parquet-mr version 1.15.0 (build 4665401d36e468c988322f78621da7c2d1c22ede)
  num_columns: 1
  num_rows: 1000
  num_row_groups: 1
  format_version: 1.0
  serialized_size: 259

column_name: test_field {'type': 'INT32', 'logical_type': <pyarrow._parquet.ParquetLogicalType object at 0x7fda6d7198f0>
  None, 'encodings': [('PLAIN', 'BIT_PACKED')], 'compression': ['UNCOMPRESSED'], 'num_values': [1000], 'row_group_sizes': [4103]}

'encodings': [('PLAIN', 'BIT_PACKED')]
However, my understanding is that for INT32 columns with no dictionary encoding and no RLE usage, only PLAIN should be relevant.

My questions are:

1. What does it mean when both PLAIN and BIT_PACKED appear in the encodings list?

2. Is BIT_PACKED actually used in this case (e.g., for definition levels or repetition levels)?

3. Is there a reliable way to check whether BIT_PACKED was applied to actual data, or just to the repetition/definition levels?

4. If the column is required and has no nulls, would any level encoding (BIT_PACKED or RLE) still apply?

Thanks in advance for any insights!

[pitrou]

cc @mapleFU

[mapleFU]

Would you mind upload an un-important data in this thread? Let me check about this file and tell why ( or you can wait me few ours and wait me home).

[willtemperley]

I was looking into this a bit and I'm fairly sure parquet-java 1.15.1 is writing def levels as BIT_PACKED by default. The attached file is a small test file I created with this version to confirm. I haven't inspected it at the byte level yet though.

bitpack-test.parquet.zip

[mapleFU]

@wgtmac do you know the format generation rule in parquet-java? I didn't check it a bit

[willtemperley]

I'm fairly sure this is happening in DevNullValuesWriter.

  public Encoding getEncoding() {
    return BIT_PACKED;
  }

It's not a big deal I guess, just a bit confusing and a few wasted bytes. I opened an issue anyway:
apache/parquet-java#3307

[wgtmac]

Just replied in the above issue. I think it is better to fix parquet-java not to add the encoding when levels are not physically written.

[willtemperley]

You have to look in the actual column data to find this - it's not in the footer FileMetadata.

A column chunk includes a list of pages, with an optional dictionary page first. Each of these, including the dictionary are prepended with an uncompressed PageHeader in Thrift format.

The DataPageHeader [1] in the PageHeaders that are data pages (PageHeader.type == DataPage) will have a DataPageHeader, in which you can find the encodings for rep/def levels.

I've noticed in Overture Maps parquet files which are also encoded with parquet-mr (aka parquet-java), BIT_PACKED is stated as the encoding for the definition levels when the definition levels are empty. This might be an encoding artifact, but I haven't confirmed it. Given that definition_level_encoding is a required metadata field, this is certainly a possibility.

[1] From the Parquet schema:

/** Data page header */
struct DataPageHeader {
  /**
   * Number of values, including NULLs, in this data page.
   *
   * If a OffsetIndex is present, a page must begin at a row
   * boundary (repetition_level = 0). Otherwise, pages may begin
   * within a row (repetition_level > 0).
   **/
  1: required i32 num_values

  /** Encoding used for this data page **/
  2: required Encoding encoding

  /** Encoding used for definition levels **/
  3: required Encoding definition_level_encoding;

  /** Encoding used for repetition levels **/
  4: required Encoding repetition_level_encoding;

  /** Optional statistics for the data in this page **/
  5: optional Statistics statistics;
}