Parquet row_group_size has confusing units

[ThermodynamicBeta]

For parquet file writing, there is a row_group_size parameter:
https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html

The usage and first sentence of the docstring seem to imply that its supposed to be the number of rows:

However, it also mentions 1024 * 1024 and 64Mi, which suggests its units are bytes. From a user perspective, I don't know how big my rows are, it would be much more convenient to put a number of bytes here. Can someone explain this decision, and confirm if its supposed to be the number of rows?

[amoeba]

Hi @ThermodynamicBeta, thanks for starting this discussion. The units are in terms of rows and I believe the docstring was written with that knowledge and intent. You can see this by looking at #34280 and #34435.

...it also mentions 1024 * 1024 and 64Mi, which suggests its units are bytes.

Do you say that "Mi" suggests bytes because Mi is a base 2 prefix and we don't tend to use base 2 when counting? I want to make sure I understand your take on this since others are likely to be confused too.

I think its current use here is correct and would be even more correct if we used MiRows (mebirows). Better even would be a small rewrite to avoid the confusion entirely.

Let me know on the above question and, additionally, if you'd be interested in filing a PR with an improvement. Otherwise, I'd be happy to have a go.

[ThermodynamicBeta]

More or less, e.g. the other day I wrote

file_size_gb = file_path.stat().st_size / (1024**3)

I guess when I see Mi, I assume MiB because I haven't seen it anywhere else. Saying MiRows would definitely help someone like me in the future

[amoeba]

I'll file a PR to update the wording. I think we can get away with just giving the number without hoping the reader is comfortable with mebi-rows.

[amoeba]

PR made, could you have a look and let me know if it goes far enough to clear it up? Feel free to comment there or here. #46653