perf(v2): static parquet page buffer size

[kolesnikovae]

In this PR, I'm increasing the parquet buffers and making them static.

Currently, parquet buffer sizes are determined based on the table size, with lower and upper bounds of 64KB and 1MB, respectively. This approach helps prevent high memory consumption in the segment-writer, compaction-worker, and query-backend services, which typically handle many small tables concurrently. While this strategy works well in the common case – keeping memory usage low – it can negatively impact query performance in certain scenarios, and are generally overly conservative.

---

An example of what performance degradation looks like and how it can be identified.

Given a query trace with query-backend Invoke span (3ac41dbe32f4c111c67181c236628a7e). It spent 2.3s of CPU time and took 17s of the real time. The span profile:

Here we see that CPU time was spent in object storage i/o and parquet decoding (both amd64-avx2 and arm64-naïve implementations demonstrate comparable performance; the profile is collected on arm64).

The bottleneck is RepeatedRowColumnIterator, where we actually read pages – 52 events (the last one is summary):

The even log suggests that:

• Fetched around 50MB of data, ~4.5K profiles.
• Fetched 50 adjacent pages => the data range is contiguous.

It matches expectations: 1MB page, the data range is contiguous, the data chunk is of a manageable size (50MB). Although I find it weird that the number of rows per page varies this much.

The change won't resolve the problem: we trade memory for I/O throughput.

[kolesnikovae]

• The decoding performance might be suboptimal (read: the choice of encoding might be suboptimal). We should benchmark it.
• We may run into OOM issues because of the increased read buffer.
• We do not want to perform a remote call (Get) for every buffer read (refill).
• It would be nice to have column-specific buffers: many of them are tiny but we will retain a large buffer, utilizing memory inefficiently.
• We store and process way too much redundant data of low value.

[kolesnikovae]

It does make sense to increase the parquetReadBufferSize size further, e.g., 4MB. However, we need to estimate the actual impact on memory consumption.

I'm less confident about parquetPageWriteBufferSize.

[kolesnikovae]

I recently have discovered quite an interesting page access log:

25MB, ~40 page reads, 9/10 of fetched rows are discarded. For the given query ({service_name="x", profile_type="y"}), the access should be fully sequential, without gaps.

[kolesnikovae]

Regarding the previous comment on gaps in row ranges: this is expected due to the row ordering – first by series_id, then by timestamp. As a result, when the time range isn't fully covered, we must skip some rows for each series. Unfortunately, due to the way parquet-go is implemented, this almost always leads to a request to object storage – it's perfectly possible that a full scan is cheaper in this case. Also, see #2192: I would consider swapping the timestamp and series_id sort order: series_id, timestamp -> timestamp, series_id.

While investigating the issue, I discovered that #4036 causes parquet reader to perform SeekToRow in an extremely inefficient way: on every call, it scans the entire column chunk up to the specified row from the very beginning (even if advancing by just one row); #4036 is fixed in #4209 – enabling the index significantly improves performance in pathological cases with high read amplification.

Another interesting finding: parquet.List of structs colocates fields on the same page. This means that we should never create an individual iterator for each of the columns in such cases (example): in fact, we fetch same pages repeatedly.

I'll create issues.

[simonswine]

Regarding the previous comment on gaps in row ranges: this is expected due to the row ordering – first by series_id, then by timestamp. As a result, when the time range isn't fully covered, we must skip some rows for each series. Unfortunately, due to the way parquet-go is implemented, this almost always leads to a request to object storage – it's perfectly possible that a full scan is cheaper in this case. Also, see #2192: I would consider swapping the timestamp and series_id sort order: series_id, timestamp -> timestamp, series_id.

Theoretically we could also separate different profile_types into different row groups, each would of those we could order by timestamps, so we might be able to skip them fairly efficiently at edges, while still having the same profile types physically together. I also think the full scan idea sounds promising.

[simonswine]

LGTM! Thanks for tweaking the buffer sizes and the investigations/trialing around it.

I think we should make sure that we don't miss the ideas of that issue and create follow up tasks.