Skip to content

Convert JSON to VariantArray without copying (8 - 32% faster) #7911

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Jul 23, 2025

Conversation

alamb
Copy link
Contributor

@alamb alamb commented Jul 11, 2025

Which issue does this PR close?

Rationale for this change

In a quest to have the fastest and most efficient Variant implementation I would like to avoid copies if at all possible
Right now, to make a VariantArray first requires completing an individual buffer and appending it
to the array.

Let's make that faster by having the VariantBuilder append directly into the buffer

What changes are included in this PR?

  1. Add VariantBuilder::new_from_existing
  2. Add a VariantArrayBuilder::variant_builder that reuses the buffers

Are these changes tested?

  1. New unit tests
  2. Yes by existing tests

Are there any user-facing changes?

Hopefully faster performance

Copy link
Contributor

@scovich scovich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is remarkably simpler than I had imagined it would need to be. Handing off ownership back and forth was a very useful trick.

My only concern is whether we might ever need to support a builder that isn't backed by Vec? I'm guessing not, but wanted to double check.

@alamb
Copy link
Contributor Author

alamb commented Jul 12, 2025

My only concern is whether we might ever need to support a builder that isn't backed by Vec? I'm guessing not, but wanted to double check.

I think eventually we might, but I think the only way to really do so is via some sort of trait and a templated builder. I think we can pretty far without doing so and Vec seems to be the best / fastes / etc thing for managing owned memory in Rust

And there are zero copy APIs to/from Vec for the underlying Arrow arrays which I think is a pretty nice property too

@alamb alamb force-pushed the alamb/append_variant_builder branch from bb1502a to e07069d Compare July 12, 2025 20:31
alamb added a commit that referenced this pull request Jul 13, 2025
… buffers (#7912)

# Which issue does this PR close?

- closes #7805
- part of #6736
- part of #7911

# Rationale for this change

I would like to be able to write Variants directly into the target
buffer when writing multiple variants However, the current
VariantBuilder allocates a new bufffer for each variant

# What changes are included in this PR?

1. Add `VariantBuilder::new_with_buffers` and docs and tests

You can see how this API can be used to write directly into a buffer in
VariantArrayBuilder in this PR:
- #7911

# Are these changes tested?
Yes new tests

# Are there any user-facing changes?
New API
@alamb alamb force-pushed the alamb/append_variant_builder branch from e07069d to 8166cb6 Compare July 13, 2025 12:08
@github-actions github-actions bot removed the parquet Changes to the parquet crate label Jul 13, 2025
@alamb alamb force-pushed the alamb/append_variant_builder branch from 8166cb6 to e10d41d Compare July 13, 2025 12:29
@github-actions github-actions bot added the parquet Changes to the parquet crate label Jul 13, 2025
@alamb
Copy link
Contributor Author

alamb commented Jul 13, 2025

Update here is I think I have incorporated @scovich's comments and I am quite pleased with how it is looking

I think this code needs a few more tests and a benchmark or two and we'll be good.

I'll try and work on those in the next few days

@alamb alamb self-assigned this Jul 14, 2025
impl<'a> Drop for VariantArrayVariantBuilder<'a> {
/// If the builder was not finished, roll back any changes made to the
/// underlying buffers (by truncating them)
fn drop(&mut self) {
Copy link
Contributor

@scovich scovich Jul 14, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really like this approach. I was thinking over the weekend that we may want to rework the other builders to follow a similar approach:

  • They can truncate the metadata dictionary on rollback, which would eliminate the false allocations that survive a rollback today
  • We can allocate the value bytes directly in the base buffer (instead of using a separate Vec)
    • On rollback, just truncate (like here)
    • On success, use Vec::splice to insert value offset and field id arrays, which slides over all the other bytes
  • Once we're using splice, it opens the door to pre-allocate the space for the value offset and field arrays, in case the caller knows how many fields or array elements there are.
    • If the prediction was correct, splice just replaces the pre-allocated space.
    • If incorrect, the pre-allocation is wasted (but we're no worse off than before -- the bytes just inject in)
    • The main complication would be guessing how many bytes to encode each offset with.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They can truncate the metadata dictionary on rollback, which would eliminate the false allocations that survive a rollback today

That is an excellent point

We can allocate the value bytes directly in the base buffer (instead of using a separate Vec)

That sounds like a great way to avoid the extra allocation

Once we're using splice, it opens the door to pre-allocate the space for the value offset and field arrays, in case the caller knows how many fields or array elements there are.

This is also a great idea 🤯

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a follow up, @klion26 has a PR up to implement this:

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That other PR is nice improvement, but the splice call still shifts bytes.

In order to not shift bytes at all, we'd have to pre-allocate exactly the right number of header bytes before recursing into the field values. And then the splice call would just replace the zero-filled header region with the actual header bytes, after they're known (shifting bytes only if the original guess was incorrect).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the only way to do this is add some API in the ObjectBuilder somehow to pre-allocate this space (new_object_with_capacity() perhaps 🤔 )

@alamb
Copy link
Contributor Author

alamb commented Jul 14, 2025

I added some benchmarks and my local results suggest that avoiding the allocations makes parsing small repeated json objects about 10% faster.

I think once we stop copying stuff around in the sub builders, the other bencmarks will be quite a bit faster too

Gnuplot not found, using plotters backend
small_repeated_json 8k string
                        time:   [5.3628 ms 5.3743 ms 5.3862 ms]
                        change: [−11.062% −10.867% −10.654%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

Benchmarking small_random_json 8k string: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.2s, or reduce sample count to 60.
small_random_json 8k string
                        time:   [71.132 ms 71.245 ms 71.364 ms]
                        change: [−1.7432% −1.4915% −1.2496%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  9 (9.00%) high mild
  1 (1.00%) high severe

@@ -1047,16 +1047,16 @@ impl Drop for ObjectBuilder<'_> {
///
/// Allows users to append values to a [`VariantBuilder`], [`ListBuilder`] or
/// [`ObjectBuilder`]. using the same interface.
pub trait VariantBuilderExt<'m, 'v> {
fn append_value(&mut self, value: impl Into<Variant<'m, 'v>>);
pub trait VariantBuilderExt {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no reason for the lifetimes to be attached to the trait itself -- if it is that means that the lifetimes trickle into the values -- since this trait is for actually constructing variant values (and copying the underlying bytes) I moved the lifetimes to just the arguments that need it

// TODO make this more efficient by avoiding the intermediate buffers
let mut variant_builder = VariantBuilder::new();
variant_builder.append_value(variant);
let (metadata, value) = variant_builder.finish();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole point of this PR is to avoid this copy here and instead write directly into the output

@alamb alamb force-pushed the alamb/append_variant_builder branch from 30ad86b to 3b6aef6 Compare July 16, 2025 20:54
alamb added a commit that referenced this pull request Jul 18, 2025
# Which issue does this PR close?

We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax.

- Part of #7911
- Part of  #6736
- Follow on to #7905


# Rationale for this change

I wrote benchmark some changes to the json decoder in
#7911 but they are non trivial.
To keep #7911 easier to review I
have pulled the benchmarks out to their own PR

# What changes are included in this PR?

1. Add new json  benchmark
2. Include the `variant_get` benchmark added in
#7919 by @Samyak2
# Are these changes tested?

I tested them manually and clippy CI coverage ensures they compile

# Are there any user-facing changes?

No these are only benchmarks
@alamb alamb force-pushed the alamb/append_variant_builder branch from 3b6aef6 to 92e5c12 Compare July 18, 2025 15:29
json_to_variant(input_string_array.value(i), &mut vb)?;
let (metadata, value) = vb.finish();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole point if this PR is to avoid this copy / append

@alamb

This comment was marked as outdated.

@alamb

This comment was marked as outdated.

@alamb alamb force-pushed the alamb/append_variant_builder branch from 92e5c12 to fce19eb Compare July 18, 2025 15:42
@@ -55,9 +55,14 @@ use std::sync::Arc;
/// };
/// builder.append_variant_buffers(&metadata, &value);
///
/// // Use `variant_builder` method to write values directly to the output array
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the key new API -- a builder that can write directly to the correct output array location

impl<'a> Drop for VariantArrayVariantBuilder<'a> {
/// If the builder was not finished, roll back any changes made to the
/// underlying buffers (by truncating them)
fn drop(&mut self) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a follow up, @klion26 has a PR up to implement this:

@alamb
Copy link
Contributor Author

alamb commented Jul 18, 2025

🤖 ./gh_compare_arrow.sh Benchmark Script Running
Linux aal-dev 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/append_variant_builder (fce19eb) to 99eb1bc diff
BENCH_NAME=variant_kernels
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench variant_kernels
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_append_variant_builder
Results will be posted here when complete

@alamb
Copy link
Contributor Author

alamb commented Jul 18, 2025

🤖: Benchmark completed

Details

group                                                                alamb_append_variant_builder           main
-----                                                                ----------------------------           ----
batch_json_string_to_variant json_list 8k string                     1.00     26.0±0.08ms        ? ?/sec    1.10     28.7±0.09ms        ? ?/sec
batch_json_string_to_variant random_json(2633 bytes per document)    1.00    364.4±6.10ms        ? ?/sec    1.08    393.1±6.36ms        ? ?/sec
batch_json_string_to_variant repeated_struct 8k string               1.00      7.5±0.02ms        ? ?/sec    1.14      8.6±0.03ms        ? ?/sec
variant_get_primitive                                                1.00   1040.3±4.08µs        ? ?/sec    1.32   1378.3±3.39µs        ? ?/sec

@alamb alamb changed the title Convert JSON to VariantArray without copying Convert JSON to VariantArray without copying (8 - 32% faster) Jul 18, 2025
@alamb alamb marked this pull request as ready for review July 18, 2025 16:01
@alamb
Copy link
Contributor Author

alamb commented Jul 18, 2025

This one is now ready for review. I am quite pleased it already shows some benchmarks going 30% faster

Along with the following PR from @klion26

I think our JSON conversion is about as fast as it is going to get until we move away from serde_json for parsing :bowtie:

@alamb
Copy link
Contributor Author

alamb commented Jul 22, 2025

@scovich / @viirya / @klion26 I wonder if you have time to review this PR (or if you have, are you happy to have it merged)?

I ask because avoiding this extra copy has come up a few times while contemplating how to do implement shredded writes so it would be nice to get it in so we can move on to shredding

Copy link
Member

@klion26 klion26 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the comment has not been submitted before, the change LGTM

let value_offset = self.value_offset;

// get the buffers back from the variant builder
let (mut metadata_buffer, mut value_buffer) =
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, we can transfer the ownership with this way.

Co-authored-by: Congxian Qiu <qcx978132955@gmail.com>
@alamb
Copy link
Contributor Author

alamb commented Jul 22, 2025

Sorry for the comment has not been submitted before, the change LGTM

Thanks @klion26 🙏

@@ -55,9 +55,14 @@ use std::sync::Arc;
/// };
/// builder.append_variant_buffers(&metadata, &value);
Copy link
Member

@viirya viirya Jul 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new approach is more efficient. Do we still need to have append_variant_buffers? Does it still make sense to create a VariantBuilder, build metadata and buffer and append them with append_variant_buffers?

Looks like the new variant_builder can cover it. If so, I think it doesn't make sense to keep existing approach which is less efficient. Users may use it unintentionally.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does seem preferable to have only one good way of doing things, rather than leaving a less efficient other way?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. removed in 99ea0c4

Comment on lines 309 to 328
// Sanity Check: if the buffers got smaller, something went wrong (previous data was lost)
let metadata_len = metadata_buffer
.len()
.checked_sub(metadata_offset)
.expect("metadata length decreased unexpectedly");
let value_len = value_buffer
.len()
.checked_sub(value_offset)
.expect("value length decreased unexpectedly");

if self.finished {
// if the object was finished, commit the changes by putting the
// offsets and lengths into the parent array builder.
self.array_builder
.metadata_locations
.push((metadata_offset, metadata_len));
self.array_builder
.value_locations
.push((value_offset, value_len));
self.array_builder.nulls.append_non_null();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, these (sanity check, location updates) are finish-related operations, why don't put int finish? Seems not proper to put in drop.


// get the buffers back from the variant builder
let (mut metadata_buffer, mut value_buffer) =
std::mem::take(&mut self.variant_builder).finish();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recall the discussion in the drop impl PR before, I think it is not good to put finish in drop. Should we do it in finish?

Copy link
Contributor

@scovich scovich Jul 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am on the fence about this. My natural inclination would also be to put finishing logic in finish itself... but there's also a certain symmetry to having the finished-vs-not logic branches together in the impl Drop?

As long as it's infallible, and gated by a finished check, I don't think it makes any meaningful difference? We need to track the finished flag either way, in order for drop to roll changes back correctly when finish wasn't called. Which was the problem previously -- unconditionally finishing on drop, even if the failure to finish was intentional.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One potential reason to move the finish logic out of drop -- we rely on a finish call at L307 above; imagine if that finish call didn't actually trigger any changes until the unwinding stack frame dropped the object? Most likely we'd be working with unexpected buffers for the rest of this method?

Tho I guess if there were any observable side effects like that, the corresponding &mut reference should still be live until drop, and the compiler would block any badness?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Recall the discussion in the drop impl PR before, I think it is not good to put finish in drop. Should we do it in finish?

@scovich 's rationale about the symmetry is what lead me to this approach. However I agree moving the code out of drop is a good idea and I will do it

I looked briefly into doing this -- one challenge I found is that there is no API currently to get back the underlying buffer from a VariantBuilder (aka the equivalent of into_inner() or something). I can add this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤔 I played around with it and I agree it was a good change

Among other things this prevents the metadata builder from writing bytes into the metadata builder just to have to roll them back

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 65714e5

Copy link
Contributor

@scovich scovich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few cleanups to consider, and some mild finish vs. drop controversy, but otherwise LGTM

@@ -55,9 +55,14 @@ use std::sync::Arc;
/// };
/// builder.append_variant_buffers(&metadata, &value);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does seem preferable to have only one good way of doing things, rather than leaving a less efficient other way?

impl<'a> Drop for VariantArrayVariantBuilder<'a> {
/// If the builder was not finished, roll back any changes made to the
/// underlying buffers (by truncating them)
fn drop(&mut self) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That other PR is nice improvement, but the splice call still shifts bytes.

In order to not shift bytes at all, we'd have to pre-allocate exactly the right number of header bytes before recursing into the field values. And then the splice call would just replace the zero-filled header region with the actual header bytes, after they're known (shifting bytes only if the original guess was incorrect).


// get the buffers back from the variant builder
let (mut metadata_buffer, mut value_buffer) =
std::mem::take(&mut self.variant_builder).finish();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One potential reason to move the finish logic out of drop -- we rely on a finish call at L307 above; imagine if that finish call didn't actually trigger any changes until the unwinding stack frame dropped the object? Most likely we'd be working with unexpected buffers for the rest of this method?

Tho I guess if there were any observable side effects like that, the corresponding &mut reference should still be live until drop, and the compiler would block any badness?

Comment on lines 313 to 317
.expect("metadata length decreased unexpectedly");
let value_len = value_buffer
.len()
.checked_sub(value_offset)
.expect("value length decreased unexpectedly");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These assertions should be impossible to trigger, no? And if anyway the result is a panic, is the unchecked integer underflow panic somehow worse than a failed expect?

(if we think these could actually trigger in practice, that seems like a reason to move the checks to a fallible finish instead of infallible drop?)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(meanwhile, we should probably move those inside the if finished block, since the else doesn't use them and a panic during unwind is an immediate double-fault abort?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they should be "impossible" to trigger

These assertions should be impossible to trigger, no? And if anyway the result is a panic, is the unchecked integer underflow panic somehow worse than a failed expect?

I think an integer underflow only panics in debug builds on rust (they silently underflow in release builds)

If the buffers have somehow been truncated prior to where they started, I think we should panic as soon as possible as something is seriously wrong / there is a serious bug somewhere.

@alamb
Copy link
Contributor Author

alamb commented Jul 22, 2025

🤖 ./gh_compare_arrow.sh Benchmark Script Running
Linux aal-dev 6.11.0-1016-gcp #16~24.04.1-Ubuntu SMP Wed May 28 02:40:52 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing alamb/append_variant_builder (f23e029) to 291e6e5 diff
BENCH_NAME=variant_kernels
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench variant_kernels
BENCH_FILTER=
BENCH_BRANCH_NAME=alamb_append_variant_builder
Results will be posted here when complete

@alamb
Copy link
Contributor Author

alamb commented Jul 22, 2025

🤖: Benchmark completed

Details

group                                                                alamb_append_variant_builder           main
-----                                                                ----------------------------           ----
batch_json_string_to_variant json_list 8k string                     1.00     25.8±0.09ms        ? ?/sec    1.10     28.5±0.12ms        ? ?/sec
batch_json_string_to_variant random_json(2633 bytes per document)    1.00    365.3±6.21ms        ? ?/sec    1.07    389.5±6.43ms        ? ?/sec
batch_json_string_to_variant repeated_struct 8k string               1.00      7.6±0.02ms        ? ?/sec    1.11      8.5±0.03ms        ? ?/sec
variant_get_primitive                                                1.00   1051.3±7.41µs        ? ?/sec    1.32   1382.7±4.79µs        ? ?/sec

pub fn finish(mut self) {
self.finished = true;

let metadata_offset = self.metadata_offset;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code for finishing / finalizing is moved to finish()


// get the buffers back from the variant builder
let (mut metadata_buffer, mut value_buffer) =
std::mem::take(&mut self.variant_builder).into_buffers();
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note this now calls into_buffers to get ownership of the buffers back but doesn't call finish

@alamb alamb merged commit 50f5562 into apache:main Jul 23, 2025
12 checks passed
@alamb
Copy link
Contributor Author

alamb commented Jul 23, 2025

Thanks everyone. I am pretty stoked about this one.

Now, on to shredding!

@alamb alamb deleted the alamb/append_variant_builder branch July 23, 2025 10:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
parquet Changes to the parquet crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Variant] Convert JSON to Variant with fewer copies
4 participants