Skip to content

feat: add lz4 support #331

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Mar 25, 2025
Merged

feat: add lz4 support #331

merged 9 commits into from
Mar 25, 2025

Conversation

arpankapoor
Copy link
Contributor

This PR adds LZ4 support, drawing from the gzip (encoder) and bzip2 (decoder) implementations.

@NobodyXu
Copy link
Collaborator

NobodyXu commented Mar 2, 2025

Thank you!

I have some questions for the PR

Copy link
Collaborator

@NobodyXu NobodyXu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for taking my advice!

For the impl, I think there are some bugs and some rooms for improvements

Comment on lines 128 to 138
let (dst_buffer, dst_max_size) = if direct_output {
let output_len = output.unwritten().len();
(output.unwritten_mut(), output_len)
} else {
let buffer_size = self.block_buffer_size;
let buffer = self
.maybe_buffer
.get_or_insert_with(|| PartialBuffer::new(Vec::with_capacity(buffer_size)));
buffer.reset();
(buffer.unwritten_mut(), buffer_size)
};
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let (dst_buffer, dst_max_size) = if direct_output {
let output_len = output.unwritten().len();
(output.unwritten_mut(), output_len)
} else {
let buffer_size = self.block_buffer_size;
let buffer = self
.maybe_buffer
.get_or_insert_with(|| PartialBuffer::new(Vec::with_capacity(buffer_size)));
buffer.reset();
(buffer.unwritten_mut(), buffer_size)
};
let (dst_buffer, dst_max_size) = if direct_output {
let output_len = output.unwritten().len();
output.unwritten_mut()
} else {
let buffer = self
.maybe_buffer
.get_or_insert_with(|| PartialBuffer::new(Vec::with_capacity(self.block_buffer_size)));
buffer.reset();
buffer.unwritten_mut()
};
let dst_max_size = dst_buffer.len();

Copy link
Collaborator

@NobodyXu NobodyXu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!
It LGTM except for one comment

cc @robjtede wdyt

LZ4F_compressBegin(
self.ctx.get_mut().ctx,
dst_buffer,
min_dst_size,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
min_dst_size,
dst_buffer.unwritten().len(),

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dst_buffer is a mut pointer now, so this is not possible. min_dst_size is the worst case size of the destination buffers required by the lz4 C functions, hence it is safe to use here instead of the actual destination buffer size.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could evaluate its length before the function call, then there would only be one active mutable borrow

Copy link
Collaborator

@NobodyXu NobodyXu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, LGTM!

cc @robjtede if this look good to you, I'll merge it

@NobodyXu NobodyXu added this pull request to the merge queue Mar 25, 2025
@NobodyXu
Copy link
Collaborator

Given that there's no objection, I will merge this PR in

Merged via the queue into Nullus157:main with commit ca9cd30 Mar 25, 2025
17 checks passed
Copy link

codecov bot commented Mar 25, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 0.00%. Comparing base (beb5c6b) to head (49375b4).
Report is 10 commits behind head on main.

Additional details and impacted files
@@     Coverage Diff     @@
##   main   #331   +/-   ##
===========================
===========================

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants