`decode_malformed_zlib_message` test fails when built against `zlib-ng`

[strophy]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I'm trying to package vector 0.47.0 for Alpine Linux and build against system zlib-ng libraries. My patch looks like this:

diff --git a/Cargo.toml b/Cargo.toml
index 3453d2307..ad490ffe2 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -139,7 +139,7 @@ cfg-if = { version = "1.0.0", default-features = false }
 chrono = { version = "0.4.40", default-features = false, features = ["clock", "serde"] }
 chrono-tz = { version = "0.10.3", default-features = false, features = ["serde"] }
 clap = { version = "4.5.34", default-features = false, features = ["derive", "error-context", "env", "help", "std", "string", "usage", "wrap_help"] }
-flate2 = { version = "1.0.35", default-features = false, features = ["default"] }
+flate2 = { version = "1.0.35", default-features = false, features = ["zlib-ng"] }
 futures = { version = "0.3.31", default-features = false, features = ["compat", "io-compat", "std"], package = "futures" }
 glob = { version = "0.3.2", default-features = false }
 hickory-proto = { version = "0.25.1", default-features = false, features = ["dnssec-ring"] }
@@ -392,7 +392,7 @@ thread_local = { version = "1.1.8", default-features = false, optional = true }
 typetag = { version = "0.2.20", default-features = false }
 url = { version = "2.5.4", default-features = false, features = ["serde"] }
 warp = { version = "0.3.7", default-features = false }
-zstd = { version = "0.13.0", default-features = false }
+zstd = { version = "0.13.0", default-features = false, features = ["pkg-config"] }
 arr_macro = { version = "0.2.1" }
 
 # depending on fork for bumped nix dependency
@@ -468,7 +468,7 @@ default-cmake = ["api", "api-client", "enrichment-tables", "rdkafka?/cmake_build
 # Default features for *-pc-windows-msvc
 # TODO: Enable SASL https://github.com/vectordotdev/vector/pull/3081#issuecomment-659298042
 default-msvc = ["api", "api-client", "enrichment-tables", "rdkafka?/cmake_build", "sinks", "sources", "transforms", "secrets"]
-default-musl = ["api", "api-client", "enrichment-tables", "rdkafka?/cmake_build", "sinks", "sources", "sources-dnstap", "transforms", "unix", "rdkafka?/gssapi-vendored", "secrets"]
+default-musl = ["api", "api-client", "enrichment-tables", "rdkafka?/dynamic-linking", "sinks", "sources", "sources-dnstap", "transforms", "unix", "rdkafka?/gssapi", "secrets"]
 default-no-api-client = ["api", "enrichment-tables", "sinks", "sources", "sources-dnstap", "transforms", "unix", "rdkafka?/gssapi-vendored", "secrets"]
 default-no-vrl-cli = ["api", "sinks", "sources", "sources-dnstap", "transforms", "unix", "rdkafka?/gssapi-vendored", "secrets"]
 tokio-console = ["dep:console-subscriber", "tokio/tracing"]

We build like this:

prepare() {
	default_prepare

	# Rust target triple.
	local target=$(rustc -vV | sed -n 's/host: //p')

	# Build against system-provided zlib-ng
	mkdir -p .cargo
	cat >> .cargo/config.toml <<-EOF
		[target.$target]
		z-ng = { rustc-link-lib = ["z-ng"], rustc-cfg = ["zng"] }
	EOF

	# "cargo auditable build" still tries to pull dependencies for other platforms
	cargo fetch --locked
}

build() {
	cargo auditable build \
		--release \
		--frozen \
		--no-default-features \
		--features default-musl,component-validation-runner

	# cargo-nextest drops the release build, so back it up here
	cp -r target/release releasebuild
}

This results in the following error in the test phase:

  TRY 1 FAIL [   0.018s] codecs decoding::framing::chunked_gelf::tests::decode_malformed_zlib_message
──── TRY 1 STDOUT:       codecs decoding::framing::chunked_gelf::tests::decode_malformed_zlib_message
running 1 test
test decoding::framing::chunked_gelf::tests::decode_malformed_zlib_message ... FAILED
failures:
failures:
    decoding::framing::chunked_gelf::tests::decode_malformed_zlib_message
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 144 filtered out; finished in 0.00s
──── TRY 1 STDERR:       codecs decoding::framing::chunked_gelf::tests::decode_malformed_zlib_message
thread 'decoding::framing::chunked_gelf::tests::decode_malformed_zlib_message' panicked at lib/codecs/src/decoding/framing/chunked_gelf.rs:1073:14:
decoding should fail: Some(b"\x113")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Why does this test fail and how can I fix it? Should test be updated to consider different possible errors from different zlib implementations?

Configuration

Version

0.47.0

Debug Output

Example Data

No response

Additional Context

No response

References

No response

[jorgehermo9]

There was a discussion related to this: #22533 (comment). The test you are pointing was changed in #22533 and the change included in Vector 0.48.0, so I suggest to update your Vector version to 0.48.0 first.

There was a bug in miniz_oxide that for some input (the one that we were using on the test you pointed) miniz_oxide returned an error but other backends succeed (for example, zlib-ng, that's why you are seeing that error). You can find the fix for that here, were miniz_oxide would succeed for the input we were using and not fail. Therefore, the decode_malformed_zlib_message had to be changed to another payload that failed, work which was done in #22533.

I think this issue should be closed

[strophy]

Thanks @jorgehermo9 #22533 looks like it should address the issue I'm seeing as well. 0.48.0 isn't released yet, so I'll wait for that release (or make a patch) and report back with my results here.

[jorgehermo9]

My mistake, sorry. The flate2 update I'm referring to was released on 0.47.0. I saw on your diff that the flate2 version that you are using is 1.0.35

-flate2 = { version = "1.0.35", default-features = false, features = ["default"] }

And therefore I thought you were not using the latest Vector release. But, for Vector 0.47.0 (the version your are using), you should be using flate2 version 1.1.1. Your Cargo.toml is not updated with the Vector's v0.47.0 branch's tip

Please, see this https://github.com/vectordotdev/vector/blob/v0.47/Cargo.toml#L143 and try to update your diff to something like

+flate2 = { version = "1.1.1", default-features = false, features = ["zlib-ng"] }

[strophy]

Thanks for the clarification @jorgehermo9 I agree that I did something wrong with my patch against 0.47.0, likely because I started this work against 0.46.1. The build succeeds now, so the fix you explained earlier also works with zlib-ng. Thanks for the help!