Skip to content

feat(integration-tests): Add loseless round-trip compression tests for xz, zstd, lz4 log archives. #1311

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

feat(integration-tests): Add loseless round-trip compression tests for xz, zstd, lz4 log archives. #1311

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
78c6458
Add xz compression test
Bill-hbrhbr Sep 17, 2025
4741ef9
complete reset of the compression formats
Bill-hbrhbr Sep 17, 2025
70a7529
Pass in various compression tool path prefixes via taskflow variables
Bill-hbrhbr Sep 17, 2025
45be4a9
Improve python comments and work tasks
Bill-hbrhbr Sep 18, 2025
3065f15
Merge branch 'main' into tests-archive-ingestion
Bill-hbrhbr Sep 18, 2025
4369f5c
Merge branch 'main' into tests-archive-ingestion
Bill-hbrhbr Sep 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions integration-tests/.pytest.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,11 @@ addopts =
env =
D:CLP_BUILD_DIR=../build
D:CLP_CORE_BINS_DIR=../build/core
D:CLP_DEPS_CORE_DIR=../build/deps/core
D:CLP_LIBLZMA_ROOT=../build/deps/core/LibLZMA-install
D:CLP_LZ4_ROOT=../build/deps/core/lz4-install
D:CLP_PACKAGE_DIR=../build/clp-package
D:CLP_ZSTD_ROOT=../build/deps/core/zstd-install
log_cli = True
log_cli_date_format = %Y-%m-%d %H:%M:%S,%f
log_cli_format = %(name)s %(asctime)s [%(levelname)s] %(message)s
Expand Down
8 changes: 8 additions & 0 deletions integration-tests/tests/fixtures/integration_test_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@

from tests.utils.config import (
CoreConfig,
DepsConfig,
IntegrationTestConfig,
PackageConfig,
)
Expand All @@ -18,12 +19,19 @@ def integration_test_config() -> IntegrationTestConfig:
core_config = CoreConfig(
clp_core_bins_dir=Path(get_env_var("CLP_CORE_BINS_DIR")).expanduser().resolve()
)
deps_config = DepsConfig(
clp_deps_core_dir=Path(get_env_var("CLP_DEPS_CORE_DIR")).expanduser().resolve(),
clp_liblzma_root=Path(get_env_var("CLP_LIBLZMA_ROOT")).expanduser().resolve(),
clp_lz4_root=Path(get_env_var("CLP_LZ4_ROOT")).expanduser().resolve(),
clp_zstd_root=Path(get_env_var("CLP_ZSTD_ROOT")).expanduser().resolve(),
)
package_config = PackageConfig(
clp_package_dir=Path(get_env_var("CLP_PACKAGE_DIR")).expanduser().resolve()
)
test_root_dir = Path(get_env_var("CLP_BUILD_DIR")).expanduser().resolve() / "integration-tests"
return IntegrationTestConfig(
core_config=core_config,
deps_config=deps_config,
package_config=package_config,
test_root_dir=test_root_dir,
)
36 changes: 33 additions & 3 deletions integration-tests/tests/fixtures/integration_test_logs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ def _download_and_extract_dataset(
curl_bin,
"--fail",
"--location",
"--output", str(integration_test_logs.tarball_path),
"--output", str(integration_test_logs.tar_gz_path),
"--show-error",
tarball_url,
]
Expand All @@ -78,7 +78,7 @@ def _download_and_extract_dataset(

unlink(integration_test_logs.extraction_dir)
shutil.unpack_archive(
integration_test_logs.tarball_path, integration_test_logs.extraction_dir
integration_test_logs.tar_gz_path, integration_test_logs.extraction_dir
)
except Exception as e:
err_msg = f"Failed to download and extract dataset `{name}`."
Expand All @@ -89,7 +89,37 @@ def _download_and_extract_dataset(
if chmod_bin is None:
err_msg = "chmod executable not found"
raise RuntimeError(err_msg)
subprocess.run([chmod_bin, "-R", "gu+w", integration_test_logs.extraction_dir], check=True)
subprocess.run([chmod_bin, "-R", "gu+w", str(integration_test_logs.extraction_dir)], check=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick

Unnecessary string conversion for Path argument.

The subprocess.run function accepts Path objects directly, so converting to string is redundant here.

Apply this diff to simplify:

-    subprocess.run([chmod_bin, "-R", "gu+w", str(integration_test_logs.extraction_dir)], check=True)
+    subprocess.run([chmod_bin, "-R", "gu+w", integration_test_logs.extraction_dir], check=True)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
subprocess.run([chmod_bin, "-R", "gu+w", str(integration_test_logs.extraction_dir)], check=True)
subprocess.run([chmod_bin, "-R", "gu+w", integration_test_logs.extraction_dir], check=True)
🤖 Prompt for AI Agents 
In integration-tests/tests/fixtures/integration_test_logs.py around line 92, the
call to subprocess.run converts a Path to a string unnecessarily; change the
argument list to pass the Path object directly (remove str(...) around
integration_test_logs.extraction_dir) so subprocess.run receives the Path
instance, keeping check=True and preserving the rest of the call.


# Create base tar stream object to be compressed into different formats
gzip_bin = shutil.which("gzip")
if gzip_bin is None:
err_msg = "gzip executable not found"
raise RuntimeError(err_msg)
gzip_cmds = [gzip_bin, "--decompress", "--stdout", str(integration_test_logs.tar_gz_path)]
with integration_test_logs.base_tar_path.open(mode="wb") as fout:
subprocess.run(gzip_cmds, check=True, stdout=fout, stdin=subprocess.DEVNULL)

Comment on lines +94 to +102
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick

Consider error handling for base tar creation.

The base tar file creation could fail if disk space is insufficient or write permissions are missing. While the current implementation will raise an exception, a more specific error message would aid debugging.

Consider wrapping in a try-except to provide clearer error context:

     # Create base tar stream object to be compressed into different formats
     gzip_bin = shutil.which("gzip")
     if gzip_bin is None:
         err_msg = "gzip executable not found"
         raise RuntimeError(err_msg)
     gzip_cmds = [gzip_bin, "--decompress", "--stdout", str(integration_test_logs.tar_gz_path)]
-    with integration_test_logs.base_tar_path.open(mode="wb") as fout:
-        subprocess.run(gzip_cmds, check=True, stdout=fout, stdin=subprocess.DEVNULL)
+    try:
+        with integration_test_logs.base_tar_path.open(mode="wb") as fout:
+            subprocess.run(gzip_cmds, check=True, stdout=fout, stdin=subprocess.DEVNULL)
+    except (OSError, subprocess.CalledProcessError) as e:
+        err_msg = f"Failed to create base tar file at {integration_test_logs.base_tar_path}"
+        raise RuntimeError(err_msg) from e
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Create base tar stream object to be compressed into different formats
gzip_bin = shutil.which("gzip")
if gzip_bin is None:
err_msg = "gzip executable not found"
raise RuntimeError(err_msg)
gzip_cmds = [gzip_bin, "--decompress", "--stdout", str(integration_test_logs.tar_gz_path)]
with integration_test_logs.base_tar_path.open(mode="wb") as fout:
subprocess.run(gzip_cmds, check=True, stdout=fout, stdin=subprocess.DEVNULL)
# Create base tar stream object to be compressed into different formats
gzip_bin = shutil.which("gzip")
if gzip_bin is None:
err_msg = "gzip executable not found"
raise RuntimeError(err_msg)
gzip_cmds = [gzip_bin, "--decompress", "--stdout", str(integration_test_logs.tar_gz_path)]
try:
with integration_test_logs.base_tar_path.open(mode="wb") as fout:
subprocess.run(gzip_cmds, check=True, stdout=fout, stdin=subprocess.DEVNULL)
except (OSError, subprocess.CalledProcessError) as e:
err_msg = f"Failed to create base tar file at {integration_test_logs.base_tar_path}"
raise RuntimeError(err_msg) from e
🤖 Prompt for AI Agents 
In integration-tests/tests/fixtures/integration_test_logs.py around lines 94 to
102, the block that locates gzip and writes the decompressed tar to
base_tar_path can fail for reasons like missing disk space or permission errors
but currently surfaces only generic exceptions; wrap the file open and
subprocess.run call in a try-except that catches OSError, IOError and
subprocess.CalledProcessError and re-raises a RuntimeError with a clear
contextual message (including the original exception text) so callers get a
specific, actionable error; keep the existing gzip executable existence check
as-is and ensure the exception preserves the original traceback (e.g., using
"from e") when re-raising.

# Create lz4 tar
lz4_bin = str(integration_test_config.deps_config.lz4_binary_path)
lz4_cmds = [
lz4_bin,
str(integration_test_logs.base_tar_path),
str(integration_test_logs.tar_lz4_path),
]
subprocess.run(lz4_cmds, check=True)
Comment on lines +104 to +110
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

🧩 Analysis chain

Verify lz4 binary path existence before use.

Unlike the system binaries (gzip, curl, chmod), the dependency binaries from deps_config aren't validated for existence before use. This could lead to less clear error messages if the paths are incorrect.

🏁 Script executed:

#!/bin/bash
# Verify that the binary path properties in DepsConfig return valid paths

# Check if the binary path properties exist and point to actual files
ast-grep --pattern 'def $_binary_path(self) -> Path:
  $$$'

# Look for any validation of these binary paths before use
rg -n -C3 '(lz4|xz|zstd)_binary_path' --type=py

Length of output: 3540

Verify lz4 binary path exists before use.

integration-tests/tests/utils/config.py returns clp_lz4_root / "bin" / "lz4" but there’s no check that the file exists; integration-tests/tests/fixtures/integration_test_logs.py invokes it directly (lines 104–110). Add an explicit Path.exists()/is_file() check or validate clp_lz4_root in DepsConfig and raise a clear error before calling subprocess.run.

🤖 Prompt for AI Agents 
In integration-tests/tests/fixtures/integration_test_logs.py around lines 104 to
110, the code calls the lz4 binary directly without verifying it exists; add an
explicit check using the Path for
integration_test_config.deps_config.lz4_binary_path (e.g., Path.exists() and
Path.is_file()) before building lz4_cmds and calling subprocess.run, and if the
check fails raise a clear, descriptive exception (or ValueError) that includes
the expected path so the test setup fails fast with an actionable message;
alternatively validate clp_lz4_root when constructing DepsConfig and raise there
so callers never receive an invalid path.


# Create xz tar
xz_bin = str(integration_test_config.deps_config.xz_binary_path)
xz_cmds = [xz_bin, "--compress", "--stdout", str(integration_test_logs.base_tar_path)]
with integration_test_logs.tar_xz_path.open(mode="wb") as fout:
subprocess.run(xz_cmds, check=True, stdout=fout, stdin=subprocess.DEVNULL)

# Create zstd tar
zstd_bin = str(integration_test_config.deps_config.zstd_binary_path)
zstd_cmds = [zstd_bin, "--stdout", str(integration_test_logs.base_tar_path)]
with integration_test_logs.tar_zstd_path.open(mode="wb") as fout:
subprocess.run(zstd_cmds, check=True, stdout=fout, stdin=subprocess.DEVNULL)

Comment on lines +94 to 123
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick

Archive fan‑out pipeline is sound; consider quiet flags and a clearer log.

The gzip→lz4/xz/zstd steps are correct and stream‑friendly. Minor polish:

  • Add quiet flags (-q where available) to reduce test noise.
  • Update the final log message to reflect that compressed archives were generated, not just “extracted”.

Apply this diff for the log:

-    logger.info("Downloaded and extracted uncompressed logs for dataset `%s`.", name)
+    logger.info(
+        "Downloaded logs for `%s`, extracted, and generated .tar.gz/.tar.lz4/.tar.xz/.tar.zstd.",
+        name,
+    )

Optional: if you want reproducible downloads, we can wire in SHA256 checks for the datasets in a follow‑up.

Committable suggestion skipped: line range outside the PR's diff.

🤖 Prompt for AI Agents 
In integration-tests/tests/fixtures/integration_test_logs.py around lines 94 to
123, the subprocess commands for creating compressed archives should run quieter
and the final log message should correctly state that compressed archives were
generated (not “extracted”); update the gzip/lz4/xz/zstd command argument lists
to include the appropriate quiet flags (e.g., gzip: --quiet or -q if supported,
lz4: -q, xz: -q, zstd: -q) and adjust the final log call text to say "compressed
archives generated" or similar to accurately reflect the operation.

logger.info("Downloaded and extracted uncompressed logs for dataset `%s`.", name)
request.config.cache.set(name, True)
Expand Down
66 changes: 50 additions & 16 deletions integration-tests/tests/test_identity_transformation.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
compression and decompression.
"""

from pathlib import Path

import pytest

from tests.utils.asserting_utils import run_and_assert
Expand Down Expand Up @@ -42,46 +44,78 @@ def test_clp_identity_transform(
) -> None:
"""
Validate that compression and decompression by the core binary `clp` run successfully and are
lossless.
lossless across various input archive formats.

:param request:
:param integration_test_config:
:param test_logs_fixture:
"""
integration_test_logs: IntegrationTestLogs = request.getfixturevalue(test_logs_fixture)
logs_source_dir: Path = integration_test_logs.extraction_dir

archives_to_test = [
integration_test_logs.extraction_dir,
integration_test_logs.tar_gz_path,
integration_test_logs.tar_lz4_path,
integration_test_logs.tar_xz_path,
integration_test_logs.tar_zstd_path,
]
for archive_path in archives_to_test:
_test_clp_identity_transform_single_archive(
archive_path, integration_test_config, logs_source_dir
)


def _test_clp_identity_transform_single_archive(
compression_input: Path,
integration_test_config: IntegrationTestConfig,
logs_source_dir: Path,
) -> None:
"""
Validate that compression and decompression by the core binary `clp` run successfully and are
lossless for a single archive input format.

:param compression_input: Path to the archive for compression.
:param integration_test_config: General config for the integration tests.
:param logs_source_dir: Path to the uncompressed logs for comparison.
"""
test_paths = CompressionTestConfig(
test_name=f"clp-{integration_test_logs.name}",
logs_source_dir=integration_test_logs.extraction_dir,
test_name=f"clp-{compression_input.name}",
compression_input=compression_input,
integration_test_config=integration_test_config,
)
test_paths.clear_test_outputs()

bin_path = str(integration_test_config.core_config.clp_binary_path)
src_path = str(test_paths.logs_source_dir)
input_path = str(test_paths.compression_input)
compression_path = str(test_paths.compression_dir)
decompression_path = str(test_paths.decompression_dir)
path_prefix_to_remove = (
input_path
if test_paths.compression_input.is_dir()
else str(test_paths.compression_input.parent)
)

# fmt: off
compression_cmd = [
bin_path,
"c",
"--progress",
"--remove-path-prefix", src_path,
"--remove-path-prefix", path_prefix_to_remove,
compression_path,
src_path,
input_path,
]
# fmt: on
run_and_assert(compression_cmd)

decompression_cmd = [bin_path, "x", compression_path, decompression_path]
run_and_assert(decompression_cmd)

input_path = test_paths.logs_source_dir
output_path = test_paths.decompression_dir
decompressed_logs_path = test_paths.decompression_dir
assert is_dir_tree_content_equal(
input_path,
output_path,
), f"Mismatch between clp input {input_path} and output {output_path}."

logs_source_dir,
decompressed_logs_path,
), f"Mismatch between source {logs_source_dir} and `clp` final output {decompressed_logs_path}."
test_paths.clear_test_outputs()


Expand All @@ -105,7 +139,7 @@ def test_clp_s_identity_transform(

test_paths = CompressionTestConfig(
test_name=f"clp-s-{test_logs_name}",
logs_source_dir=integration_test_logs.extraction_dir,
compression_input=integration_test_logs.extraction_dir,
integration_test_config=integration_test_config,
)
_clp_s_compress_and_decompress(integration_test_config, test_paths)
Expand All @@ -118,13 +152,13 @@ def test_clp_s_identity_transform(
# See also: https://docs.yscope.com/clp/main/user-guide/core-clp-s.html#current-limitations
consolidated_json_test_paths = CompressionTestConfig(
test_name=f"clp-s-{test_logs_name}-consolidated-json",
logs_source_dir=test_paths.decompression_dir,
compression_input=test_paths.decompression_dir,
integration_test_config=integration_test_config,
)
_clp_s_compress_and_decompress(integration_test_config, consolidated_json_test_paths)

_consolidated_json_file_name = "original"
input_path = consolidated_json_test_paths.logs_source_dir / _consolidated_json_file_name
input_path = consolidated_json_test_paths.compression_input / _consolidated_json_file_name
output_path = consolidated_json_test_paths.decompression_dir / _consolidated_json_file_name
assert is_json_file_structurally_equal(input_path, output_path), (
f"Mismatch between clp-s input {input_path} and output {output_path}."
Expand All @@ -139,7 +173,7 @@ def _clp_s_compress_and_decompress(
) -> None:
test_paths.clear_test_outputs()
bin_path = str(integration_test_config.core_config.clp_s_binary_path)
src_path = str(test_paths.logs_source_dir)
src_path = str(test_paths.compression_input)
compression_path = str(test_paths.compression_dir)
decompression_path = str(test_paths.decompression_dir)
run_and_assert([bin_path, "c", compression_path, src_path])
Expand Down
85 changes: 73 additions & 12 deletions integration-tests/tests/utils/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

from tests.utils.utils import (
unlink,
validate_dir_exists,
validate_dir_exists_and_is_absolute,
)


Expand All @@ -24,7 +24,7 @@ def __post_init__(self) -> None:
executables.
"""
clp_core_bins_dir = self.clp_core_bins_dir
validate_dir_exists(clp_core_bins_dir)
validate_dir_exists_and_is_absolute(clp_core_bins_dir)

# Check for required CLP core binaries
required_binaries = ["clg", "clo", "clp", "clp-s", "indexer", "reducer-server"]
Expand All @@ -47,6 +47,42 @@ def clp_s_binary_path(self) -> Path:
return self.clp_core_bins_dir / "clp-s"


@dataclass(frozen=True)
class DepsConfig:
"""The configuration for dependencies used by CLP package and binaries."""

#: Install directory for all core CLP dependencies.
clp_deps_core_dir: Path
#: Install prefix of LibLZMA used by CLP.
clp_liblzma_root: Path
#: Install prefix of lz4 used by CLP.
clp_lz4_root: Path
#: Install prefix of zstd used by CLP.
clp_zstd_root: Path

def __post_init__(self) -> None:
"""Validates that dependency directories exist."""
validate_dir_exists_and_is_absolute(self.clp_deps_core_dir)
validate_dir_exists_and_is_absolute(self.clp_liblzma_root)
validate_dir_exists_and_is_absolute(self.clp_lz4_root)
validate_dir_exists_and_is_absolute(self.clp_zstd_root)

@property
def lz4_binary_path(self) -> Path:
""":return: The absolute path to the lz4 compression tool."""
return self.clp_lz4_root / "bin" / "lz4"

@property
def xz_binary_path(self) -> Path:
""":return: The absolute path to the LibLZMA xz compression tool."""
return self.clp_liblzma_root / "bin" / "xz"

@property
def zstd_binary_path(self) -> Path:
""":return: The absolute path to the zstd compression tool."""
return self.clp_zstd_root / "bin" / "zstd"

Comment on lines +50 to +84
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick

Consider validating tool binaries exist in DepsConfig.

Early failure with a clear message helps when a tool wasn’t built/installed.

Apply this diff:

     def __post_init__(self) -> None:
         """Validates that dependency directories exist."""
         validate_dir_exists_and_is_absolute(self.clp_deps_core_dir)
         validate_dir_exists_and_is_absolute(self.clp_liblzma_root)
         validate_dir_exists_and_is_absolute(self.clp_lz4_root)
         validate_dir_exists_and_is_absolute(self.clp_zstd_root)
+        # Ensure compression tools are present
+        for tool in (self.lz4_binary_path, self.xz_binary_path, self.zstd_binary_path):
+            if not tool.is_file():
+                raise ValueError(f"Compression tool not found: {tool}")
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
@dataclass(frozen=True)
class DepsConfig:
"""The configuration for dependencies used by CLP package and binaries."""
#: Install directory for all core CLP dependencies.
clp_deps_core_dir: Path
#: Install prefix of LibLZMA used by CLP.
clp_liblzma_root: Path
#: Install prefix of lz4 used by CLP.
clp_lz4_root: Path
#: Install prefix of zstd used by CLP.
clp_zstd_root: Path
def __post_init__(self) -> None:
"""Validates that dependency directories exist."""
validate_dir_exists_and_is_absolute(self.clp_deps_core_dir)
validate_dir_exists_and_is_absolute(self.clp_liblzma_root)
validate_dir_exists_and_is_absolute(self.clp_lz4_root)
validate_dir_exists_and_is_absolute(self.clp_zstd_root)
@property
def lz4_binary_path(self) -> Path:
""":return: The absolute path to the lz4 compression tool."""
return self.clp_lz4_root / "bin" / "lz4"
@property
def xz_binary_path(self) -> Path:
""":return: The absolute path to the LibLZMA xz compression tool."""
return self.clp_liblzma_root / "bin" / "xz"
@property
def zstd_binary_path(self) -> Path:
""":return: The absolute path to the zstd compression tool."""
return self.clp_zstd_root / "bin" / "zstd"
@dataclass(frozen=True)
class DepsConfig:
"""The configuration for dependencies used by CLP package and binaries."""
#: Install directory for all core CLP dependencies.
clp_deps_core_dir: Path
#: Install prefix of LibLZMA used by CLP.
clp_liblzma_root: Path
#: Install prefix of lz4 used by CLP.
clp_lz4_root: Path
#: Install prefix of zstd used by CLP.
clp_zstd_root: Path
def __post_init__(self) -> None:
"""Validates that dependency directories exist."""
validate_dir_exists_and_is_absolute(self.clp_deps_core_dir)
validate_dir_exists_and_is_absolute(self.clp_liblzma_root)
validate_dir_exists_and_is_absolute(self.clp_lz4_root)
validate_dir_exists_and_is_absolute(self.clp_zstd_root)
# Ensure compression tools are present
for tool in (self.lz4_binary_path, self.xz_binary_path, self.zstd_binary_path):
if not tool.is_file():
raise ValueError(f"Compression tool not found: {tool}")
@property
def lz4_binary_path(self) -> Path:
""":return: The absolute path to the lz4 compression tool."""
return self.clp_lz4_root / "bin" / "lz4"
@property
def xz_binary_path(self) -> Path:
""":return: The absolute path to the LibLZMA xz compression tool."""
return self.clp_liblzma_root / "bin" / "xz"
@property
def zstd_binary_path(self) -> Path:
""":return: The absolute path to the zstd compression tool."""
return self.clp_zstd_root / "bin" / "zstd"
🤖 Prompt for AI Agents 
In integration-tests/tests/utils/config.py around lines 50 to 84, DepsConfig
currently only validates the dependency root directories but does not verify
that the actual tool binaries exist or are executable; update __post_init__ to
also validate lz4_binary_path, xz_binary_path, and zstd_binary_path (either by
calling a new helper validate_file_exists_and_is_executable(path, name) or an
existing equivalent) so that each binary path is checked for existence and
executability at object construction, and compute the property paths before
calling the validation so failures produce a clear, early error message naming
the missing/non-executable binary.


@dataclass(frozen=True)
class PackageConfig:
"""The configuration for the clp package subject to testing."""
Expand All @@ -57,7 +93,7 @@ class PackageConfig:
def __post_init__(self) -> None:
"""Validates that the CLP package directory exists and contains all required directories."""
clp_package_dir = self.clp_package_dir
validate_dir_exists(clp_package_dir)
validate_dir_exists_and_is_absolute(clp_package_dir)

# Check for required package script directories
required_dirs = ["bin", "etc", "lib", "sbin"]
Expand All @@ -77,6 +113,8 @@ class IntegrationTestConfig:
#:
core_config: CoreConfig
#:
deps_config: DepsConfig
#:
package_config: PackageConfig
#: Root directory for integration tests output.
test_root_dir: Path
Expand Down Expand Up @@ -105,8 +143,6 @@ class IntegrationTestLogs:
tarball_url: str
integration_test_config: InitVar[IntegrationTestConfig]
#:
tarball_path: Path = field(init=False, repr=True)
#:
extraction_dir: Path = field(init=False, repr=True)

def __post_init__(self, integration_test_config: IntegrationTestConfig) -> None:
Expand All @@ -115,26 +151,51 @@ def __post_init__(self, integration_test_config: IntegrationTestConfig) -> None:
if 0 == len(name):
err_msg = "`name` cannot be empty."
raise ValueError(err_msg)

logs_download_dir = integration_test_config.logs_download_dir
validate_dir_exists(logs_download_dir)
validate_dir_exists_and_is_absolute(logs_download_dir)

object.__setattr__(self, "name", name)
object.__setattr__(self, "tarball_path", logs_download_dir / f"{name}.tar.gz")
object.__setattr__(self, "extraction_dir", logs_download_dir / name)

@property
def base_tar_path(self) -> Path:
""":return: The absolute path to the tar archive."""
return self.extraction_dir.with_suffix(".tar")

@property
def tar_gz_path(self) -> Path:
""":return: The absolute path to the tar gzip archive."""
return self.extraction_dir.with_suffix(".tar.gz")

@property
def tar_lz4_path(self) -> Path:
""":return: The absolute path to the tar lz4 archive."""
return self.extraction_dir.with_suffix(".tar.lz4")

@property
def tar_xz_path(self) -> Path:
""":return: The absolute path to the tar xz archive."""
return self.extraction_dir.with_suffix(".tar.xz")

@property
def tar_zstd_path(self) -> Path:
""":return: The absolute path to the tar zstd archive."""
return self.extraction_dir.with_suffix(".tar.zstd")


@dataclass(frozen=True)
class CompressionTestConfig:
"""Compression test configuration providing per-test metadata for artifacts and directories."""

#:
test_name: str
#: Directory containing the original (uncompressed) log files used by this test.
logs_source_dir: Path
#: Path to the CLP compressionm input archive or directory.
compression_input: Path
integration_test_config: InitVar[IntegrationTestConfig]
#: Path to store compressed archives generated by the test.
#: Directory to store generated compressed CLP archives.
compression_dir: Path = field(init=False, repr=True)
#: Path to store decompressed logs generated by the test.
#: Directory to store logs decompressed from CLP archives.
decompression_dir: Path = field(init=False, repr=True)

Comment on lines +193 to 200
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick

Fix minor typo in docstring.

“compressionm” → “compression”.

Apply this diff:

-    #: Path to the CLP compressionm input archive or directory.
+    #: Path to the CLP compression input archive or directory.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
#: Path to the CLP compressionm input archive or directory.
compression_input: Path
integration_test_config: InitVar[IntegrationTestConfig]
#: Path to store compressed archives generated by the test.
#: Directory to store generated compressed CLP archives.
compression_dir: Path = field(init=False, repr=True)
#: Path to store decompressed logs generated by the test.
#: Directory to store logs decompressed from CLP archives.
decompression_dir: Path = field(init=False, repr=True)
#: Path to the CLP compression input archive or directory.
compression_input: Path
integration_test_config: InitVar[IntegrationTestConfig]
#: Directory to store generated compressed CLP archives.
compression_dir: Path = field(init=False, repr=True)
#: Directory to store logs decompressed from CLP archives.
decompression_dir: Path = field(init=False, repr=True)
🤖 Prompt for AI Agents 
In integration-tests/tests/utils/config.py around lines 193 to 200, fix the
minor typo in the docstring for the compression_input field: change
"compressionm" to "compression" so the comment reads "Path to the CLP
compression input archive or directory." and preserve surrounding formatting and
punctuation.

def __post_init__(self, integration_test_config: IntegrationTestConfig) -> None:
Expand All @@ -144,7 +205,7 @@ def __post_init__(self, integration_test_config: IntegrationTestConfig) -> None:
err_msg = "`test_name` cannot be empty."
raise ValueError(err_msg)
test_root_dir = integration_test_config.test_root_dir
validate_dir_exists(test_root_dir)
validate_dir_exists_and_is_absolute(test_root_dir)

object.__setattr__(self, "test_name", test_name)
object.__setattr__(self, "compression_dir", test_root_dir / f"{test_name}-archives")
Expand Down
Loading
Loading