Skip to content

Execute intermediate data wrapper in output recon file naming tests #578

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 1, 2025

Conversation

yousefmoazzam
Copy link
Collaborator

Fixes #577

Based on information in the hdf5 docs on file locks saying that the locks are not needed when running multiple processes working together (such as when running processes using MPI), I had noted that running the 6 offending tests in test_save_intermediate_data under mpirun -n 1 never produced the error, and chalked it off as some anomaly where those tests needed to be run under MPI to pass locally.

However, since then, I noticed that those offending tests only ever create the hdf5 file for saving, and the execute() method on the wrapper object is never called.

This test is parameterised to have 4 of the 6 failing cases:

wrp = make_method_wrapper(
make_mock_repo(mocker, implementation="cpu"),
"httomo.methods",
"save_intermediate_data",
MPI.COMM_WORLD,
make_mock_preview_config(mocker),
loader=loader,
prev_method=prev_method,
)
assert isinstance(wrp, SaveIntermediateFilesWrapper)
files = get_files(tmp_path)
assert len(files) == 1
assert Path(files[0]).name == expected_filename

and this test is parameterised with 2 of the 6 failing cases:

wrp = make_method_wrapper(
make_mock_repo(mocker, implementation="cpu"),
"httomo.methods",
"save_intermediate_data",
MPI.COMM_WORLD,
make_mock_preview_config(mocker),
loader=loader,
prev_method=prev_method,
)
assert isinstance(wrp, SaveIntermediateFilesWrapper)
files = get_files(tmp_path)
assert len(files) == 1
assert Path(files[0]).name == EXPECTED_FILENAME

where the absence of wrp.execute() should be noted.

The execute() method is what actually closes the hdf5 file:

if block.is_last_in_chunk:
self._file.close()

This was confirmed to indeed be the issue, where simply running the execute() method on the intermediate data wrapper objects makes the hdf5 file locks not appear anymore when running tests locally.

Checklist

  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have made corresponding changes to the documentation

@yousefmoazzam yousefmoazzam merged commit 4e4b9d9 into main May 1, 2025
6 of 8 checks passed
@yousefmoazzam yousefmoazzam deleted the issue-577 branch May 1, 2025 11:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Hdf5 file lock present in some tests cause them to fail locally
1 participant