Skip to content

Add timeout if expected files don't materialize in object store #533

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ponyisi opened this issue Dec 6, 2024 · 1 comment
Open

Add timeout if expected files don't materialize in object store #533

ponyisi opened this issue Dec 6, 2024 · 1 comment

Comments

@ponyisi
Copy link
Collaborator

ponyisi commented Dec 6, 2024

In the 3.0.1 client, we will wait indefinitely for files to show up in the S3 storage when the transformation is reported complete. We should add a timeout so that if the files don't appear within some reasonable time the client gives up and reports an error.

@ponyisi
Copy link
Collaborator Author

ponyisi commented Jan 24, 2025

This is much less likely to be a problem now given that the server should not report transform complete until all outputs have been transferred to S3. However we might still consider a way of exiting the loop if we do run into some weird situation where the number of files in S3 stays less than the number reported by the transform for an extended period once the transform is complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant