Skip to content

Do not retry processing when there is no picture #199

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions pictures/tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,15 @@ def _process_picture(
old = old or []
storage = utils.reconstruct(*storage)
if new:
with storage.open(file_name) as fs:
with Image.open(fs) as img:
for picture in new:
picture = utils.reconstruct(*picture)
picture.save(img)
try:
with storage.open(file_name) as fs:
with Image.open(fs) as img:
for picture in new:
picture = utils.reconstruct(*picture)
picture.save(img)
except FileNotFoundError:
# The file no longer exists (for example, because it was deleted or replaced).
return
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just silently ignoring missing files, will make it challenging to debug potential errors. I believe all message queues we use have a native behavior to fail without retrying on explicit exceptions. I'd recommend exploring that route.

Copy link
Collaborator Author

@amureki amureki Feb 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@codingjoe hm, this is a good suggestion!
I am struggling with figuring out the good way of doing so in django-rq (the interface is quite different from what I used to see in the other queues).
I added an example commit of how I see it with celery and dramatiq:
7c73c82

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have much experience with RQ either, let's ask @krtko1
Do you have any idea how to solve this in RQ?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amureki my search only brought up this https://github.com/rq/rq/pull/1480/files


for picture in old:
picture = utils.reconstruct(*picture)
Expand Down
15 changes: 15 additions & 0 deletions tests/test_tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,5 +22,20 @@ def test_process_picture__file_cannot_be_reopened(image_upload_file):
)


@pytest.mark.django_db
def test_process_picture__file_missing(image_upload_file):
obj = SimpleModel.objects.create(picture=image_upload_file)
setattr(
obj.picture.file,
"open",
Mock(side_effect=FileNotFoundError("The file does not exist anymore.")),
)
tasks._process_picture(
obj.picture.storage.deconstruct(),
obj.picture.name,
new=[i.deconstruct() for i in obj.picture.get_picture_files_list()],
)


def test_noop():
tasks.noop() # does nothing