Have sideband tasks in app container attempt indefinite reconnection to RabbitMQ

[ponyisi]

If the RabbitMQ container dies for some reason, by default celery will retry connections 100 times then stop. This means a sufficiently long outage will leave the app in a state where jobs can be submitted, but the file list never gets dispatched to the transformers because that worker has exited.

The update here is only for the worker spun up in the app container. The DID finders in principle have the same problem but since the Celery workers are the primary tasks in those containers, when they exit k8s will restart them so we don't see the issue there. I guess something similar happens with the transformer sidecars although that is certainly going to be ugly.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.77%. Comparing base (e68e6a4) to head (4d450f8).

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1201      +/-   ##
===========================================
- Coverage    85.83%   85.77%   -0.07%     
===========================================
  Files           94       94              
  Lines         3191     3191              
  Branches       359      359              
===========================================
- Hits          2739     2737       -2     
- Misses         378      380       +2     
  Partials        74       74              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.