-
Notifications
You must be signed in to change notification settings - Fork 13
Open
Description
I was testing Text Alignment on Production (and also for fix1305
) and noticed that the job was not using NVIDIA GPU, even with proper setup. nvidia-smi
shows nothing.
Here's the log from make gpu-celery_log
on Production server:
[2025-06-27 13:39:44,017: INFO/MainProcess] Received task: Text Alignment[4366a2d8-1385-40f1-abd2-410ed7d1683a]
[2025-06-27 13:39:44,114: INFO/ForkPoolWorker-48] started running the task!
[2025-06-27 13:39:44,115: INFO/ForkPoolWorker-48] Text Alignment[4366a2d8-1385-40f1-abd2-410ed7d1683a]: {'OCR Model': 0}
[2025-06-27 13:39:44,988: INFO/ForkPoolWorker-48] Text Alignment[4366a2d8-1385-40f1-abd2-410ed7d1683a]: processing image...
[2025-06-27 13:39:44,988: WARNING/ForkPoolWorker-48] identifying text lines...
WARNING 2025-06-27 13:41:45,292 celery.redirected: /usr/local/lib/python3.7/dist-packages/tensorflow_addons/utils/ensure_tf_install.py:67: UserWarning: Tensorflow Addons supports using Python ops for all Tensorflow versions above or equal to 2.9.0 and strictly below 2.12.0 (nightly versions are not supported).
The versions of TensorFlow you are currently using is 2.5.1 and is not supported.
Some things might work, some things might not.
If you were to encounter a bug, do not file an issue.
If you want to make sure you're using a tested and supported configuration, either change the TensorFlow version or the TensorFlow Addons's version.
You can find the compatibility matrix in TensorFlow Addon's readme:
https://github.com/tensorflow/addons
UserWarning,
INFO 2025-06-27 13:41:45,424 tfaip.device.device_config: Setting up device config DeviceConfigParams(gpus=None, gpu_auto_tune=False, gpu_memory=None, soft_device_placement=True, dist_strategy=<DistributionStrategy.DEFAULT: 'default'>)
INFO 2025-06-27 13:41:45,502 calamari_ocr.ocr.savedmodel.sa: Checkpoint version 5 is up-to-date.
INFO 2025-06-27 13:41:45,535 tfaip.device.device_config: Setting up device config DeviceConfigParams(gpus=None, gpu_auto_tune=False, gpu_memory=None, soft_device_placement=True, dist_strategy=<DistributionStrategy.DEFAULT: 'default'>)
WARNING 2025-06-27 13:41:50,355 celery.redirected: Prediction: 0%| | 0/10 [00:00<?, ?it/s]
WARNING 2025-06-27 13:41:59,592 celery.redirected: Prediction: 80%|######## | 8/10 [00:09<00:01, 1.19it/s]
WARNING 2025-06-27 13:41:59,678 celery.redirected: Prediction: 100%|##########| 10/10 [00:09<00:00, 1.07it/s]
WARNING 2025-06-27 13:41:59,685 celery.redirected: performing alignment...
[...]
[2025-06-27 13:42:01,520: INFO/ForkPoolWorker-48] Text Alignment[4366a2d8-1385-40f1-abd2-410ed7d1683a]: writing output to json...
INFO 2025-06-27 13:42:01,522 rodan: ran the task and the returned object is True
INFO 2025-06-27 13:42:03,010 celery.app.trace: Task Text Alignment[4366a2d8-1385-40f1-abd2-410ed7d1683a] succeeded in 138.98804084863514s: 'FINISHED | master_task: 5465a894-4014-4d04-956a-48972c32518c'
Notice that:
INFO 2025-06-27 13:41:45,424 tfaip.device.device_config: Setting up device config DeviceConfigParams(gpus=None, gpu_auto_tune=False, gpu_memory=None, soft_device_placement=True, dist_strategy=<DistributionStrategy.DEFAULT: 'default'>)
and
WARNING 2025-06-27 13:41:45,292 celery.redirected: /usr/local/lib/python3.7/dist-packages/tensorflow_addons/utils/ensure_tf_install.py:67: UserWarning: Tensorflow Addons supports using Python ops for all Tensorflow versions above or equal to 2.9.0 and strictly below 2.12.0 (nightly versions are not supported).
The versions of TensorFlow you are currently using is 2.5.1 and is not supported.
Some things might work, some things might not.
Metadata
Metadata
Assignees
Labels
No labels