Skip to content

Text Alignment not using GPU properly #1308

@notkaramel

Description

@notkaramel

I was testing Text Alignment on Production (and also for fix1305) and noticed that the job was not using NVIDIA GPU, even with proper setup. nvidia-smi shows nothing.

Here's the log from make gpu-celery_log on Production server:

[2025-06-27 13:39:44,017: INFO/MainProcess] Received task: Text Alignment[4366a2d8-1385-40f1-abd2-410ed7d1683a]
[2025-06-27 13:39:44,114: INFO/ForkPoolWorker-48] started running the task!
[2025-06-27 13:39:44,115: INFO/ForkPoolWorker-48] Text Alignment[4366a2d8-1385-40f1-abd2-410ed7d1683a]: {'OCR Model': 0}
[2025-06-27 13:39:44,988: INFO/ForkPoolWorker-48] Text Alignment[4366a2d8-1385-40f1-abd2-410ed7d1683a]: processing image...
[2025-06-27 13:39:44,988: WARNING/ForkPoolWorker-48] identifying text lines...
WARNING  2025-06-27 13:41:45,292              celery.redirected: /usr/local/lib/python3.7/dist-packages/tensorflow_addons/utils/ensure_tf_install.py:67: UserWarning: Tensorflow Addons supports using Python ops for all Tensorflow versions above or equal to 2.9.0 and strictly below 2.12.0 (nightly versions are not supported).
 The versions of TensorFlow you are currently using is 2.5.1 and is not supported.
Some things might work, some things might not.
If you were to encounter a bug, do not file an issue.
If you want to make sure you're using a tested and supported configuration, either change the TensorFlow version or the TensorFlow Addons's version.
You can find the compatibility matrix in TensorFlow Addon's readme:
https://github.com/tensorflow/addons
  UserWarning,
INFO     2025-06-27 13:41:45,424     tfaip.device.device_config: Setting up device config DeviceConfigParams(gpus=None, gpu_auto_tune=False, gpu_memory=None, soft_device_placement=True, dist_strategy=<DistributionStrategy.DEFAULT: 'default'>)
INFO     2025-06-27 13:41:45,502 calamari_ocr.ocr.savedmodel.sa: Checkpoint version 5 is up-to-date.
INFO     2025-06-27 13:41:45,535     tfaip.device.device_config: Setting up device config DeviceConfigParams(gpus=None, gpu_auto_tune=False, gpu_memory=None, soft_device_placement=True, dist_strategy=<DistributionStrategy.DEFAULT: 'default'>)
WARNING  2025-06-27 13:41:50,355              celery.redirected: Prediction:   0%|          | 0/10 [00:00<?, ?it/s]
WARNING  2025-06-27 13:41:59,592              celery.redirected: Prediction:  80%|########  | 8/10 [00:09<00:01,  1.19it/s]
WARNING  2025-06-27 13:41:59,678              celery.redirected: Prediction: 100%|##########| 10/10 [00:09<00:00,  1.07it/s]
WARNING  2025-06-27 13:41:59,685              celery.redirected: performing alignment...
[...]
[2025-06-27 13:42:01,520: INFO/ForkPoolWorker-48] Text Alignment[4366a2d8-1385-40f1-abd2-410ed7d1683a]: writing output to json...
INFO     2025-06-27 13:42:01,522                          rodan: ran the task and the returned object is True
INFO     2025-06-27 13:42:03,010               celery.app.trace: Task Text Alignment[4366a2d8-1385-40f1-abd2-410ed7d1683a] succeeded in 138.98804084863514s: 'FINISHED  |  master_task: 5465a894-4014-4d04-956a-48972c32518c'

Notice that:

INFO     2025-06-27 13:41:45,424     tfaip.device.device_config: Setting up device config DeviceConfigParams(gpus=None, gpu_auto_tune=False, gpu_memory=None, soft_device_placement=True, dist_strategy=<DistributionStrategy.DEFAULT: 'default'>)

and

WARNING  2025-06-27 13:41:45,292              celery.redirected: /usr/local/lib/python3.7/dist-packages/tensorflow_addons/utils/ensure_tf_install.py:67: UserWarning: Tensorflow Addons supports using Python ops for all Tensorflow versions above or equal to 2.9.0 and strictly below 2.12.0 (nightly versions are not supported).
 The versions of TensorFlow you are currently using is 2.5.1 and is not supported.
Some things might work, some things might not.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions