Smoldocling is stuck

[kerbymart]

Setup

• CPU: Intel(R) Xeon(R) W-2123 CPU @ 3.60GHz Intel64 Family 6 Model 85 Stepping 4
• System RAM: 8GB
• GPU: NVIDIA Quadro P4000

Run docling

docling --pipeline vlm --vlm-model smoldocling --device cuda -vv --from pdf --to md --num-threads 8 --abort-on-error https://arxiv.org/pdf/2206.01062

Problem

The problem is that the docling is stuck but the GPU utilization is at ~30% to ~40% and the VRAM utilization is at 5GB or 8GB total.
Output

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): arxiv.org:443
DEBUG:urllib3.connectionpool:https://arxiv.org:443 "GET /pdf/2206.01062 HTTP/1.1" 200 4310680
INFO:docling.document_converter:Going to convert document batch...
INFO:docling.document_converter:Initializing pipeline for VlmPipeline with options hash ef4ecfe0bf633dff68912f3cd26a99f6
INFO:docling.utils.accelerator_utils:Accelerator device: 'cuda:0'
DEBUG:docling.models.hf_vlm_model:Available device for HuggingFace VLM: cuda:0
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): huggingface.co:443
DEBUG:urllib3.connectionpool:https://huggingface.co:443 "GET /api/models/ds4sd/SmolDocling-256M-preview/revision/main HTTP/1.1" 200 4385
INFO:docling.pipeline.base_pipeline:Processing document 2206.01062v1.pdf
Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.

⏳Even waiting for hours the process does not finish.

[kerbymart]

Running docling with default parameters work:

 docling -vv --num-threads 8 https://arxiv.org/pdf/2206.01062
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): arxiv.org:443
DEBUG:urllib3.connectionpool:https://arxiv.org:443 "GET /pdf/2206.01062 HTTP/1.1" 200 4310680
INFO:docling.document_converter:Going to convert document batch...
INFO:docling.document_converter:Initializing pipeline for StandardPdfPipeline with options hash fb50e9d66ef387dbb5642315ca1df089
INFO:docling.utils.accelerator_utils:Accelerator device: 'cuda:0'
INFO:docling.utils.accelerator_utils:Accelerator device: 'cuda:0'
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): huggingface.co:443
DEBUG:urllib3.connectionpool:https://huggingface.co:443 "GET /api/models/ds4sd/docling-models/revision/v2.1.0 HTTP/1.1" 200 1266
DEBUG:docling_ibm_models.layoutmodel.layout_predictor:LayoutPredictor settings: {'safe_tensors_file': 'C:\\Users\\Kerby\\.cache\\huggingface\\hub\\models--ds4sd--docling-models\\snapshots\\36bebf56681740529abd09f5473a93a69373fbf0\\model_artifacts\\layout\\model.safetensors', 'device': 'cuda', 'num_threads': 8, 'image_size': 640, 'threshold': 0.3}
DEBUG:urllib3.connectionpool:https://huggingface.co:443 "GET /api/models/ds4sd/docling-models/revision/v2.2.0 HTTP/1.1" 200 1266
INFO:docling.utils.accelerator_utils:Accelerator device: 'cuda:0'
INFO:docling.models.factories.base_factory:Loading plugin 'docling_defaults'
INFO:docling.models.factories:Registered picture descriptions: ['vlm', 'api']
INFO:docling.pipeline.base_pipeline:Processing document 2206.01062v1.pdf
DEBUG:docling.pipeline.base_pipeline:Finished converting page batch time=152749.359
DEBUG:docling.pipeline.base_pipeline:Finished converting page batch time=152774.687
DEBUG:docling.pipeline.base_pipeline:Finished converting page batch time=152800.171
INFO:docling.document_converter:Finished converting document 2206.01062v1.pdf in 79.44 sec.
INFO:docling.cli.main:writing Markdown output to 2206.01062v1.md
INFO:docling.cli.main:Processed 1 docs, of which 0 failed
INFO:docling.cli.main:All documents were converted in 79.62 seconds.

[kerbymart]

Changing OCR to --ocr-engine easyocr allowed it to finish within 3087.00 seconds (almost an hour 🤔)

[nriemenschneider]

I am facing the same performance issue on my GTX 1080 8GB. Converting the sample PDF "source = "https://arxiv.org/pdf/2408.09869" # document per local path or URL" takes 11 seconds with the regular pipeline, the VlmPipeline takes ~10 minutes.
And worst of all, result.document.export_to_markdown() exports an empty string. But result.pages shows the page contents correctly as doctags.

[geoHeil]

I can observe something similar - is it clear how to get smoldocling/vlm to operate and produce more than the empty string?

[geoHeil]

#1513 is this related/ the root cause? as docling --pipeline vlm my_document.pdf just returns an empty markdown file

[Saidgurbuz]

The issue in #1513 is resolved in the latest version (v2.33.0). Please update Docling to get rid of this issue.

[nriemenschneider]

Updating docling to 2.33 has fixed the empty markdown export when using Smoldocling in the VlmPipeline. But it still takes 3.5 Minutes to process just the first two pages of the Docling Technical report. I assume that is mainly due to my dated hardware. Although my GPU rarely goes above 30% load and 30% power usage, so I guess I have more ressources to throw at the problem.

[geoHeil]

for me MLX is fast - but the cuda options are quite slow (on my macbook without proper acceleration)