Controlled requests to external inference provider.

[swtb3-ryder]

Requested feature

When calling an external ollama server, we don't want to overwhelm it. Could we use httpx sephamore to control the number of active requests in a given batch?

[dolfim-ibm]

The parallelism comes from these two points:

• https://github.com/docling-project/docling/blob/main/docling/models/api_vlm_model.py#L70
• https://github.com/docling-project/docling/blob/main/docling/models/picture_description_api_model.py#L63

The default is concurrency=1, so I don't really see where you see concurrent requests to Ollama.

Do you have a few more examples/details?

[swtb3]

Ahh I thought when doing convert_all() it would possibly call the external provider from each worker in the map. If the requests aren't concurrent I guess this feature doesn't matter. Unless there is performance to be gained from concurrently making the calls.