Getting image descriptions to export to markdown

[thearn]

Hi, loving the library so far, but am running into a snag. I have the following code to convert a pdf to markdown. I'd like to have picture descriptions with the file:

pipeline_options = PdfPipelineOptions()
pipeline_options.generate_page_images = False
pipeline_options.generate_picture_images = True
pipeline_options.enable_remote_services=True

pipeline_options.do_code_enrichment = True
pipeline_options.do_formula_enrichment = True
pipeline_options.do_picture_classification = True
pipeline_options.do_picture_description = True

pipeline_options.picture_description_options = (
    smolvlm_picture_description  # <-- the model choice
)
pipeline_options.picture_description_options.prompt = (
    "Describe the image in three sentences. Be consise and accurate."
)

doc_converter = DocumentConverter(
    format_options={
        InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)
    }
)

result = doc_converter.convert(pdf_file_name)
md = result.document.export_to_markdown("out.md", image_mode=ImageRefMode.EMBEDDED)

but the resulting markdown code never seems to contain the descriptions of the images. Am I not invoking the right options/commands, or is this a limitation of the markdown exporter? The examples in the documentation unfortunately didn't make it clear.

[mudassir206]

you can refer this code--
def main():
logging.basicConfig(level=logging.INFO)

# input_doc_path = Path("./tests/data/pdf/2206.01062.pdf")
input_doc_path = Path("./tests/data/pdf/2305.03393v1-pg9.pdf")

pipeline_options = VlmPipelineOptions(
    enable_remote_services=True  # <-- this is required!
)

# The ApiVlmOptions() allows to interface with APIs supporting
# the multi-modal chat interface. Here follow a few example on how to configure those.

# One possibility is self-hosting model, e.g. via Ollama.
# Example using the Granite Vision  model: (uncomment the following lines)
pipeline_options.vlm_options = ollama_vlm_options(
    model="granite3.2-vision:2b",
    prompt="OCR the full page to markdown.",
)

# Another possibility is using online services, e.g. watsonx.ai.
# Using requires setting the env variables WX_API_KEY and WX_PROJECT_ID.
# Uncomment the following line for this option:
# pipeline_options.vlm_options = watsonx_vlm_options(
#     model="ibm/granite-vision-3-2-2b", prompt="OCR the full page to markdown."
# )

# Create the DocumentConverter and launch the conversion.
doc_converter = DocumentConverter(
    format_options={
        InputFormat.PDF: PdfFormatOption(
            pipeline_options=pipeline_options,
            pipeline_cls=VlmPipeline,
        )
    }
)
result = doc_converter.convert(input_doc_path)
print(result.document.export_to_markdown())

[dolfim-ibm]

We have now the possibility to customize how the serialization is done for each format and/or chunking. We will (very) soon have examples on how you can easily get the description in markdown and in your chunks.

[Bill-XU]

Hi @dolfim-ibm ,

Are there any updates on this?
I have the same issue as well.
I implemented a custom OCR plugin which uses GPT-4o to perform OCR and semantic analysis on images.
The plugin and my code work well on some PDFs, markdown files exported from them have texts extracted by custom OCR plugin.
But it fails on one of my PDFs, there are only <!-- image placeholder --> in exported markdown files.

I have tried dumping the result of conversion to json, and found that none of ocr results exists in the json.
I think it is related to conversion process, but don't know how to manipulate its behavior or where to investigate.
Do you have any idea?

Bill

[dolfim-ibm]

You can find examples on how to write a custom serializer at

• docs: https://docling-project.github.io/docling/concepts/serialization/
• examples: https://docling-project.github.io/docling/examples/serialization/

similar to how the picture description is done, you could add the text of the ocr-ed images in the output with a custom serializer.

[Bill-XU]

Hi @dolfim-ibm

Thank you for the advice. Here are my feedbacks.

TL;DR

Serializer does not work, the cell has been dropped during conversion.
In an OCR model, the indices of text cells created by the model should not be duplicated with existing cells.
If index of a cell is duplicating with any other cells, it may be dropped by LayoutPostProcessor in case the cell is not assigned to a layout cluster.
The RapidOcrModel needs to be fixed as it currently sets indices of text cells regardless of indices of existing cells.
(Maybe other ocr models need fix as well)

Debug & Trace

1. This happens during document conversion (DocumentConvertor.convert), before serialization (or export);
2. Since I am retrieving texts from a PDF, the document convertor uses StandardPdfPipeline to do the work;
3. Inside the pipeline, 5 models are executed one by one during building document stage:
   • PagePreprocessingModel
   • ocr model (can be customized)
   • LayoutModel
   • TableStructureModel
   • PageAssembleModel
4. After the execution of LayoutModel, the cells generated by ocr model disappeared.
5. Inside the LayoutModel, the document is processed as a whole image for layout detection, which outputs a list of cluster. After then, the LayoutPostProcessor processes the clusters with existing cells.
6. Inside the method _process_regular_clusters of LayoutPostProcessor, the existing cells will first be assigned to clusters determined by overlapping rate of their bounding boxes, and unassigned cells will be collected for creating new clusters.
7. And here, the method "_find_unassigned_cells" of LayoutPostProcessor conflicts with what ocr model is doing. Inside "_find_unassigned_cells", a unassigned cell is determined by whether its index does exist in any cell of any cluster. So, If a cell with the same index is already assigned to a cluster, the unassigned one will be ignored (dropped).

Here are related source code of LayoutPostProcessor.

docling/docling/utils/layout_postprocessor.py

Lines 607 to 614 in 45265bf

	def _find_unassigned_cells(self, clusters: List[Cluster]) -> List[TextCell]:
	"""Find cells not assigned to any cluster."""
	assigned = {cell.index for cluster in clusters for cell in cluster.cells}
	return [
	cell
	for cell in self.cells
	if cell.index not in assigned and cell.text.strip()
	]

and related source code of RapidOcrModel.

docling/docling/models/rapid_ocr_model.py

Lines 110 to 134 in 45265bf

	cells = [
	TextCell(
	index=ix,
	text=line[1],
	orig=line[1],
	confidence=line[2],
	from_ocr=True,
	rect=BoundingRectangle.from_bounding_box(
	BoundingBox.from_tuple(
	coord=(
	(line[0][0][0] / self.scale)
	+ ocr_rect.l,
	(line[0][0][1] / self.scale)
	+ ocr_rect.t,
	(line[0][2][0] / self.scale)
	+ ocr_rect.l,
	(line[0][2][1] / self.scale)
	+ ocr_rect.t,
	),
	origin=CoordOrigin.TOPLEFT,
	)
	),
	)
	for ix, line in enumerate(result)
	]

You can see the index of TextCell created by RapidOcrModel begins from 0 as ix is a variable from enumerate method.

I tried changing my custom ocr model setting index to begin after the last index of existing cells, and it worked, the cells created by ocr model can be exported correctly.

Bill

[Bill-XU]

Hi @dolfim-ibm

Here I attach the PDFs for reproducing the issue.
cannot_reproduce_the_issue.pdf
can_reproduce_the_issue.pdf

Bill

[Bill-XU]

@dolfim-ibm

Sorry, previously I said " exported correctly", in which I have missed some other things that I have done.
I will create issues for them.