Create language-image conversations from a Docling Document

[ceberam]

Background

• The DoclingDocument is the self-contained datatype that encapsulates a document converted by Docling. Leveraging the pydantic library, it defines components, such as tables, lists, or headers that are typically found in documents from different formats like PDF, HTML, or docx.
• In a Docling Document, the content of a document is organized in either the body (the main content like paragraphs or tables) or the furniture (auxiliary components such as headers or footers). These content layers are defined in the ContentLayer class.
• Among several AI downstream applications of Docling documents, one can leverage data transforms (see Data Prep Kit) to build large-scale multimodal training datasets.
• Using synthetically data generation, those data transforms can be used to build datasets for instruction tuning large language models (LLMs) like Granite Vision 3.2 2B. Docling recently released a new open-source library for language question-answering SDG (docling-sdg).
• A popular multimodal instrucion-following dataset is LLaVA-Instruct-150K, which includes 58K conversations.

Goal

The goal is to provide the Docling project with integrated tools for building multimodal datasets. In particular:

• extend the data schema of the Docling Document to self-contain synthetic conversations
• enable the serialization of those conversations
• support text, tables, and images

What to do

This is a high-level description of the required development.

• In docling-core project:
  • Add a new content layer to identify items generated from a DoclingDocument and to differentiate them from its content. For instance, a content layer named generated.
  • Create a data schema that defines language-image instruction-following samples. For instance, a pydantic model named conversation.
    • Ensure that this data model is able to encapsulate the concepts in popular datasets like DocVQA and others related to the document visual question answering challenge.
    • The conversations should support several layers, not just disconnected question-answer-context items.
  • Extend the DoclingDocument with an optional component for those conversations, which should be stored under the new content layer.
    • The conversations should be self-contained as part of a Docling Document instance.
    • The provenance of the conversation context should be referenced (the parts of the documents in which the conversation is grounded).
• As a proof-of-concept, we should be able to derive a dataset from the DocVQA dataset with Docling Documents containing the conversations.
• In a Docling project repository, provide a feature that creates synthetically-generated conversations from Docling Document content, leveraging multimodal models. These data should be attached to the Docling documents with the schema defined above.
  • This feature could be implemented as a docling-core transform tool, similar to the chunkers, or by extending the capabilities of the Docling's SDG for question-answering docling-sdg.
• In docling-core, create a new serializer that exports the Docling document conversations in a widely adopted format of instruction-following data for instruction tuning LLMs. For instance, the format in LLaVA-Instruct-150K dataset.

[PeterStaar-IBM]

@ceberam Very nice! Exactly what we need!

@cau-git Please let me know what you think.

[dolfim-ibm]

@ceberam @cau-git @PeterStaar-IBM The content layer proposal works pretty good for the use case you explain above.

However, I wonder if this is the scalable design approach we can continue to take for other use cases.
For example, what about the extracted/enriched data like NLP annotations or KV extractions?
We discussed a few times the idea of keeping the DoclingDocument "pure" (i.e. only what is really in the document) and wrapping it in an auxiliary structure to save generated content.

In short, if we take now this route, very likely we won't come back to the "pure" DoclingDocument approach.

[PeterStaar-IBM]

@dolfim-ibm KV extractions are already now in the KeyValueItem. I think the concept of the "pure" DoclingDocument is very ill defined, the moment we introduced the concept of content-layers. The more I think about content-layers, the more I like it: annotations are part of the document at the end, in a very similar way to slide-notes or manual annotation notes in PDF (there is literally an /Annots key in the pdf standard).

[dolfim-ibm]

Yes, if we embrace this design, we should simply continue with it. I'm mostly pointing out that going in this direction we should stop thinking in the other ways.

Let's validate how good this would match to other type of extraction/generation. (I'm pretty optimistic it works well)

• NLP entities could fit the point raised by @cau-git about the prov coming from "less than a full doc item" (i.e. the char span? the table cells?)
• Schema-based extraction. Those would be structured output referring to some doc item (if not the whole document).
• Should picture description be ported to a new content type as well? They overlap quite a bit with the concept of generated content.
• What about other picture annotations (classification, charts understanding, chemistry annotations)?

[PeterStaar-IBM]

1. picture description and picture annotations should remain where they are. I dont think they live in a different content layer, they are simply a non-pixel representation of the image/chart.
2. Schema-based extraction: yes, I would put those in a different content-layer, similar with NLP based extractions.

I think the difference between the picture annotations and the NLP annotations is obvious: the NLP annotations depend on the model (you could do POS, specific entities, etc). The picture annotations should be independent of the model in the ideal case: they represent the content in the picture.

[anastasds]

In another project, I've also independently come up with the idea of "decorating" base data with what I called "views" and you are calling "content layers" - the idea being that the original data is immutable and then synthetic augmentation is a "view" of that data from some sort of context, which may be thrown away and regenerated when there are better models, change in business requirements, whatever. Same idea, different names, so consider this comment a "+1".

I'm reminded of the shearing layers idea; what that paper discusses is informed by ideas in architecture (traditional, brick and stone architecture, not software architecture) of identifying which parts of a structure get changed at different rates, and then applies that to software architecture - I think that this may be a rediscovery of the same idea for information architecture: here the "shearing layers" would be something like document -> extracted content -> model annotation -> synthetic augmentation (in order of increasing expected rate of change).

[gnusupport]

How do I use docling with Granite Vision model?

[PeterStaar-IBM]

please look here: https://github.com/docling-project/docling/blob/main/docs/examples/minimal_vlm_pipeline.py#L40