Docling and splitting

[mophilly]

The docling project seems like a great fit for many IDP cases. I have just arrived at the need to split large files for submission to an LLM. It appears that splitting is a fundamental element in the decling examples.

Is splitting, as expressed in ExtractThinker, a complement to docling or would using docling replace it?

[enoch3712]

Hello @mophilly.

DocumentLoader works separately from the Splitter, i made sure that was done in each one.

And works great with docling, the one i would not advice is MarkitDown, because it doesnt allow splitting page out of the box, Docling yes.

To understand a documentLoader, just read this:
https://enoch3712.github.io/ExtractThinker/core-concepts/document-loaders/

always uses one function, load, that contains an array of pages, each page with content and image (if is vision).

If you want to use splitter, just take a look at:
https://github.com/enoch3712/ExtractThinker/blob/main/tests/test_process.py

PS: sorry for the delay, im finishing another article, this one is a big boy!

[mophilly]

Hello, @enoch3712.

Docling was easy to implement in this project. It outputs pages, which is nice.
The markdown result appears to be complete on a small test.

I would like additional output options, such as json, css, and html. Docling offers support for these. How might this fit into the vision for the project?

Adding output options to document_loader_docling.py, def load() seems like one place to put that. On or about line 231 is the assignment of conv_result, which later in the code provides the export_to_markdown. A branching statement could be placed there to support other formats.

OTOH, raising the scope of conv_result in the class would allow for additional methods to invoke specific output types. That might result in more concise code and flexibility.

[enoch3712]

Hello @mophilly !

In terms of ouput options, i will keep it JSON. Why? Because the parsing comes from Pydantic -> JSON. Then you can create the next formats. If you want to keep HTML of CSV, just use Docling.

Remember, that's a bit out of scope, that that is what Docling already does. In this project we take care of the Document Intelligence part

[mophilly]

I don't understand where docling fits in the extract thinker system.
The current docling document loader returns markdown. The Process for classification does accept that; it accepts a file path. Likewise with the Extractor.

[enoch3712]

Docling is the DocumentLoader. Only extracts the content from the document. The rest ET does. beyond that: