Feasibility of adding TextComposer::ComposeJSON to output JSON bounding boxes

[phrrngtn]

Hi,
I have been shredding PDFs for some time using a variety of tools (including pdftohtml) and only recently discovered your library. I really like the API and the CMake-based build system which makes it very easy to incorporate into projects. Thanks!

I have a question on how best to add a feature that returns the text elements as JSON representation of the bounding boxes (I plan to package this as a SQLite extension that would allow you to query out the (text) contents of a PDF)

Here is a sample from the XML generated by pdftohtml. I find this schema fine as-is and would like to do something similar with your library but take advantage of how much easier PDFHummus is to incorporate than pdftohtml). I assume that this work is not particularly difficult (basing off what has already been done) but wanted to get your advice before jumping into it.

<pdf2xml producer="poppler" version="20.11.0">
<page number="10" position="absolute" top="0" left="0" height="1188" width="918">
        <fontspec id="0" size="54" family="ArialMT" color="#b6b6b6"/>
        <fontspec id="1" size="16" family="ArialMT" color="#000000"/>
        <fontspec id="2" size="23" family="TrebuchetMS" color="#000000"/>
        <fontspec id="3" size="16" family="Arial" color="#000000"/>
        <fontspec id="4" size="16" family="Arial" color="#999999"/>
<text top="11" left="459" width="15" height="60" font="0"> </text>
<text top="109" left="108" width="701" height="18" font="1">began in the 1960s with merely identifying the systems from space. Researchers then</text>
<text top="131" left="108" width="701" height="18" font="1">developed a technique to estimate intensity from the storm cloud structure and lifetime. See</text>

Let me know if you have any thoughts wrt returning a string (either a single document or one document per page) or as an nlohmann::json data-structure. Because the JSON support in SQLite is very good, I find it easy to write simple scalar functions that return a blob of JSON and then convert that into a table-like value via a Common Table Expression.

I am delighted to have found your library and to start using it.

[galkahana]

Hi Paul, So if i get it right you want to create a json representation of the. TextExtraction::GetResultsAsText translates the textsForPages to a text output representation, and you could create something that does something similar, and maybe it doesn't necessarily have to be code added to TextExtraction but rather something external (up to you, i guess at the time i was and probably still am looking at TextExtraction as an orchestrator but then i could create another level and leave "serializing" out of it. not sure it matters. I see that in the TableExtraction i did both csv and txt outputs as members of it...so i guess for me it depends on the day). Anyways, Use TextExtraction to build the textsForPages structure with TextExtraction::ExtractText and then read textsForPage and compose it to the end result json. You can definitely use TextExtraction::GetResultsAsText iteration as guideline (though it's just a for loop...nothing much to learn there) and the code of TextComposer, which role is to convert a single struct of page to a txt output, to do what job you have in mind. Some parts of TextComposer might be reusable for the task of json writing and it'd make sense to extract them. Not sure the right thing to do is to add stuff to TextComposer itself. Creating a JsonComposer is probably making more sense i guess is what i would do. Good luck 🙂. Regards, Gal.

…

________________________________ From: Paul Harrington ***@***.***> Sent: Thursday, November 21, 2024 11:42 PM To: galkahana/pdf-text-extraction ***@***.***> Cc: Subscribed ***@***.***> Subject: [galkahana/pdf-text-extraction] Feasibility of adding TextComposer::ComposeJSON to output JSON bounding boxes (Issue #27) Hi, I have been shredding PDFs for some time using a variety of tools (including pdftohtml) and only recently discovered your library. I really like the API and the CMake-based build system which makes it very easy to incorporate into projects. Thanks! I have a question on how best to add a feature that returns the text elements as JSON representation of the bounding boxes (I plan to package this as a SQLite extension that would allow you to query out the (text) contents of a PDF) Here is a sample from the XML generated by pdftohtml. I find this schema fine as-is and would like to do something similar with your library but take advantage of how much easier PDFHummus is to incorporate than pdftohtml). I assume that this work is not particularly difficult (basing off what has already been done) but wanted to get your advice before jumping into it. <pdf2xml producer="poppler" version="20.11.0"> <page number="10" position="absolute" top="0" left="0" height="1188" width="918"> <fontspec id="0" size="54" family="ArialMT" color="#b6b6b6"/> <fontspec id="1" size="16" family="ArialMT" color="#000000"/> <fontspec id="2" size="23" family="TrebuchetMS" color="#000000"/> <fontspec id="3" size="16" family="Arial" color="#000000"/> <fontspec id="4" size="16" family="Arial" color="#999999"/> <text top="11" left="459" width="15" height="60" font="0"> </text> <text top="109" left="108" width="701" height="18" font="1">began in the 1960s with merely identifying the systems from space. Researchers then</text> <text top="131" left="108" width="701" height="18" font="1">developed a technique to estimate intensity from the storm cloud structure and lifetime. See</text> Let me know if you have any thoughts wrt returning a string (either a single document or one document per page) or as an nlohmann::json data-structure. Because the JSON support in SQLite is very good, I find it easy to write simple scalar functions that return a blob of JSON and then convert that into a table-like value via a Common Table Expression. I am delighted to have found your library and to start using it. — Reply to this email directly, view it on GitHub<#27>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AACCLZYPWLZFIU5CO2P3AB32BZHSXAVCNFSM6AAAAABSH7N5VSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGY4DCMBWHE2TONY>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[phrrngtn]

Thanks for your response. I have provided some more context below to explain what I am attempting and why I want to do it.

I will hack away at it. My C++ skills are poor (I stopped writing it in in the early 1990's and the language has evolved a lot since then!) but I don't think this will be so difficult as I have a really clear idea of what I want to achieve so I think I will do a lot of the development work in gdb!! (figuring out the data-structures)

I agree with your JsonComposer comment and am amused by your "depends on the day" comment as when I was looking through the source code and looking at the "bundled" vs "unbundled" interfaces and was wondering if there was something more going on and I remember saying to myself "he probably felt like doing it that way that day".

pjjH

There are two applications that I have in mind to build on top of an extract text contents as tuples of (page, font, top, left, width, height, text) mechanism:

1. Geometry/bounding-box table-extraction
2. Using geometrical information to guide placement of dynamically-generated annotations when showing the provenance of query results (especially with results from LLMs)

It is really useful to have a SQL-based mechanism for generating these bounding boxes on demand and my past development efforts in these areas have been frustrated by the need to pre-process entire documents (via pdftohtml) or to "shell out" (again, to pdftohtml or similar command-line tool). It also looks like your API will support a values-based interface i.e. does not need file-names or URLs ... some APIs have very reference-oriented APIS (i.e. paths or URLs) which is totally reasonable and I get why they do it like that but it does not lend itself as well for composition.

[phrrngtn]

I dumped out the localBbox and globalBbox structures from sortedTextCommands() and it looks like a JSON serializer for ParsedTextPlacement and/or ParsedTextPlacementList would deliver the bulk of what is needed, and is trivial to implement. It would also be useful to include some information on fonts/styles/glyphs so I will see about adding that into/alongside ParsedTextPlacement.

[phrrngtn]

The JSON part was indeed trivial but not without a learning curve: I have not been successful yet in getting the font information extracted as I got a bit distracted by cmake and trying to get nlohmann::json added as a dependency. Which leads me to:

and maybe it doesn't necessarily have to be code added to TextExtraction but rather something external

I did not understand your comment the first time around but, of course, since textsForPages is public, the JSON serialization can indeed live outside. In any case, I will continue with the investigation into extraction of the font-related information as it is quite helpful for table-extraction. If I am successful, I will submit a PR that adds the additional fields into TextPlacements. Another enhancement that I would find quite useful are vcpkg.io package definitions for the packages in this repo to make them easy to integrate.

regards,
pjjH

void TextExtraction::ComposeJSON(const ParsedTextPlacementList& inTextPlacements) {
    json res;
    ParsedTextPlacementList::const_iterator itTextPlacement = inTextPlacements.begin();
    for(; itTextPlacement != inTextPlacements.end();++itTextPlacement) {
       res.push_back(json{itTextPlacement->text,
                          itTextPlacement->matrix,
                          itTextPlacement->localBbox,
                          itTextPlacement->globalBbox,
                          itTextPlacement->spaceWidth,
                          itTextPlacement->globalSpaceWidth
                          });
    }
    cout << res;
}

[phrrngtn]

I would appreciate your advice on how to reliably extract and decode the font-related metadata

I was unable to find accessors for FontName, FontFamily, FontStretch in any of the reader code (I did find all of them in FontDescriptorWriter). I was kind of expecting them to be there but I guess if it is not being used for text extraction then perhaps not so surprising that they are not there.

I hacked TextExtraction/lib/font-translation/FontDecoder.cpp to extract FontName and it gave me something semi-plausible (QQAAAA+DejaVuSerif). I added a (string) field to ParsedTextPlacement for debugging and would like to change it to an integer which can be used to lookup the fonts that are used for the text elements on a page.

[galkahana]

So PDF Writer lib doesn't provide much when it comes to font parsing. also - you need to know that in general Font data in PDF is fairly focused on what's interesting for PDF - drawing the glyphs...so it's fairly scarce and tehre's no 100% guarantee to get much more than the drawing and there's no guarantee to even be able to translate the text.
Thanks to the prevalence of the need to copy and paste text from PDF there's more PDFs out there (by process of evolution) that have glyph to text data, and this is utilized by this parser.

There's probably more data in the PDF font definitions that you can read to get what data you are looking for...but be aware that it's not mandatory to have it in the PDF.

Where the text extractor uses and parses font def is in FontDecoder.h (As you identified yourself). That's a class that parses a font dict and a font descriptor to be able to grab such data as glyph to unicode translation and physical data like width and height to be able to understand the dimensions of the text (in order to describe bbox).

You can consider adding more reading of font data either directly under TextInterpeter::OnResourcesRead, side by side with the decoder reading of objects, or add to the decoder itself. For example, FontFamily, FontStretch and such are available for reading in the font descriptor (which is what you saw written by FontDescriptorWriter). You can see examples of parsing the descriptor in FontDecoder::ParseFontDescriptor (so you can add your code there).

The connection to the text itself is made TextInterpeter::OnTextElementComplete (there's a GetDecoderForFont). This is where you can add the data to the outcome ParsedTextPlacement struct.

For anything else, you may want to read the specs a bit (here's a link to the version im using)
https://opensource.adobe.com/dc-acrobat-sdk-docs/pdfstandards/pdfreference1.7old.pdf, specifically on Font and FontDescriptor structs.

[phrrngtn]

Your comments are very useful. Thanks! I will read the specs and put in the minimal font-decoding capabilities I need.
I will use other tools (e.g. pdffonts, pdftohtml) to groundtruth the decoding. This may take me some time to implement as PDF is complex and I don't have any experience with hacking it at a low-level (I am amazed that you single-handedly took on such a mammoth task and started this library in the first place!). Once the library work is done, I would not be at all surprised if my client code turns out to be 20-50 lines of code.

warm regards,
pjjH

[phrrngtn]

Hi!
Thanks for your help. I now have a very clear idea of how to proceed. I don't expect to submit a PR soon but perhaps in early 2025.

Details

I added support (in development) for maintaining and emitting font-id -> font characteristics map; page numbers and bounding boxes (in [x, y, width, height] form). Although I am very happy to have a working implementation (modulo possible memory leaks because I don't have sufficient experience with modern C++), I want to do some more work on it because it is very 'batchy' rather than stream-oriented. To that end, the stretch goals are:

1. Update API to work with streams (rather than files) since I intend to hook it up to SQLite blobs
2. See if the JSON serialization can be stream-friendly (start to emit bounding boxes while parsing pages)

These other goals are quite straightforward and I may do them first before attempting the other ones (which I think should work but don't know yet how to do them)

1. vcpkg support
2. SQLite/duckdb extension
3. fix the spelling of stuff like textInterpeter

138 Arial XKFMKE+ArialMT
143 Arial YMONYC+Arial-ItalicMT
146 Arial OXOFOW+Arial-BoldItalicMT
147 Cambria Math XQCCKE+CambriaMath
151 Trebuchet MS RHASEQ+TrebuchetMS-Italic
2041 Arial XSDMKE+ArialMT
2043 Trebuchet MS FMCISO+TrebuchetMS
2045 Trebuchet MS VXCVII+TrebuchetMS-Bold
2047 Arial KESHKE+Arial-BoldMT
0 2041 [306.000, 746.520,  10.008,  49.140]
0 2041 [432.000,  78.151,   7.939,  14.988] U
0 2041 [439.920,  78.151,  36.695,  14.988] pdated
0 2041 [476.645,  78.151,  15.207,  14.988] Ma
0 2041 [491.949,  78.151,   5.468,  14.988] y
0 2041 [497.408,  78.151,   3.052,  14.988]
0 2041 [500.471,  78.151,  30.448,  14.988] 2024 |
0 2041 [530.813,  78.151,   3.118,  14.988]

[galkahana]

Cheers, dont worry too much about PR...more important that you fully develop a solution that works for you. If there's a way for you to share the work i can maybe help review those pesky memory leaks. especially if it's complexities that the pdf library may introduce.

[phrrngtn]

You can see what I have done in the two commits so far in https://github.com/phrrngtn/pdf-text-extraction fork.
I think they give a good "proof by construction" of what I want to achieve. I do not like my implementation very much (especially the PageNumberList pageNumbers which I used to zip the page-numbers and the list of page-placement lists, ParsedTextPlacementListList) and I would prefer something more stream oriented: rather that having accessors that copy private data-structures from callbacks/event-handlers, I am going to experiment with a StreamLayout or StreamTextExtraction that enqueues decoded PDF fragments (e.g. fonts, pages, media boxes, text placements) that have been re-encoded as something self-evident (e.g. JSON or structs). I plan on building most of the textbox interpretation/reassembly at the database level any/all of my C++ messing is directed at that end goal (which is probably why I am so biased towards a stream/cursor-based design)

I am not even sure if there are any memory leaks, to be honest, as I don't know how STL maps work and I was just messing around with the signatures to get things to compile!
phrrngtn@a00bf77#diff-803950b148cc3b0619d761046ac6d0d40d07e567f29731df4c45f21c8e89b809

In any case, I have enough to proceed with my main work. I don't think my changes should be incorporated into your library in this form but they may give you an idea of additional APIs/capabilities. I will experiment with the stream idea (as I think it should be quite straightforward to integrate into the handle-based architecture). If you have suggestions on how to do idiomatic C++ stream-of-structured-objects using streams/generators then I would be happy to receive them.

[galkahana]

Hi @phrrngtn.
was fairly busy last week, so didn't get a chance to look at your solution.

memory-leak wise, it doesn't look like you do much of dynamic allocation, so i think you are good. if there's one thing im missing is that TextExtraction/lib/font-translation/FontDecoder.cpp seems to have a GetFontDescription that allocates new FontDescription. im not seeing where they are deleted. so this is something to take note of and make sure to free before the app goes away. and well, there's simple enough methods to locate memory leaks with a given main executable, so you'll probably be able to pick up anything else if it's interesting.

As for streams reading. im not sure i totally got you, but do note that PDFWriter has a read stream capacity. if, for example, we look at TextExtraction::ExtractText you'll see that it passes sourceFile.GetInputStream() to parser.StartPDFParsing call. The parser can work with any custom implementation of IByteReaderWithPosition, which is an interface the library defines. it should be very easy to add an override of ExtractText which goes something like EStatusCode TextExtraction::ExtractText(IByteReaderWithPosition* inPDFStream, long inStartPage, long inEndPage) that other than feeding directly from a stream, is quite the same as the original one.
It is true, however, that IByteReaderWithPosition is not forward only, and that has to do with the nature parsing PDF which is fairly random access. but maybe there's a way to read the whole PDF file into mem and then implement IByteReaderWithPosition for that. not sure how this relates to what you have in mind, but figured i'll mention it.
I'm not sure what i think of structured objects representations of PDF. If there's a way to pre-process the PDFs...might as well extract the text no?

Good luck :).