Possible to ignore headers/footers?

[OldishCoder]

I have some pdf documents that have headers, footers and sidebar elements that I do not want to parse/include.

After reading the documentation and some code, I've arrived at this, which runs (slowly) but totally ignores the directives to skip headers and footers and could not find even references to sidebars, let alone a way to skip them...:

from pathlib import Path
from docling.datamodel.pipeline_options import PdfPipelineOptions, EasyOcrOptions, AcceleratorOptions
from docling.document_converter import DocumentConverter, PdfFormatOption, PipelineOptions, InputFormat

# 1. Create artifacts directory
artifacts_path = Path("./docling_artifacts")
artifacts_path.mkdir(exist_ok=True)

# 2a set acceleration options
accelerator_options = AcceleratorOptions(
    device="cuda:0"  # or "cuda:0" for first GPU
)

# 2b set OCR options
ocr_options = EasyOcrOptions(
    lang=["en"],             # or your preferred language(s)
    download_enabled=True    # <-- This enables model downloads!
)
 
# 2c Configure PDF pipeline options (VALID parameters only)
pipeline_opts = PdfPipelineOptions(
    artifacts_path=artifacts_path,
    do_ocr=True,         # Required for text extraction
    do_table_structure=True,  # Disable table analysis if not needed
    ocr_options=ocr_options
)

# 3. Configure PDF format options with region exclusion
pdf_format_option = PdfFormatOption(
    pipeline_options=pipeline_opts,
    exclude_regions=["Page-header", "Page-footer"],  # Correct placement
    reading_order="natural"
)

# 4. Initialize converter
format_options = {
    InputFormat.PDF: pdf_format_option
}
converter = DocumentConverter(format_options=format_options)

# 5. Process document
result = converter.convert("CorporateStandard.pdf")
print(result.document.export_to_markdown())```

Is it possible to actually skip headers and footers? (and maybe sidebar elements?)

[dolfim-ibm]

Page headers, footers and sidebars should are already skipped in the markdown output. This is done by the our layout detection model which is flagging them as furniture elements.

[OldishCoder]

this command should make a visualization showing which layout components have been identified for the regions you are looking for. this will give a clue on why they get ignored.

Where and how would I see this? Is it embedded in the Markdown it produces? Does it show up in some other file? I'm not seeing anything other than the same markdown it produces without the directive. Are there some "worked" examples?

[OldishCoder]

@dolfim-ibm ? I'm still stuck with header/footer spam in my conversions. Any other ideas?

[OldishCoder]

@AnyBody? - Still stuck...

[simonschoe]

Correct my if I am wrong but:

• The layout model in docling does support detecting a predefined set of layout elements. Importantly, it was not explicitely trained for detecting sidebar-type elements.
• Most built-in conversion functions in Docling (e.g., PDF export to markdown) skip detected page footer and page header elements since they are automatically flagged as furniture.
• Therefore, sidebars will always appear in the final output, since they are not explicitely classified by docling. You could check (using the visualization tips by @dolfim-ibm above) which labels are assigned to sidebars in your examples. For example, they might be classified PICTURE or simply as TEXT. Either way, you would have to implement some custom logic to skip these. With regards to footers and headers: If you find that these are still appearing in the final markdown output, this (to my understanding) means that docling does not recognize them as PAGE_HEADER or PAGE_FOOTER. Again, you could use the visualizations to infer how they are missclassified and then use some custom logic to correct this or contribute your documents as training data to the docling team, such that they can further improve the layout model (especially in terms of detecting footer and header elements).

[OldishCoder]

@simonschoe, my issue is not with sidebars.... smoldocling seems to ignore that element.

MY issue is that for my document, smoldocling is NOT detecting and ignoring the header and footer. How can I fix lack of detection for header and footer? MY documents have a large amount of text and even logo-graphics in the header and footer and I only want the body text.

The visualization tips from @dolfim-ibm do not seem to work. When I add the docling --debug-visualize-layout FILE I do not see anything? Where will I see it? Please see more detail above.