MarkdownTextSplitter vs. MarkdownHeaderTextSplitter for chunking

[4MIR2000]

Hi all,
do you know what the difference between the MarkdownTextSplitter and MarkdownHeaderTextSplitter is. Which one would you choose?
Can you also define chunk_size and overlap in MarkdownHeaderTextSplitter?

Thank you in advance

[dosubot[bot]]

Hey @4MIR2000, I'm here to help you out with any questions or issues you might have. Let's tackle this together! 😊

The MarkdownTextSplitter and MarkdownHeaderTextSplitter differ in their approach to splitting markdown documents:

• MarkdownTextSplitter is a general-purpose splitter that divides text based on markdown syntax without focusing on headers.
• MarkdownHeaderTextSplitter specifically splits text based on headers defined in the markdown document. It allows you to specify which headers to split on and can retain or strip these headers from the output.

If you want to split text based on specific headers and possibly retain header information in metadata, MarkdownHeaderTextSplitter is the better choice.

Regarding chunk_size and overlap in MarkdownHeaderTextSplitter, these parameters are not directly supported. Instead, you can use another splitter like RecursiveCharacterTextSplitter after splitting with MarkdownHeaderTextSplitter to control chunk size and overlap:

# MD splits
markdown_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=headers_to_split_on, strip_headers=False
)
md_header_splits = markdown_splitter.split_text(markdown_document)

# Char-level splits
from langchain_text_splitters import RecursiveCharacterTextSplitter

chunk_size = 250
chunk_overlap = 30
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=chunk_size, chunk_overlap=chunk_overlap
)

# Split
splits = text_splitter.split_documents(md_header_splits)

This method allows you to first split the document by headers and then refine the chunks by size and overlap using another splitter ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}