The Markdown Text Splitter is a specialized text splitting component designed to divide Markdown content into smaller, manageable chunks based on Markdown headers. This node is particularly useful for processing large Markdown documents while maintaining the structural integrity of the content.
Name: markdownTextSplitter
Type: MarkdownTextSplitter
Category: Text Splitters
Version: 1.0
The Markdown Text Splitter accepts two optional parameters:
Chunk Size
Label: Chunk Size
Name: chunkSize
Type: number
Description: Number of characters in each chunk
Default: 1000
Optional: Yes
Chunk Overlap
Label: Chunk Overlap
Name: chunkOverlap
Type: number
Description: Number of characters to overlap between chunks
Default: 200
Optional: Yes
Input: The node takes Markdown text as input (implicitly, through the text splitting process).
Output: The node outputs an instance of the MarkdownTextSplitter class, which can be used to split Markdown text into chunks.
The Markdown Text Splitter is initialized with the specified chunk size and overlap. When used, it will:
Analyze the structure of the Markdown document.
Split the text into chunks, respecting Markdown headers as natural break points.
Ensure each chunk is approximately the specified size (chunkSize).
Create an overlap between chunks to maintain context (chunkOverlap).
This node is particularly useful in workflows where you need to process large Markdown documents while preserving their structure, such as in document analysis, content summarization, or when preparing text for large language models with input size limitations.
The node uses the MarkdownTextSplitter
class from the ‘langchain/text_splitter’ library.
It extends the INode
interface, making it compatible with the larger node-based system it’s part of.
The init
method creates and returns an instance of MarkdownTextSplitter
with the specified parameters.
Adjust the chunk size based on your specific use case and the requirements of downstream processes.
Use a chunk overlap to ensure context is maintained between chunks, especially for tasks that require understanding across section boundaries.
Consider the structure of your Markdown documents when setting the chunk size to avoid breaking important sections mid-content.