Markdown Text Splitter

​

Node Details

• Name: markdownTextSplitter

• Type: MarkdownTextSplitter

• Category: Text Splitters

• Version: 1.0

​

Parameters

The Markdown Text Splitter accepts two optional parameters:

1. Chunk Size

   • Label: Chunk Size

   • Name: chunkSize

   • Type: number

   • Description: Number of characters in each chunk

   • Default: 1000

   • Optional: Yes

2. Chunk Overlap

   • Label: Chunk Overlap

   • Name: chunkOverlap

   • Type: number

   • Description: Number of characters to overlap between chunks

   • Default: 200

   • Optional: Yes

​

Input/Output

• Input: The node takes Markdown text as input (implicitly, through the text splitting process).

• Output: The node outputs an instance of the MarkdownTextSplitter class, which can be used to split Markdown text into chunks.

​

Usage

The Markdown Text Splitter is initialized with the specified chunk size and overlap. When used, it will:

1. Analyze the structure of the Markdown document.

2. Split the text into chunks, respecting Markdown headers as natural break points.

3. Ensure each chunk is approximately the specified size (chunkSize).

4. Create an overlap between chunks to maintain context (chunkOverlap).

This node is particularly useful in workflows where you need to process large Markdown documents while preserving their structure, such as in document analysis, content summarization, or when preparing text for large language models with input size limitations.

​

Implementation Details

• The node uses the MarkdownTextSplitter class from the ‘langchain/text_splitter’ library.

• It extends the INode interface, making it compatible with the larger node-based system it’s part of.

• The init method creates and returns an instance of MarkdownTextSplitter with the specified parameters.

​

Best Practices

• Adjust the chunk size based on your specific use case and the requirements of downstream processes.

• Use a chunk overlap to ensure context is maintained between chunks, especially for tasks that require understanding across section boundaries.

• Consider the structure of your Markdown documents when setting the chunk size to avoid breaking important sections mid-content.