
Node Details
- Name: markdownTextSplitter
- Type: MarkdownTextSplitter
- Category: Text Splitters
- Version: 1.0
Parameters
The Markdown Text Splitter accepts two optional parameters:-
Chunk Size
- Label: Chunk Size
- Name: chunkSize
- Type: number
- Description: Number of characters in each chunk
- Default: 1000
- Optional: Yes
-
Chunk Overlap
- Label: Chunk Overlap
- Name: chunkOverlap
- Type: number
- Description: Number of characters to overlap between chunks
- Default: 200
- Optional: Yes
Input/Output
- Input: The node takes Markdown text as input (implicitly, through the text splitting process).
- Output: The node outputs an instance of the MarkdownTextSplitter class, which can be used to split Markdown text into chunks.
Usage
The Markdown Text Splitter is initialized with the specified chunk size and overlap. When used, it will:- Analyze the structure of the Markdown document.
- Split the text into chunks, respecting Markdown headers as natural break points.
- Ensure each chunk is approximately the specified size (chunkSize).
- Create an overlap between chunks to maintain context (chunkOverlap).
Implementation Details
-
The node uses the
MarkdownTextSplitter
class from the ‘langchain/text_splitter’ library. -
It extends the
INode
interface, making it compatible with the larger node-based system it’s part of. -
The
init
method creates and returns an instance ofMarkdownTextSplitter
with the specified parameters.
Best Practices
- Adjust the chunk size based on your specific use case and the requirements of downstream processes.
- Use a chunk overlap to ensure context is maintained between chunks, especially for tasks that require understanding across section boundaries.
- Consider the structure of your Markdown documents when setting the chunk size to avoid breaking important sections mid-content.