Text Splitters Overview

Available Components

Splits text based on a specified character separator for versatile general text splitting needs

Splits code documents based on language-specific syntax, ideal for processing code files

Converts HTML to Markdown and splits based on headers for processing HTML content

Splits Markdown content based on headers while maintaining document structure

Splits documents recursively using multiple separators for customizable text splitting

Splits text based on token count using TikToken library for language model preparation

Text Splitters are beneficial in various scenarios, including:

Preparing large documents for processing by language models with input size limitations
Breaking down text for more granular analysis or summarization tasks
Splitting content for parallel processing or distributed computing
Enhancing information retrieval and question-answering systems by creating