Text File Document Loader

​
Node Details

  • Name: Text_DocumentLoaders
  • Label: Text File
  • Version: 3.0
  • Type: Document
  • Category: Document Loaders

​
Input Parameters

  1. Txt File (required)
    • Type: file
    • Supported formats: .txt, .html, .aspx, .asp, .cpp, .c, .cs, .css, .go, .h, .java, .js, .less, .ts, .php, .proto, .python, .py, .rst, .ruby, .rb, .rs, .scala, .sc, .scss, .sol, .sql, .swift, .markdown, .md, .tex, .ltx, .vb, .xml
  2. Text Splitter (optional)
    • Type: TextSplitter
    • Purpose: Splits the loaded text into smaller chunks
  3. Additional Metadata (optional)
    • Type: JSON
    • Description: Extra metadata to be added to the extracted documents
  4. Omit Metadata Keys (optional)
    • Type: string
    • Description: Comma-separated list of metadata keys to omit from the default set. Use * to omit all keys except those specified in Additional Metadata.

​
Outputs

  1. Document
    • Description: Array of document objects containing metadata and pageContent
    • Base Classes: Document, json
  2. Text
    • Description: Concatenated string from pageContent of documents
    • Base Classes: string, json

​
Functionality

  1. File Loading:
    • Supports loading from local storage or base64-encoded file data
    • Can handle single files or multiple files (passed as a JSON array)
  2. Text Processing:
    • Uses TextLoader from langchain to load text content
    • Optionally splits text using provided TextSplitter
  3. Metadata Management:
    • Adds user-provided additional metadata
    • Can omit specific or all default metadata keys
    • Merges existing and new metadata
  4. Output Formatting:
    • Can output either as Document objects or concatenated text
    • Handles escape characters in text output

​
Use Cases

  • Loading and processing text-based documents from various sources
  • Preparing text data for further NLP or machine learning tasks
  • Extracting and managing metadata from text documents
  • Splitting large text documents into manageable chunks

​
Notes

  • The node is flexible in handling file inputs, supporting both direct file uploads and references to files in storage
  • It integrates well with text splitting operations, allowing for easy segmentation of large documents
  • The metadata management features provide fine-grained control over what information is attached to each document