Cheerio Web Scraper

​
Node Details

  • Name: cheerioWebScraper
  • Type: Document
  • Version: 1.1
  • Category: Document Loaders

​
Input Parameters

  1. URL (required)
    • Type: string
    • Description: The URL of the webpage to scrape or the starting point for crawling.
  2. Text Splitter (optional)
    • Type: TextSplitter
    • Description: A text splitter to process the extracted content.
  3. Get Relative Links Method (optional)
    • Type: options
    • Default: “webCrawl”
    • Options:
      • Web Crawl: Crawl relative links from HTML URL
      • Scrape XML Sitemap: Scrape relative links from XML sitemap URL
    • Description: Method to retrieve relative links for multi-page scraping.
  4. Get Relative Links Limit (optional)
    • Type: number
    • Default: 10
    • Description: Limits the number of relative links to process. Set to 0 for no limit.
  5. Selector (CSS) (optional)
    • Type: string
    • Description: CSS selector to target specific content on the page.
  6. Additional Metadata (optional)
    • Type: json
    • Description: Custom metadata to add to the extracted documents.
  7. Omit Metadata Keys (optional)
    • Type: string
    • Description: Comma-separated list of metadata keys to omit from the output. Use ”*” to omit all default metadata.

​
Functionality

  1. Validates the input URL.
  2. Sets up the Cheerio loader with optional CSS selector.
  3. Handles single-page scraping or multi-page crawling based on the selected method.
  4. Applies text splitting if a TextSplitter is provided.
  5. Manages metadata, including adding custom metadata and omitting specified keys.
  6. Handles errors and provides debug logging.

​
Output

The node outputs an array of IDocument objects, each representing a scraped page or section, including the extracted content and associated metadata.

​
Use Cases

  • Web content extraction for analysis or processing
  • Building datasets from web sources
  • Automating data collection from websites
  • Creating custom web crawlers for specific domains

​
Notes

  • The node includes error handling for invalid URLs and unsupported file types (e.g., PDFs).
  • It respects rate limiting and responsible scraping practices.
  • Debug logging is available when the DEBUG environment variable is set to ‘true’.