Puppeteer Web Scraper

​

Node Details

• Name: puppeteerWebScraper
• Type: Document
• Category: Document Loaders
• Version: 1.0

​

Input Parameters

1. URL (required)

   • Type: string
   • Description: The URL of the webpage to scrape.

2. Text Splitter (optional)

   • Type: TextSplitter
   • Description: A text splitter to process the scraped content.

3. Get Relative Links Method (optional)

   • Type: options
   • Options:
     • Web Crawl: Crawl relative links from HTML URL
     • Scrape XML Sitemap: Scrape relative links from XML sitemap URL
   • Default: Web Crawl

4. Get Relative Links Limit (optional)

   • Type: number
   • Default: 10
   • Description: Limit the number of relative links to retrieve. Set to 0 for no limit.

5. Wait Until (optional)

   • Type: options
   • Options:
     • Load: When the initial HTML document’s DOM has been loaded and parsed
     • DOM Content Loaded: When the complete HTML document’s DOM has been loaded and parsed
     • Network Idle 0: Navigation is finished when there are no more than 0 network connections for at least 500 ms
     • Network Idle 2: Navigation is finished when there are no more than 2 network connections for at least 500 ms

6. Wait for selector to load (optional)

   • Type: string
   • Description: CSS selector to wait for before scraping (e.g., “.div” or “#div”)

7. Additional Metadata (optional)

   • Type: json
   • Description: Additional metadata to be added to the extracted documents

8. Omit Metadata Keys (optional)

   • Type: string
   • Description: Comma-separated list of metadata keys to omit from the final document. Use ”*” to omit all default metadata.

​

Output

The node outputs an array of IDocument objects, each representing a scraped webpage with its content and metadata.

​

Functionality

1. Validates the input URL
2. Configures Puppeteer options based on input parameters
3. Scrapes the specified URL(s) using Puppeteer
4. Processes the scraped content with a text splitter if provided
5. Handles relative link crawling if specified
6. Applies additional metadata and omits specified metadata keys
7. Returns the processed documents

​

Use Cases

• Web content extraction for analysis or processing
• Building training datasets from web content
• Automating web research tasks
• Creating web archives or snapshots

​

Notes

• The node uses headless Chrome for scraping, which allows it to handle JavaScript-rendered content.
• Be mindful of the website’s terms of service and robots.txt file when using this scraper.
• Large-scale scraping may require additional considerations for rate limiting and respecting server resources.