FireCrawl Document Loader

​

Node Details

• Name: FireCrawl
• Type: Document
• Category: Document Loaders
• Version: 1.0

​

Parameters

1. Text Splitter (optional)
   • Type: TextSplitter
   • Description: A text splitter to process the loaded documents
2. URLs
   • Type: string
   • Description: URL to be crawled/scraped
3. Crawler type
   • Type: options
   • Options:
     • Crawl: Crawl a URL and all accessible subpages
     • Scrape: Scrape a URL and get its content
   • Default: Crawl
4. Max Crawl Pages (implied from the code)
   • Type: string
   • Description: Maximum number of pages to crawl
5. Generate Image Alt Text (implied from the code)
   • Type: boolean
   • Description: Whether to generate alternative text for images
6. Return Only URLs (implied from the code)
   • Type: boolean
   • Description: Whether to return only URLs without content
7. Only Main Content (implied from the code)
   • Type: boolean
   • Description: Whether to extract only the main content of the page
8. URL Patterns Excludes (implied from the code)
   • Type: string
   • Description: Comma-separated list of URL patterns to exclude from crawling
9. URL Patterns Includes (implied from the code)
   • Type: string
   • Description: Comma-separated list of URL patterns to include in crawling
10. Metadata (optional, implied from the code)
    • Type: string or object
    • Description: Additional metadata to add to the documents

​

Credentials

• FireCrawl API
  • Type: credential
  • Credential Names: fireCrawlApi

​

Input

​

Output

• pageContent: The content of the web page (in Markdown format)
• metadata: Associated metadata for the document

​

Functionality

1. The node initializes a FireCrawlLoader with the provided parameters.
2. It then uses the loader to either crawl or scrape the specified URL(s).
3. The resulting data is converted into Document objects.
4. If a text splitter is provided, the documents are split accordingly.
5. Additional metadata can be added to the documents if specified.

​

Use Cases

• Web scraping for content analysis
• Building training datasets from web content
• Creating knowledge bases from websites
• Automating data collection for research or business intelligence

​

Notes

• The FireCrawl API key is required and should be set up in the credentials.
• The node supports both crawling (multiple pages) and scraping (single page) modes.
• Various options allow for customization of the crawling/scraping process, such as limiting the number of pages, including/excluding URL patterns, and focusing on main content.