SQLite Record Manager
The SQLite Record Manager is a component used to keep track of document writes into vector databases. It utilizes SQLite as the backend storage system to manage records of document insertions, updates, and deletions.
Node Information
-
Name: SQLiteRecordManager
-
Type: SQLite RecordManager
-
Version: 1.0
-
Category: Record Manager
Purpose
This node is designed to maintain a record of document operations in vector databases. It’s particularly useful for:
-
Tracking when documents were last updated
-
Managing incremental updates to vector stores
-
Facilitating cleanup operations for outdated or deleted documents
Input Parameters
-
Database File Path
-
Type: string
-
Description: The file path where the SQLite database will be stored
-
Example:
C:\Users\User\.ardor\database.sqlite
-
-
Additional Connection Configuration (Optional)
-
Type: JSON
-
Description: Additional SQLite connection options
-
-
Table Name (Optional)
-
Type: string
-
Default: “upsertion_records”
-
Description: Name of the table to store records
-
-
Namespace (Optional)
-
Type: string
-
Description: Namespace for the records. If not specified, chatflowid will be used
-
-
Cleanup
-
Type: options
-
Options:
-
None: No cleanup of old content
-
Incremental: Delete previous versions if content has changed
-
Full: Delete previous versions and remove deleted documents
-
-
Description: Determines how old records are managed
-
-
SourceId Key (Optional)
-
Type: string
-
Default: “source”
-
Description: Key used to identify the true source of a document
-
Functionality
The SQLite Record Manager provides several key functions:
-
Schema Creation: Initializes the SQLite database with the necessary table structure.
-
Record Updates: Allows updating records with new timestamps and group IDs.
-
Existence Checking: Verifies if specific keys exist in the database.
-
Key Listing: Retrieves keys based on various criteria like time range and group IDs.
-
Key Deletion: Removes specified keys from the database.
Usage
This node is typically used in conjunction with vector database operations to:
-
Track when documents were last inserted or updated
-
Facilitate incremental updates by identifying changed documents
-
Support cleanup operations to remove outdated or deleted documents from vector stores
It’s particularly useful in scenarios where you need to maintain an up-to-date vector database, especially when dealing with large document collections that change over time.
Integration
The SQLite Record Manager integrates well with other nodes in a document processing or vector database pipeline. It can be used before or after vector store operations to ensure data consistency and facilitate efficient updates.
Note
Proper configuration of the cleanup mode and sourceId key is crucial for effective incremental updates and maintenance of the vector store. Ensure that your document metadata includes the necessary information (like source identifiers) to fully leverage the capabilities of this record manager.