[BUG] Error: vectorsService.upsertVector - Error: Error: Not implemented.

[BadrLaajali]

Describe the bug
When attempting to upsert data from a Notion database into a Postgres vector database using pgvector, an error occurs during the upsert process after modifying a page in Notion. The error message states "Error: vectorsService.upsertVector - Error: Error: Not implemented." Despite this error, new duplicate records are created in the vector database, indicating that the system is not properly recognizing or updating existing records. This occurs even though a Postgres Record Manager is being used to detect modifications in the flow.

To Reproduce
Steps to reproduce the behavior:

1. Set up a Flowise flow with Notion Database, Postgres Record Manager, and Postgres Vector Store nodes.
2. Configure Postgres Record Manager to detect modifications in the flow.
3. Create 3 pages in a Notion database.
4. Execute the initial upsert to create vectors in the Postgres database with pgvector.
5. Modify the title of one of the Notion pages.
6. Run the upsert process again.
7. Observe the error: "Error: vectorsService.upsertVector - Error: Error: Not implemented."
8. Check the Postgres vector database and notice duplicate entries.

Expected behavior
The Postgres Record Manager should detect the modifications in the Notion pages. The upsert process should then recognize existing records based on this information and update them instead of creating duplicates.

Screenshots

Setup

• Installation : Docker using Railway
• Flowise Version : [email protected]
• OS: macOS
• Browser : Arc

Additional context
The issue persists even when switching the Record Manager to Supabase.
The error occurs specifically during the upsert process after modifying existing Notion pages.
The log shows successful initialization of components (Recursive Character Text Splitter, Cohere Embeddings, Notion Database, Postgres Record Manager) before encountering the error.
The error seems to be originating from the Postgres_VectorStores.upsert method, suggesting a potential issue with the implementation of the upsert functionality in the Postgres vector store component.
Log details :

2024-08-13 01:31:13 [DEBUG]: [server]: Initializing Recursive Character Text Splitter (recursiveCharacterTextSplitter_0)

2024-08-13 01:31:13 [DEBUG]: [server]: Finished initializing Recursive Character Text Splitter (recursiveCharacterTextSplitter_0)

2024-08-13 01:31:13 [DEBUG]: [server]: Initializing Cohere Embeddings (cohereEmbeddings_0)

2024-08-13 01:31:13 [DEBUG]: [server]: Finished initializing Cohere Embeddings (cohereEmbeddings_0)

2024-08-13 01:31:13 [DEBUG]: [server]: Initializing Notion Database (notionDB_0)

2024-08-13 01:31:16 [DEBUG]: [server]: Finished initializing Notion Database (notionDB_0)

2024-08-13 01:31:16 [DEBUG]: [server]: Initializing Postgres Record Manager (postgresRecordManager_0)

2024-08-13 01:31:16 [DEBUG]: [server]: Finished initializing Postgres Record Manager (postgresRecordManager_0)

2024-08-13 01:31:16 [DEBUG]: [server]: Upserting Postgres (postgres_0)

2024-08-13 01:31:16 [ERROR]: Error: Not implemented.

Error: Error: Not implemented.

at Postgres_VectorStores.upsert (/usr/local/lib/node_modules/flowise/node_modules/flowise-components/dist/nodes/vectorstores/Postgres/Postgres.js:82:27)

at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

at async buildFlow (/usr/local/lib/node_modules/flowise/dist/utils/index.js:408:37)

at async upsertVector (/usr/local/lib/node_modules/flowise/dist/utils/upsertVector.js:115:32)

at async Object.upsertVectorMiddleware (/usr/local/lib/node_modules/flowise/dist/services/vectors/index.js:9:16)

at async createInternalUpsert (/usr/local/lib/node_modules/flowise/dist/controllers/vectors/index.js:28:29)

2024-08-13 01:31:16 [ERROR]: [server]: Error: Error: Error: Not implemented.

Error: Error: Error: Not implemented.

at buildFlow (/usr/local/lib/node_modules/flowise/dist/utils/index.js:483:19)

at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

at async upsertVector (/usr/local/lib/node_modules/flowise/dist/utils/upsertVector.js:115:32)

at async Object.upsertVectorMiddleware (/usr/local/lib/node_modules/flowise/dist/services/vectors/index.js:9:16)

at async createInternalUpsert (/usr/local/lib/node_modules/flowise/dist/controllers/vectors/index.js:28:29)

[Astriel]

I'm also having some problems with pgvector. It used to work before, but now, each time I'm trying to make an upsert, I'm receveing a OK response, but nothing into the database while it used to work.

[BadrLaajali]

It's working fine with Pinecone and Qdrant.
Maybe it's linked to Postgres pgvector database...

[Astriel]

I've tried with Pinecone, same problem. Flowise says data are upsert, but still nothing in the vector database.

[Astriel]

I'm having the same error if I change my embedding to Bedrock from OpenAI.

[jackborst]

I'm experiencing a similar problem, possibly related to the same root cause.

When using the PostgreSQL Record Manager (RM) and PostgreSQL Vector Store, changing the "cleanup" parameter in the RM module from "none" to "full" results in a "Not implemented" error.

Steps to reproduce:

1. Set up Flowise with PostgreSQL Record Manager and PostgreSQL Vector Store
2. Successfully add new documents to the vector store
3. Change the "cleanup" parameter in the RM module from "none" to "full"
4. Attempt to delete a document or run the upsert process again

Error message:
Error: Error: Error: Not implemented.

Additional observations:

• Adding new documents works fine
• Deleting documents or running upsert after a document is removed from the source folder causes an error
• The issue looks isolated to Flowise's PostgreSQL retriever implementation. Supabase-hosted vectorstore and PostgreSQL Record Manager work correctly for the same operations.

[msiwanHahn]

Hi,
same issue as described above (#3004 (comment) & #3004 (comment)).

Flowise hosted in Azure. Managed PG in Azure, too.

Flow: upsert API Chatflow.json

Additionally the Record Manager keeps PG connections in IDLE "ClientRead" for an unlimited amount of time.
We are reading PDFs from a file share and using the upsert API, same occurs when using the Flowise UI and Upload File from the PDF Loader Node.

[HenryHengZJ]

Here's what I have tried:

OpenAI Embeddings + Postgres Vector Store + Postgres Record Manager

My local Postgres is using Docker, and Flowise is using local git installation

# Compose a postgres database together with the extension pgvector
services:
  db:
    hostname: 127.0.0.1
    image: pgvector/pgvector:pg15
    ports:
     - 5432:5432
    restart: always
    environment:
      - POSTGRES_DB=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=1234
    volumes:
     - ./init.sql:/docker-entrypoint-initdb.d/init.sql

Start the postgres docker using docker compose up

Then
1.) Have a document store with following upsert config:

2.) Upsert data successfully:

3.) Change to Full:

4.) Works correctly as it is able to skipped upserting same documents:

5.) Change to Incremental

6.) See "Not Implemented" error:

So I was able to replicate the error only in Incremental, looks like its opposite from what you observed @jackborst

[HenryHengZJ]

Update: this PR #3180 fixes this