How to use custom embedding model?

[xrd]

If I run this without USE_GLUCOSE=1 the code works. But, when I run with that env var, it crashes with:

(.venv) (base) chrisdawson@Chriss-MacBook-Air qdrant-experiments % USE_GLUCOSE=1 python run.py
2024-06-25 16:50:01.871 | WARNING  | fastembed.embedding:<module>:7 - DefaultEmbedding, FlagEmbedding, JinaEmbedding are deprecated.Use from fastembed import TextEmbedding instead.
Traceback (most recent call last):
  File "/Users/chrisdawson/Projects/qdrant-experiments/run.py", line 11, in <module>
    embedding_model = TextEmbedding(model_name="pkshatech/GLuCoSE-base-ja", max_length=768)
  File "/Users/chrisdawson/Projects/qdrant-experiments/.venv/lib/python3.9/site-packages/fastembed/text/text_embedding.py", line 77, in __init__
    raise ValueError(
ValueError: Model pkshatech/GLuCoSE-base-ja is not supported in TextEmbedding.Please check the supported models using `TextEmbedding.list_supported_models()`

The code I'm trying to use with qdrant is here (more or less copied from https://github.com/qdrant/qdrant-rag-eval/blob/master/agentic_rag_with_unify/notebook/Unify_Qdrant_Agentic_RAG.ipynb):

(I have this model working with chromadb with a custom embedding function.)

import qdrant_client
import datetime
import json
import numpy as np
from typing import Tuple, List
from fastembed.embedding import TextEmbedding
import os

client = qdrant_client.QdrantClient(location=":memory:")
if os.getenv("USE_GLUCOSE"):
    embedding_model = TextEmbedding(model_name="pkshatech/GLuCoSE-base-ja", max_length=768)
    qdrant_client.set_model(embedding_model_name=embedding_model_name)

def default_encoder(obj):
    if isinstance(obj, datetime.datetime):
        return obj.isoformat()
    raise TypeError(f"Object of type {type(obj)} is not JSON serializable")

case_2 = [
    {
        "body": "Hey Jane! My name is John, and I'm part of the engineering team on XYZ, I would love to ask you to present your paper 'Cool Agents stuff' on one of our reading group sessions!"
    },
    {
        "body": "Hey Alice! My name is John, and I'm part of the engineering team on XYZ, I would love to ask you to present your paper 'Cool Rag stuff' on one of our reading group sessions!"
    },
]

case_1 = [
    {
        "body": "Hey John! Sorry, something urgent has come up. I won't be able to join."
    },
    {
        "body": "Yeah, I think I should be able to join!"
    },
]

def np_email_array(email_cases: List[List[str]]) -> Tuple[np.ndarray, np.ndarray]:
    """
    Convert a list of email threads into numpy arrays.
    
    Args:
        email_cases (List[List[str]]): A list of email threads, where each thread is a list of strings.
    
    Returns:
        Tuple[np.ndarray, np.ndarray]: A tuple containing two numpy arrays:
            - The first array contains individual emails.
            - The second array contains string representations of emails.
    """
    emails_array = []
    emails_array_str = []
    for thread in email_cases:
        for email in thread:
            emails_array_str.append(str(email))
            emails_array.append(email)
    return np.array(emails_array), np.array(emails_array_str)

email_cases = [case_1, case_2]
email_strings = [json.dumps(email, default=default_encoder) for email in email_cases]
emails_array, emails_array_str = np_email_array(email_cases)
print(emails_array)
client.add(collection_name="bodies", documents=emails_array_str, )
stuff = client.query(query_text="rag", collection_name="bodies")
print(f"Stuff: {stuff}")

[atarora]

Qdrant is agnostic to the embedding models , while the code above uses Fastembed to generate the embeddings leveraging one of the supported models

Looks like a more relevant of a model request ticket for https://github.com/qdrant/fastembed/issues.

[xrd]

qdrant/fastembed#294