Empty Nodes Returned by get_relevant_documents() in Vector Store for YouTube Transcripts

[dhruv-scogo]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

Use Case:
Loading Transcripts: I am loading YouTube transcripts stored in .txt files from a directory into the vector store using the following method:

@staticmethod
def transcripts_from_directory(transcript_directory: str):
    try:
        reader = TextLoader(file_path=transcript_directory)
        documents = reader.load()
        text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
        documents = text_splitter.split_documents(documents)

        for doc in documents:
            file_path = doc.metadata.get("file_path", "")
            transcript_id = file_path.split("transcript_")[-1].split(".txt")[0]
            yt_link = f"https://www.youtube.com/watch?v={transcript_id}"
            doc.metadata["yt_link"] = yt_link

        logger.info(f"Loaded {len(documents)} documents from {transcript_directory}")
        return documents
    except Exception as e:
        logger.error(f"Error loading documents from directory: {e}")
        raise

Vector Store Retrieval: I use the get_relevant_documents() method to retrieve documents related to a specific query:

vector_store_manager = VectorStoreManager(URI=self.URI)
vectorstore = vector_store_manager._get_vector_store("BEST", "youtube")

chat_prompt = self.prompts.get_timestamp_prompt().format(query=user_query)

llm = self.models.lite_llm
retriever = vectorstore.as_retriever()
nodes = retriever.get_relevant_documents(user_query)
pprint(nodes)

Expected Behavior:
The get_relevant_documents() method should return a list of relevant nodes containing documents related to the user_query.

Actual Behavior:
The method returns an empty list of nodes, despite the documents being successfully loaded and indexed in the vector store.

Additional Context:
The transcripts are stored in .txt format, loaded from a directory, split into chunks, and indexed with metadata containing YouTube links.
The issue persists across multiple queries, and the documents are confirmed to be loaded into the vector store.

### Description

When attempting to retrieve relevant documents from a vector store containing indexed YouTube transcripts, the get_relevant_documents() method consistently returns an empty list of nodes. This issue occurs despite successfully loading and indexing the transcript data.

### System Info

System Information

OS: Windows
OS Version: 10.0.22631
Python Version: 3.11.9 | packaged by Anaconda, Inc. | (main, Apr 19 2024, 16:40:41) [MSC v.1916 64 bit (AMD64)]

Package Information

langchain_core: 0.2.27
langchain: 0.2.12
langchain_community: 0.2.11
langsmith: 0.1.85
langchain_cli: 0.0.28
langchain_mongodb: 0.1.7
langchain_openai: 0.1.7
langchain_text_splitters: 0.2.2
langgraph: 0.2.3
langserve: 0.2.2

[dosubot[bot]]

Hey @dhruv-scogo! 👋 I'm here to help you with bug fixes, answer questions, and guide you on becoming a contributor.

To resolve the issue where the get_relevant_documents() method in a vector store containing indexed YouTube transcripts consistently returns an empty list of nodes, you should ensure that the search_type and search_kwargs are correctly set and that the similarity search methods are properly implemented and invoked.

Here is a checklist to help you debug and resolve the issue:

1. Check search_type and search_kwargs:
   Ensure that the search_type is set to one of the allowed values: "similarity", "similarity_score_threshold", or "mmr". Also, verify that search_kwargs contains the necessary parameters for the search method.

2. Verify Document Indexing:
   Ensure that the documents are correctly indexed in the vector store. You can do this by checking the return value of the add_documents method to confirm that document IDs are being generated.

3. Implement and Test Similarity Search Methods:
   Ensure that the similarity search methods (similarity_search, similarity_search_with_relevance_scores, max_marginal_relevance_search) are correctly implemented in your vector store subclass.

Here is an example of how you might implement and test the similarity_search method:

class MyVectorStore(VectorStore):
    def similarity_search(self, query: str, k: int = 4, **kwargs: Any) -> List[Document]:
        # Implement the similarity search logic here
        # For example, use a pre-trained model to get embeddings and perform a vector search
        # Return a list of Document objects that are most similar to the query
        pass

    def similarity_search_with_relevance_scores(
        self, query: str, k: int = 4, **kwargs: Any
    ) -> List[Tuple[Document, float]]:
        # Implement the similarity search with relevance scores logic here
        pass

    def max_marginal_relevance_search(self, query: str, k: int = 4, **kwargs: Any) -> List[Document]:
        # Implement the max marginal relevance search logic here
        pass

4. Test the get_relevant_documents Method:
   Ensure that the get_relevant_documents method is correctly invoking the search methods and returning the expected results.

Here is an example of how you might test the get_relevant_documents method:

# Create an instance of your vector store
vector_store = MyVectorStore()

# Add documents to the vector store
documents = [Document(page_content="Sample YouTube transcript", metadata={})]
vector_store.add_documents(documents)

# Set the search type and search kwargs
search_type = "similarity"
search_kwargs = {"k": 4}

# Create a query
query = "Sample query text"

# Get relevant documents
relevant_docs = vector_store.get_relevant_documents(query, search_type=search_type, search_kwargs=search_kwargs)

# Print the relevant documents
print(relevant_docs)

By following these steps, you should be able to identify and resolve the issue with the get_relevant_documents() method returning an empty list of nodes.

Additionally, ensure that the search method in your VectorStore class is correctly implemented to handle different search types:

def search(self, query: str, search_type: str, **kwargs: Any) -> List[Document]:
    """Return docs most similar to query using a specified search type.

    Args:
        query: Input text
        search_type: Type of search to perform. Can be "similarity",
            "mmr", or "similarity_score_threshold".
        **kwargs: Arguments to pass to the search method.

    Returns:
        List of Documents most similar to the query.

    Raises:
        ValueError: If search_type is not one of "similarity",
            "mmr", or "similarity_score_threshold".
    """
    if search_type == "similarity":
        return self.similarity_search(query, **kwargs)
    elif search_type == "similarity_score_threshold":
        docs_and_similarities = self.similarity_search_with_relevance_scores(query, **kwargs)
        return [doc for doc, _ in docs_and_similarities]
    elif search_type == "mmr":
        return self.max_marginal_relevance_search(query, **kwargs)
    else:
        raise ValueError(
            f"search_type of {search_type} not allowed. Expected "
            "search_type to be 'similarity', 'similarity_score_threshold' or 'mmr'."
        )

By ensuring that these methods are correctly implemented and tested, you should be able to resolve the issue with the get_relevant_documents() method ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[dhruv-scogo]

Hi @dosu,

Thank you for the guidance. I had already implemented steps 1, 2, and 3 as per your instructions. However, when I executed step 4, I encountered the following error:

'MongoDBAtlasVectorSearch' object has no attribute 'get_relevant_documents'

It seems that the MongoDBAtlasVectorSearch class does not have the get_relevant_documents method. Could you please advise on how to proceed further? Is there an alternative approach or method that I should use for retrieving relevant documents from the vector store?

[dosubot[bot]]

To retrieve relevant documents from the MongoDBAtlasVectorSearch class, you should use the search method instead of get_relevant_documents. Here's how you can proceed:

Using the search Method

You can specify the type of search you want to perform, such as "similarity", "similarity_score_threshold", or "mmr". Here is an example:

# Assuming you have an instance of MongoDBAtlasVectorSearch named `vector_store`
query = "your search query"
search_type = "similarity"  # or "similarity_score_threshold" or "mmr"
relevant_documents = vector_store.search(query, search_type)

Asynchronous Search

If you need to perform the search asynchronously, you can use the asearch method:

import asyncio

# Assuming you have an instance of MongoDBAtlasVectorSearch named `vector_store`
query = "your search query"
search_type = "similarity"  # or "similarity_score_threshold" or "mmr"

async def get_relevant_documents():
    relevant_documents = await vector_store.asearch(query, search_type)
    return relevant_documents

# Run the async function
relevant_documents = asyncio.run(get_relevant_documents())

Example Code for Retrieving Documents

Here is a complete example of how you might retrieve relevant documents:

# Create an instance of MongoDBAtlasVectorSearch
vector_store = MongoDBAtlasVectorSearch(
    collection=MONGODB_COLLECTION,
    embedding=OpenAIEmbeddings(model="text-embedding-3-small"),
    index_name=ATLAS_VECTOR_SEARCH_INDEX_NAME,
    relevance_score_fn="cosine",
)

# Add documents to the vector store
documents = [Document(page_content="Sample YouTube transcript", metadata={})]
vector_store.add_documents(documents)

# Set the search type and search kwargs
search_type = "similarity"
search_kwargs = {"k": 4}

# Create a query
query = "Sample query text"

# Get relevant documents
relevant_docs = vector_store.search(query, search_type=search_type, **search_kwargs)

# Print the relevant documents
print(relevant_docs)

By using the search or asearch methods, you should be able to retrieve the relevant documents from the MongoDBAtlasVectorSearch class ^[1][2].

To continue talking to Dosu, mention @dosu.

[dhruv-scogo]

@dosu
I have tried the above approach but still there is not change in my response
Still there is an empty array [ ] returned meaning there are no relevant_docs fetched

[dosubot[bot]]

The search method of the MongoDBAtlasVectorSearch class might return an empty list due to the pre-filter conditions not being met. Specifically, ensure that the pre-filter conditions match the documents in your vector store.

Here is an example of how to set up and use the MongoDBAtlasVectorSearch class correctly:

1. Ensure Proper Indexing:
   Make sure your MongoDB Atlas vector store is correctly set up and indexed. Here is an example of how to create a vector search index on your MongoDB Atlas cluster:

   {
     "mappings": {
       "dynamic": true,
       "fields": {
         "embedding": {
           "dimensions": 1536,
           "similarity": "cosine",
           "type": "knnVector"
         },
         "genre": {
           "type": "token"
         },
         "ratings": {
           "type": "number"
         },
         "year": {
           "type": "number"
         }
       }
     }
   }

2. Instantiate and Use MongoDBAtlasVectorSearch:
   Here is an example of how to instantiate and use the MongoDBAtlasVectorSearch class:

   from pymongo import MongoClient
   from langchain_mongodb.vectorstores import MongoDBAtlasVectorSearch
   from langchain_openai import OpenAIEmbeddings
   
   # Initialize MongoDB python client
   client = MongoClient(MONGODB_ATLAS_CLUSTER_URI)
   
   DB_NAME = "langchain_test_db"
   COLLECTION_NAME = "langchain_test_vectorstores"
   ATLAS_VECTOR_SEARCH_INDEX_NAME = "langchain-test-index-vectorstores"
   
   MONGODB_COLLECTION = client[DB_NAME][COLLECTION_NAME]
   
   vector_store = MongoDBAtlasVectorSearch(
       collection=MONGODB_COLLECTION,
       embedding=OpenAIEmbeddings(),
       index_name=ATLAS_VECTOR_SEARCH_INDEX_NAME,
       relevance_score_fn="cosine",
   )
   
   # Add documents
   from langchain_core.documents import Document
   
   document_1 = Document(page_content="foo", metadata={"baz": "bar"})
   document_2 = Document(page_content="thud", metadata={"bar": "baz"})
   document_3 = Document(page_content="i will be deleted :(")
   
   documents = [document_1, document_2, document_3]
   ids = ["1", "2", "3"]
   vector_store.add_documents(documents=documents, ids=ids)
   
   # Delete documents
   vector_store.delete(ids=["3"])
   
   # Search
   results = vector_store.similarity_search(query="thud", k=1)
   for doc in results:
       print(f"* {doc.page_content} [{doc.metadata}]")
   
   # Search with filter
   results = vector_store.similarity_search(query="thud", k=1, post_filter=[{"bar": "baz"}])
   for doc in results:
       print(f"* {doc.page_content} [{doc.metadata}]")
   
   # Search with score
   results = vector_store.similarity_search_with_score(query="qux", k=1)
   for doc, score in results:
       print(f"* [SIM={score:3f}] {doc.page_content} [{doc.metadata}]")

3. Check Pre-Filter Conditions:
   Ensure that the pre-filter conditions match the actual metadata values in your documents. For example:

   matches_filter = vector_store.similarity_search(
       "Sandwich", k=3, pre_filter={"c": {"$gt": 0}}
   )
   assert len(matches_filter) == 1

By ensuring that your MongoDB Atlas cluster is set up correctly, your documents and metadata are properly indexed, and the pre-filter conditions match the documents, you should be able to retrieve relevant documents successfully ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.