Pinecone Read Units Usage for SelfQueryRetriever within Langchain

[GurjotTatras]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain.llms import Bedrock
import boto3
from botocore.config import Config
from langchain.vectorstores import Pinecone
from langchain.retrievers.self_query.base import SelfQueryRetriever


bedrock = boto3.client(
        service_name="bedrock-runtime",
        aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
        aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
        region_name=settings.AWS_REGION,
        config=Config(read_timeout=1000),
    )
from langchain_community.chat_models import BedrockChat
index_name = "" #enter index name
llm = BedrockChat(client=bedrock, model_id="anthropic.claude-3-sonnet-20240229-v1:0")

vectorstore = Pinecone.from_existing_index(index_name=index_name, namespace=collection_name, embedding = SentenceTransformerEmbeddings())

document_content_description = ""
metadata_field_info = ""

retriever = SelfQueryRetriever.from_llm(
    vectorstore=vectorstore,
    llm=llm,
    document_contents=document_content_description,
    metadata_field_info=metadata_field_info,
    enable_limit=True,
    return_source_documents=True,
    include_metadata=True,
    search_kwargs={"k": 15},
)

Description

This does return the right results to me, but the default pinecone implemnetation of returning the namespace and usage keys is not returned and only documents are returned.
Could you suggest changes to the SelfQueryRetriever class so that I can fetch the read_units used under usage key in Pinecone's response dict?

System Info

langchain==0.3.3
langchain-aws==0.2.2
langchain-chroma==0.1.4
langchain-community==0.3.2
langchain-core==0.3.10
langchain-openai==0.2.2
langchain-pinecone==0.2.0
langchain-text-splitters==0.3.0
langchainhub==0.1.21

[GurjotTatras]

Or if someone could suggest a similar change in the following class EnsembleRetriever, so that invoking this return the read units from Pinecone

ensemble_retriever = EnsembleRetriever(
            retrievers=[self_query_retriever, hybrid_retriever]
        )