Getting TypeError when uploading metadata to Pinecone index

[Abe410]

Hi

I am uploading text files to Pinecone index, and along with it some metadata as follows:

text_df = pd.DataFrame({'text':[text]})

# Load text into loader
loader = DataFrameLoader(text_df)
docs = loader.load()

# split into chunks
text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(chunk_size=500, chunk_overlap=30)
chunks = text_splitter.split_documents(docs)

# Inserting to index
Pinecone.from_documents(chunks, embeddings, index_name=INDEX_NAME, namespace=NAMESPACE, metadatas=[{link_variable}])

But doing so, I get the error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[55], line 38
     35                 chunks = text_splitter.split_documents(docs)
     37                 # Inserting to index
---> 38                 Pinecone.from_documents(chunks, embeddings, index_name=INDEX_NAME, namespace=NAMESPACE, metadatas=[{'hello'}])
     41 print('####### Finished #######')

File ~/.local/lib/python3.10/site-packages/langchain/vectorstores/base.py:332, in VectorStore.from_documents(cls, documents, embedding, **kwargs)
    330 texts = [d.page_content for d in documents]
    331 metadatas = [d.metadata for d in documents]
--> 332 return cls.from_texts(texts, embedding, metadatas=metadatas, **kwargs)

TypeError: langchain.vectorstores.pinecone.Pinecone.from_texts() got multiple values for keyword argument 'metadatas'

What could be the case here? I used the from_texts method and it worked fine with that, but I was getting a metadata size is 46933 bytes, which exceeds the limit of 40960 bytes per vector error with that, but that is from Pinecone I guess.

[dosubot[bot]]

🤖

Hi,

The error you're encountering is due to the way you're passing the metadatas argument to the from_documents method. The from_documents method internally calls the from_texts method, which also accepts a metadatas argument. The metadatas argument is expected to be a list of dictionaries, where each dictionary corresponds to the metadata associated with a text.

In your case, you're passing metadatas as a list of a single dictionary [{link_variable}] to the from_documents method. This is causing a conflict because the from_documents method is also trying to pass a metadatas argument to the from_texts method, resulting in the TypeError: langchain.vectorstores.pinecone.Pinecone.from_texts() got multiple values for keyword argument 'metadatas' error.

To resolve this issue, you should pass the metadatas argument directly to the from_texts method, like so:

text_df = pd.DataFrame({'text':[text]})

# Load text into loader
loader = DataFrameLoader(text_df)
docs = loader.load()

# split into chunks
text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(chunk_size=500, chunk_overlap=30)
chunks = text_splitter.split_documents(docs)

# Prepare texts and metadatas
texts = [d.page_content for d in chunks]
metadatas = [d.metadata for d in chunks]

# Inserting to index
Pinecone.from_texts(texts, embeddings, index_name=INDEX_NAME, namespace=NAMESPACE, metadatas=metadatas)

Regarding the metadata size is 46933 bytes, which exceeds the limit of 40960 bytes per vector error, this is indeed a limitation from Pinecone. Each vector's metadata in Pinecone is limited to 40KB. You'll need to ensure that the size of your metadata does not exceed this limit.

I hope this helps! Let me know if you have any other questions.

Sources

• libs/langchain/langchain/vectorstores/pinecone.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out and providing detailed information about the issue you're encountering. The error message you're seeing is a bit misleading, as it refers to the Pinecone.from_texts method, while you're using the Pinecone.from_documents method.

The Pinecone.from_documents method is a class method that takes in a list of Document objects, an Embeddings object, and several other optional parameters, including metadatas. The metadatas parameter should be a list of dictionaries, where each dictionary corresponds to the metadata for each document in the documents argument.

From the code snippet you've provided, it seems like you're passing a list containing a single dictionary to the metadatas parameter. This could be causing the issue if the length of this list doesn't match the length of your documents list.

Here's how you should be calling the Pinecone.from_documents method:

# Assuming `chunks` is a list of Document objects and `embeddings` is an Embeddings object
metadatas = [{link_variable} for _ in range(len(chunks))]

Pinecone.from_documents(chunks, embeddings, index_name=INDEX_NAME, namespace=NAMESPACE, metadatas=metadatas)

In this example, metadatas is a list of dictionaries, where each dictionary is {link_variable}. The length of metadatas matches the length of chunks.

Please try this and let me know if it resolves your issue. If not, could you please provide more details about the link_variable and its value? This will help me understand the issue better and provide a more accurate solution.

Best,
Dosu

Sources

• libs/langchain/langchain/vectorstores/pinecone.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Abe410]

@dosu-bot what is the equivalent of the Document objects for the Pinecone.from_texts method?

[dosubot[bot]]

🤖

Hello,

The Pinecone.from_texts method in the LangChain Python framework expects an input of type List[str] as an equivalent to Document objects. This is evident from the method signature:

@classmethod
def from_texts(
    cls,
    texts: List[str],
    embedding: Embeddings,
    metadatas: Optional[List[dict]] = None,
    ids: Optional[List[str]] = None,
    batch_size: int = 32,
    text_key: str = "text",
    namespace: Optional[str] = None,
    index_name: Optional[str] = None,
    upsert_kwargs: Optional[dict] = None,
    pool_threads: int = 4,
    embeddings_chunk_size: int = 1000,
    **kwargs: Any,
) -> Pinecone:

Here, the texts parameter is a list of strings, which are the raw documents to be processed.

The Pinecone.from_texts method is a class method that constructs a Pinecone wrapper from raw documents. This method is designed to be a user-friendly interface that embeds documents and adds them to a provided Pinecone index, making it a quick way to get started with the framework.

Here's how it works:

1. It gets a Pinecone index instance by calling the get_pinecone_index method with the provided index name and pool threads.
2. It creates a Pinecone instance with the Pinecone index, embedding, text key, and namespace.
3. It adds the texts to the Pinecone instance by calling the add_texts method with the provided texts, metadatas, ids, namespace, batch size, and embedding chunk size.

You can find more details in the source code of the LangChain Python framework:

• libs/langchain/langchain/vectorstores/pinecone.py
• libs/langchain/tests/integration_tests/vectorstores/test_pinecone.py

I hope this helps! If you have any more questions, feel free to ask.

Best,
Dosu

Sources

• libs/langchain/langchain/vectorstores/pinecone.py
• libs/langchain/tests/integration_tests/vectorstores/test_pinecone.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[ChristheIm]

@dosu-bot I am having same issue with sending metadata. How can I just remove 'text' metadata so I can just send vector embeddings instead of metadata from Pinecone?

[Charlz91108112]

@ChristheIm Possible for you to explain your problem a bit. You might have got the solution already but in case if you are still looking to find a solution then I can help. I faced similar issue and I was able to solve it.

[Charlz91108112]

In order to solve the "metadata size is 46933 bytes, which exceeds the limit of 40960 bytes per vector", I first checked the size of the text chunk and then split the text further using the splitter to get the text chunks. Here is the rough implementation:

EMBEDDING_CTX_LENGTH=8192
if cleaned_chum_size > 35*1024: # 35KB
    embeddings.chunk_size = int(int(EMBEDDING_CTX_LENGTH)/6)
    splitter_low = SemanticChunker(embeddings,  breakpoint_threshold_type="percentile")
chunks = splitter_low.split_text(cleaned_chum)

If you have anything specific, then please let me know. I was using semantic chunker from Langchain. It does really well.