How to view the embeddings generated by VectorStoreIndex

[dhirajsuvarna]

How do I see the embeddings that are generated by the VectorStoreIndex for each of the nodes.

[dosubot[bot]]

🤖

Hello @dhirajsuvarna,

I'm Dosu, a bot designed to assist you with your queries and issues related to LlamaIndex. I'm here to help answer your questions, support you in solving bugs, or guide you on becoming a contributor. Don't worry, while we're waiting for a human maintainer, I'll do my best to assist you. Looking forward to helping you navigate through this.

To view the embeddings generated by VectorStoreIndex for each node, you can modify the get_by_id method of the SupabaseVectorStore class. This method currently retrieves the row ids by document id but does not return the actual embeddings as it is set to not include the value and metadata in the query.

Here's how you can modify the get_by_id method to include the value (which is the embedding) in the query:

def get_by_id(self, doc_id: str, **kwargs: Any) -> list:
    """Get row ids and embeddings by doc id.

    Args:
        doc_id (str): document id
    """
    filters = {"doc_id": {"$eq": doc_id}}

    return self._collection.query(
        data=None,
        filters=filters,
        include_value=True,  # Include the value (embedding) in the query
        include_metadata=False,
        **kwargs,
    )

This will return a list of tuples where each tuple contains the row id and the corresponding embedding. Please note that this modification will affect all places where get_by_id is used, so make sure this is the desired behavior.

You can find the original get_by_id method in the llama_index/vector_stores/supabase.py file.

As for the structure of the VectorStoreIndex in the LlamaIndex codebase, it is a class that extends the BaseIndex class and is designed to work with a vector store, which is a type of data store optimized for storing and retrieving vectors. It has several properties and methods for managing nodes in the index, including methods for adding nodes to the index, retrieving nodes from the index, and deleting nodes from the index. You can find more details about the VectorStoreIndex class in the llama_index/indices/vector_store/base.py file.

Please let me know if you need further clarification or assistance.

Sources

• llama_index/vector_stores/supabase.py
• llama_index/indices/vector_store/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dhirajsuvarna]

This doesn't answer my question.

There is one more thing, when I iterate over the nodes of the index, i find that the TextNode object has embeddings=None.
Now, I am wondering where the embeddings are stored.

[marco-ve]

You can use the node ids as keys for querying e.g. a VectorStoreIndex like this:

index.vector_store.get(node_id)