Storing text to Milvus Vectore Store

[sbyim]

We are using LLamaIndex + Milvus, where LlamaIndex is used for metadata extraction + ingestion.
It seems Llamaindex Milvus module does not store text from the nodes as default, am i correct?
If so, is storing texts as metadata is the right way to do?
We don't want to use extra document store to store nodes..

def add(self, nodes: List[BaseNode], **add_kwargs: Any) -> List[str]: """Add the embeddings and their nodes into Milvus.

    Args:
        nodes (List[BaseNode]): List of nodes with embeddings
            to insert.

    Raises:
        MilvusException: Failed to insert data.

    Returns:
        List[str]: List of ids inserted.
    """
    insert_list = []
    insert_ids = []

    # Process that data we are going to insert
    for node in nodes:
        entry = node_to_metadata_dict(node)
        entry[MILVUS_ID_FIELD] = node.node_id
        entry[self.embedding_field] = node.embedding

        insert_ids.append(node.node_id)
        insert_list.append(entry)

    # Insert the data into milvus
    self.collection.insert(insert_list)
    self.collection.flush()

[dosubot[bot]]

🤖

Hey @sbyim, back for more LlamaIndex adventures, I see! Good to have you around.

Yes, you are correct. The LlamaIndex Milvus module does not store text from the nodes by default. The add method you've shown is designed to insert the embeddings and their associated nodes into Milvus. The node_to_metadata_dict(node) function is used to convert the node into a dictionary format that can be inserted into Milvus. This dictionary includes the node's ID and its embedding, but not its text.

As for storing the text as metadata, it depends on your specific use case. If you need to frequently access the text associated with each node and want to avoid the overhead of retrieving it from the Milvus collection each time, storing it as metadata could be beneficial. However, this would also increase the memory footprint of your application.

If you decide to store the text as metadata, you would need to modify the node_to_metadata_dict(node) function to include the node's text. Here's an example of how you might do this:

def node_to_metadata_dict(node):
    entry = {
        MILVUS_ID_FIELD: node.node_id,
        self.embedding_field: node.embedding,
        'text': node.text  # Add this line to include the node's text
    }
    return entry

Then, in the add method, you would add the node's text to the insert_list:

for node in nodes:
    entry = node_to_metadata_dict(node)
    insert_list.append(entry)

This would store the node's text in Milvus along with its ID and embedding. Please note that this would increase the amount of data stored in Milvus, which could impact performance and memory usage.

I hope this helps! If you have any further questions, please don't hesitate to ask.

Sources

• llama_index/readers/milvus.py
• llama_index/extractors/metadata_extractors.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.