Trying to Change the Node ID while using TokenTextSplitter and IngestionPipeline

[saadk8]

Hi Guys, i have been stuck with this, i hope someone can help me. Firstofall, you guys did a really good job building this package and its really helpful, also this is the first time i am seeking help (i am a new developer). Hope i get answer on this.

So i have been trying to change the Node IDs (id_).
This is my code to load documents:

from llama_index.core import SimpleDirectoryReader
documents = SimpleDirectoryReader("./data").load_data()

Now i have used the code below to successfully build nodes:

from llama_index.core.node_parser import TokenTextSplitter
from llama_index.core.ingestion import IngestionPipeline
text_splitter = TokenTextSplitter(
separator="\n\nQuestion:", chunk_size=512, chunk_overlap=0,
)
pipeline = IngestionPipeline(
transformations=[text_splitter],
)
nodes = pipeline.run(
documents=documents,
in_place=True,
show_progress=True,
)

But the thing is i can change Node IDs if i try something like:
nodes[0].id_ = "my-new-node-id"

But i dont want to do something like this, because that way i am not able to change the node IDs in the "nodes[0].relationships". I also dont want to build nodes relationships from scratch because that is really confusing, otherwise i think we can specify the node IDs while manually constructing Nodes. But i dont want to build relationships manually which is confusing. So is their a way all these nodes IDs (the ones in the relationsips also) get updated while i use TokenTextSplitter and IngestionPipeline? or is their someother way to do this? Please help.

[dosubot[bot]]

🤖

Hey @saadk8, great to see you diving into LlamaIndex and reaching out for help! Always here to lend a hand.

I'm glad to hear that you're finding the LlamaIndex package useful. To change the Node IDs, you can provide your own function to the id_func parameter when you create an instance of TokenTextSplitter. This function will be used to generate the IDs for all nodes, including those in relationships.

Here's an example of how you can do this:

def my_id_func(index, document):
    return f"my-new-node-id-{index}"

text_splitter = TokenTextSplitter(
    separator="\n\nQuestion:", 
    chunk_size=512, 
    chunk_overlap=0,
    id_func=my_id_func,  # Use your custom function here
)

pipeline = IngestionPipeline(
    transformations=[text_splitter],
)

nodes = pipeline.run(
    documents=documents,
    in_place=True,
    show_progress=True,
)

In this example, my_id_func is a function that takes an integer and a Document object and returns a string. This function generates unique IDs in the way you want. You can modify this function to suit your needs.

Please note that the id_func function should always return a unique ID for each node. If two nodes have the same ID, it may cause unexpected behavior.

You can find more information about this in the TokenTextSplitter class in the LlamaIndex repository.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• llama-index-core/llama_index/core/node_parser/text/token.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.