Question answering using own dataset

[VikasRathod314]

How much text file can we pass for question answering system. is there any specific data size limitations in Lang chain?
I am getting error when i passed almost 32000 tokens. Getting error as (InvalidRequestError: This model's maximum context length is 8191 tokens, however you requested 24068 tokens (24068 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.).Please suggest.

[tqye2000]

You may need to use splitter = CharacterTextSplitter(separator="\n\n", chunk_size=1024, chunk_overlap=0) to split your long text and then load_qa_with_sources_chain to build the chain.

[VikasRathod314]

Thanks @tqye2000 I passed long text file with embedding (docsearch = FAISS.from_texts(texts, embeddings)) but getting an error (InvalidRequestError: This model's maximum context length is 8191 tokens, however you requested 25058 tokens (25058 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.). please suggest any solution.

[tqye2000]

As the the error message says, your request texts is too long! You need to reduce the input text size. Something like:

text_chunks = []
splitter = CharacterTextSplitter(separator="\n\n", chunk_size=512, chunk_overlap=64)
for chunk in splitter.split_text(long_texts):
    text_chunks.append(Document(page_content=chunk, metadata=source.metadata))

embeddings = OpenAIEmbeddings()
search_index = FAISS.from_texts(text_chunks, embeddings)
  ...