VectorstoreIndexCreator: How to increase max_tokens for OpenAI output (default response is 256 tokens..)

[mikolodz]

Hello,

I have just jumped into the world of the langchain. Using some online tutorials I managed to setup a working colab using OpenAI API and Langchain, but I use VectorstoreIndexCreator and it seems like I can't change the max_tokens variable for OpenAI API and it outputs only 256 tokens every time.

How to increase the response max length?

I use the code below:

from langchain.document_loaders import UnstructuredPDFLoader
from langchain.indexes import VectorstoreIndexCreator
from langchain.llms import OpenAI
from langchain.chains.question_answering import load_qa_chain
from langchain.embeddings import OpenAIEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from openai.embeddings_utils import get_embedding
from langchain.vectorstores import Chroma

# set API key for OpenAI
import os
os.environ["OPENAI_API_KEY"] = "my-key"

# connect Google Drive
from google.colab import drive
drive.mount('/content/gdrive', force_remount=True)
root_dir = "/content/gdrive/My Drive/"

# read PDF files in GD directory
pdf_folder_path = f'{root_dir}/pdfs/'
os.listdir(pdf_folder_path)

# loading pdf file/files
loaders = [UnstructuredPDFLoader(os.path.join(pdf_folder_path, fn)) for fn in os.listdir(pdf_folder_path)]

# creating index using VectorstoreIndexCreator
index = VectorstoreIndexCreator(
    text_splitter=CharacterTextSplitter(chunk_size=1000, chunk_overlap=0),
    embedding=OpenAIEmbeddings(),
    vectorstore_cls=Chroma
).from_loaders(loaders)

# query the model
index.query('Prepare a contract based on the product specification')

# output is only 256 tokens length – IDK how to generate longer responses

[mikolodz]

Proper way to set the max_tokens value is to do it in a query method like:

index.query(llm=OpenAI(max_tokens=1000, model_name="text-davinci-003"), question='<query-text>')