PDF/DOC -> Pinecone + Web browser / Serp API as a Tools/Inputs together for Chat Agents?

[urbanscribe]

I've been able to

• convert my docs into embeddings and stored into pinecone index/namespace
• use that index/namespace embedding in pinecone to power a conversational agent
• and SEPERATELY, create a number of conversational agents which use collected web browser (directed) or Serp (search) web results to chain them and use ChatGPT for inference

bu i have yet to succeed at building BOTH capabilities today in a project. Can't seem to be able to find an agent that can take an existing pinecone embedding store as well as a tool entry point (for browsing/serp) to chain BOTH retrieved results into ChatGPT (etc)

said another way, I am trying to

• create embeddings from my docs, store them in pinecone
• specify some websites or specific some searches which would lead live results
• use semantic retrieval on both to get the best chunks from both sources
• use that in chaining my conversation with an agent

is that possible either now or in the future?
Thanks in advance.

Alex

[HenryHengZJ]

you can achieve this by using ChainTool:

Try copy paste below onto a new canvas:

{"nodes":[{"width":300,"height":277,"id":"serpAPI_0","position":{"x":1684.9706729083161,"y":-31.722621385807116},"type":"customNode","data":{"id":"serpAPI_0","label":"Serp API","name":"serpAPI","type":"SerpAPI","baseClasses":["SerpAPI","Tool"],"category":"Tools","description":"Wrapper around SerpAPI - a real-time API to access Google search results","inputParams":[{"label":"Serp Api Key","name":"apiKey","type":"password"}],"inputAnchors":[],"inputs":{},"outputAnchors":[{"id":"serpAPI_0-output-serpAPI-SerpAPI|Tool","name":"serpAPI","label":"SerpAPI","type":"SerpAPI | Tool"}],"selected":false},"selected":false,"positionAbsolute":{"x":1684.9706729083161,"y":-31.722621385807116},"dragging":false},{"width":300,"height":142,"id":"calculator_0","position":{"x":1725.5837990265256,"y":316.43545930063783},"type":"customNode","data":{"id":"calculator_0","label":"Calculator","name":"calculator","type":"Calculator","baseClasses":["Calculator","Tool"],"category":"Tools","description":"Perform calculations on response","inputParams":[],"inputAnchors":[],"inputs":{},"outputAnchors":[{"id":"calculator_0-output-calculator-Calculator|Tool","name":"calculator","label":"Calculator","type":"Calculator | Tool"}],"selected":false},"selected":false,"positionAbsolute":{"x":1725.5837990265256,"y":316.43545930063783},"dragging":false},{"width":300,"height":279,"id":"vectorDBQAChain_0","position":{"x":1034.1209991774203,"y":312.59111975871537},"type":"customNode","data":{"id":"vectorDBQAChain_0","label":"VectorDB QA Chain","name":"vectorDBQAChain","type":"VectorDBQAChain","baseClasses":["VectorDBQAChain","BaseChain"],"category":"Chains","description":"QA chain for vector databases","inputParams":[],"inputAnchors":[{"label":"Language Model","name":"model","type":"BaseLanguageModel","id":"vectorDBQAChain_0-input-model-BaseLanguageModel"},{"label":"Vector Store","name":"vectorStore","type":"VectorStore","id":"vectorDBQAChain_0-input-vectorStore-VectorStore"}],"inputs":{"model":"{{openAI_1.data.instance}}","vectorStore":"{{pineconeExistingIndex_0.data.instance}}"},"outputAnchors":[{"id":"vectorDBQAChain_0-output-vectorDBQAChain-VectorDBQAChain|BaseChain","name":"vectorDBQAChain","label":"VectorDBQAChain","type":"VectorDBQAChain | BaseChain"}],"selected":false},"selected":false,"positionAbsolute":{"x":1034.1209991774203,"y":312.59111975871537},"dragging":false},{"width":300,"height":471,"id":"openAI_1","position":{"x":649.0214134160765,"y":-74.86211370060312},"type":"customNode","data":{"id":"openAI_1","label":"OpenAI","name":"openAI","type":"OpenAI","baseClasses":["OpenAI","BaseLLM","BaseLanguageModel"],"category":"LLMs","description":"Wrapper around OpenAI large language models","inputParams":[{"label":"OpenAI Api Key","name":"openAIApiKey","type":"password"},{"label":"Model Name","name":"modelName","type":"options","options":[{"label":"text-davinci-003","name":"text-davinci-003"},{"label":"text-davinci-002","name":"text-davinci-002"},{"label":"text-curie-001","name":"text-curie-001"},{"label":"text-babbage-001","name":"text-babbage-001"}],"default":"text-davinci-003","optional":true},{"label":"Temperature","name":"temperature","type":"number","default":0.7,"optional":true}],"inputAnchors":[],"inputs":{"modelName":"text-davinci-003","temperature":"0"},"outputAnchors":[{"id":"openAI_1-output-openAI-OpenAI|BaseLLM|BaseLanguageModel","name":"openAI","label":"OpenAI","type":"OpenAI | BaseLLM | BaseLanguageModel"}],"selected":false},"selected":false,"positionAbsolute":{"x":649.0214134160765,"y":-74.86211370060312},"dragging":false},{"width":300,"height":601,"id":"chainTool_0","position":{"x":1385.7876556310375,"y":301.1018172104151},"type":"customNode","data":{"id":"chainTool_0","label":"Chain Tool","name":"chainTool","type":"ChainTool","baseClasses":["ChainTool","Tool"],"category":"Tools","description":"Use a chain as allowed tool for agent","inputParams":[{"label":"Chain Name","name":"name","type":"string","placeholder":"state-of-union-qa","id":"chainTool_0-input-name-string"},{"label":"Chain Description","name":"description","type":"string","rows":3,"placeholder":"State of the Union QA - useful for when you need to ask questions about the most recent state of the union address.","id":"chainTool_0-input-description-string"},{"label":"Return Direct","name":"returnDirect","type":"boolean","optional":true,"id":"chainTool_0-input-returnDirect-boolean"}],"inputAnchors":[{"label":"Base Chain","name":"baseChain","type":"BaseChain","id":"chainTool_0-input-baseChain-BaseChain"}],"inputs":{"name":"state-of-union-qa","description":"State of the Union QA - useful for when you need to ask questions about the most recent state of the union address.","returnDirect":false,"baseChain":"{{vectorDBQAChain_0.data.instance}}"},"outputAnchors":[{"id":"chainTool_0-output-chainTool-ChainTool|Tool","name":"chainTool","label":"ChainTool","type":"ChainTool | Tool"}],"outputs":{},"selected":false},"selected":false,"positionAbsolute":{"x":1385.7876556310375,"y":301.1018172104151},"dragging":false},{"width":300,"height":552,"id":"pineconeExistingIndex_0","position":{"x":651.5547843299339,"y":424.6009806171603},"type":"customNode","data":{"id":"pineconeExistingIndex_0","label":"Pinecone Load Existing Index","name":"pineconeExistingIndex","type":"Pinecone","baseClasses":["Pinecone","BaseRetriever"],"category":"Vector Stores","description":"Load existing index from Pinecone (i.e: Document has been upserted)","inputParams":[{"label":"Pinecone Api Key","name":"pineconeApiKey","type":"password","id":"pineconeExistingIndex_0-input-pineconeApiKey-password"},{"label":"Pinecone Environment","name":"pineconeEnv","type":"string","id":"pineconeExistingIndex_0-input-pineconeEnv-string"},{"label":"Pinecone Index","name":"pineconeIndex","type":"string","id":"pineconeExistingIndex_0-input-pineconeIndex-string"}],"inputAnchors":[{"label":"Embeddings","name":"embeddings","type":"Embeddings","id":"pineconeExistingIndex_0-input-embeddings-Embeddings"}],"inputs":{"embeddings":"{{openAIEmbeddings_0.data.instance}}","pineconeEnv":"us-west4-gcp","pineconeIndex":"test"},"outputAnchors":[{"name":"output","label":"Output","type":"options","options":[{"id":"pineconeExistingIndex_0-output-retriever-Pinecone|BaseRetriever","name":"retriever","label":"Pinecone Retriever","type":"Pinecone | BaseRetriever"},{"id":"pineconeExistingIndex_0-output-vectorStore-Pinecone|VectorStore","name":"vectorStore","label":"Pinecone Vector Store","type":"Pinecone | VectorStore"}],"default":"retriever"}],"outputs":{"output":"vectorStore"},"selected":false},"selected":false,"positionAbsolute":{"x":651.5547843299339,"y":424.6009806171603},"dragging":false},{"width":300,"height":277,"id":"openAIEmbeddings_0","position":{"x":291.9693257708366,"y":298.91918778665286},"type":"customNode","data":{"id":"openAIEmbeddings_0","label":"OpenAI Embeddings","name":"openAIEmbeddings","type":"OpenAIEmbeddings","baseClasses":["OpenAIEmbeddings","Embeddings"],"category":"Embeddings","description":"OpenAI API to generate embeddings for a given text","inputParams":[{"label":"OpenAI Api Key","name":"openAIApiKey","type":"password","id":"openAIEmbeddings_0-input-openAIApiKey-password"}],"inputAnchors":[],"inputs":{},"outputAnchors":[{"id":"openAIEmbeddings_0-output-openAIEmbeddings-OpenAIEmbeddings|Embeddings","name":"openAIEmbeddings","label":"OpenAIEmbeddings","type":"OpenAIEmbeddings | Embeddings"}],"outputs":{},"selected":false},"selected":false,"positionAbsolute":{"x":291.9693257708366,"y":298.91918778665286},"dragging":false},{"width":300,"height":382,"id":"conversationalAgent_0","position":{"x":2216.9525858642587,"y":333.46190229666524},"type":"customNode","data":{"id":"conversationalAgent_0","label":"Conversational Agent","name":"conversationalAgent","type":"AgentExecutor","baseClasses":["AgentExecutor","BaseChain","BaseLangChain"],"category":"Agents","description":"Conversational agent for a chat model. It will utilize chat specific prompts","inputParams":[{"label":"System Message","name":"systemMessage","type":"string","rows":4,"optional":true,"additionalParams":true,"id":"conversationalAgent_0-input-systemMessage-string"},{"label":"Human Message","name":"humanMessage","type":"string","rows":4,"optional":true,"additionalParams":true,"id":"conversationalAgent_0-input-humanMessage-string"}],"inputAnchors":[{"label":"Allowed Tools","name":"tools","type":"Tool","list":true,"id":"conversationalAgent_0-input-tools-Tool"},{"label":"Language Model","name":"model","type":"BaseLanguageModel","id":"conversationalAgent_0-input-model-BaseLanguageModel"},{"label":"Memory","name":"memory","type":"BaseChatMemory","id":"conversationalAgent_0-input-memory-BaseChatMemory"}],"inputs":{"tools":["{{serpAPI_0.data.instance}}","{{chainTool_0.data.instance}}","{{calculator_0.data.instance}}"],"model":"{{chatOpenAI_0.data.instance}}","memory":"{{bufferMemory_0.data.instance}}","systemMessage":"","humanMessage":""},"outputAnchors":[{"id":"conversationalAgent_0-output-conversationalAgent-AgentExecutor|BaseChain|BaseLangChain","name":"conversationalAgent","label":"AgentExecutor","type":"AgentExecutor | BaseChain | BaseLangChain"}],"outputs":{},"selected":false},"selected":false,"positionAbsolute":{"x":2216.9525858642587,"y":333.46190229666524},"dragging":false},{"width":300,"height":523,"id":"chatOpenAI_0","position":{"x":1759.4715980278233,"y":624.4545559521196},"type":"customNode","data":{"id":"chatOpenAI_0","label":"ChatOpenAI","name":"chatOpenAI","type":"ChatOpenAI","baseClasses":["ChatOpenAI","BaseChatModel","BaseLanguageModel","BaseLangChain"],"category":"Chat Models","description":"Wrapper around OpenAI large language models that use the Chat endpoint","inputParams":[{"label":"OpenAI Api Key","name":"openAIApiKey","type":"password","id":"chatOpenAI_0-input-openAIApiKey-password"},{"label":"Model Name","name":"modelName","type":"options","options":[{"label":"gpt-4","name":"gpt-4"},{"label":"gpt-4-0314","name":"gpt-4-0314"},{"label":"gpt-4-32k-0314","name":"gpt-4-32k-0314"},{"label":"gpt-3.5-turbo","name":"gpt-3.5-turbo"},{"label":"gpt-3.5-turbo-0301","name":"gpt-3.5-turbo-0301"}],"default":"gpt-3.5-turbo","optional":true,"id":"chatOpenAI_0-input-modelName-options"},{"label":"Temperature","name":"temperature","type":"number","default":0.9,"optional":true,"id":"chatOpenAI_0-input-temperature-number"},{"label":"Max Tokens","name":"maxTokens","type":"number","optional":true,"additionalParams":true,"id":"chatOpenAI_0-input-maxTokens-number"},{"label":"Top Probability","name":"topP","type":"number","optional":true,"additionalParams":true,"id":"chatOpenAI_0-input-topP-number"},{"label":"Frequency Penalty","name":"frequencyPenalty","type":"number","optional":true,"additionalParams":true,"id":"chatOpenAI_0-input-frequencyPenalty-number"},{"label":"Presence Penalty","name":"presencePenalty","type":"number","optional":true,"additionalParams":true,"id":"chatOpenAI_0-input-presencePenalty-number"},{"label":"Timeout","name":"timeout","type":"number","optional":true,"additionalParams":true,"id":"chatOpenAI_0-input-timeout-number"}],"inputAnchors":[],"inputs":{"modelName":"gpt-3.5-turbo","temperature":0.9,"maxTokens":"","topP":"","frequencyPenalty":"","presencePenalty":"","timeout":""},"outputAnchors":[{"id":"chatOpenAI_0-output-chatOpenAI-ChatOpenAI|BaseChatModel|BaseLanguageModel|BaseLangChain","name":"chatOpenAI","label":"ChatOpenAI","type":"ChatOpenAI | BaseChatModel | BaseLanguageModel | BaseLangChain"}],"outputs":{},"selected":false},"selected":false,"positionAbsolute":{"x":1759.4715980278233,"y":624.4545559521196},"dragging":false},{"width":300,"height":376,"id":"bufferMemory_0","position":{"x":1384.5109149593525,"y":967.5652968294461},"type":"customNode","data":{"id":"bufferMemory_0","label":"Buffer Memory","name":"bufferMemory","type":"BufferMemory","baseClasses":["BufferMemory","BaseChatMemory","BaseMemory"],"category":"Memory","description":"Remembers previous conversational back and forths directly","inputParams":[{"label":"Memory Key","name":"memoryKey","type":"string","default":"chat_history","id":"bufferMemory_0-input-memoryKey-string"},{"label":"Input Key","name":"inputKey","type":"string","default":"input","id":"bufferMemory_0-input-inputKey-string"}],"inputAnchors":[],"inputs":{"memoryKey":"chat_history","inputKey":"input"},"outputAnchors":[{"id":"bufferMemory_0-output-bufferMemory-BufferMemory|BaseChatMemory|BaseMemory","name":"bufferMemory","label":"BufferMemory","type":"BufferMemory | BaseChatMemory | BaseMemory"}],"outputs":{},"selected":false},"selected":false,"positionAbsolute":{"x":1384.5109149593525,"y":967.5652968294461},"dragging":false}],"edges":[{"source":"openAI_1","sourceHandle":"openAI_1-output-openAI-OpenAI|BaseLLM|BaseLanguageModel","target":"vectorDBQAChain_0","targetHandle":"vectorDBQAChain_0-input-model-BaseLanguageModel","type":"buttonedge","id":"openAI_1-openAI_1-output-openAI-OpenAI|BaseLLM|BaseLanguageModel-vectorDBQAChain_0-vectorDBQAChain_0-input-model-BaseLanguageModel","data":{"label":""}},{"source":"vectorDBQAChain_0","sourceHandle":"vectorDBQAChain_0-output-vectorDBQAChain-VectorDBQAChain|BaseChain","target":"chainTool_0","targetHandle":"chainTool_0-input-baseChain-BaseChain","type":"buttonedge","id":"vectorDBQAChain_0-vectorDBQAChain_0-output-vectorDBQAChain-VectorDBQAChain|BaseChain-chainTool_0-chainTool_0-input-baseChain-BaseChain","data":{"label":""}},{"source":"pineconeExistingIndex_0","sourceHandle":"pineconeExistingIndex_0-output-vectorStore-Pinecone|VectorStore","target":"vectorDBQAChain_0","targetHandle":"vectorDBQAChain_0-input-vectorStore-VectorStore","type":"buttonedge","id":"pineconeExistingIndex_0-pineconeExistingIndex_0-output-vectorStore-Pinecone|VectorStore-vectorDBQAChain_0-vectorDBQAChain_0-input-vectorStore-VectorStore","data":{"label":""}},{"source":"openAIEmbeddings_0","sourceHandle":"openAIEmbeddings_0-output-openAIEmbeddings-OpenAIEmbeddings|Embeddings","target":"pineconeExistingIndex_0","targetHandle":"pineconeExistingIndex_0-input-embeddings-Embeddings","type":"buttonedge","id":"openAIEmbeddings_0-openAIEmbeddings_0-output-openAIEmbeddings-OpenAIEmbeddings|Embeddings-pineconeExistingIndex_0-pineconeExistingIndex_0-input-embeddings-Embeddings","data":{"label":""}},{"source":"chatOpenAI_0","sourceHandle":"chatOpenAI_0-output-chatOpenAI-ChatOpenAI|BaseChatModel|BaseLanguageModel|BaseLangChain","target":"conversationalAgent_0","targetHandle":"conversationalAgent_0-input-model-BaseLanguageModel","type":"buttonedge","id":"chatOpenAI_0-chatOpenAI_0-output-chatOpenAI-ChatOpenAI|BaseChatModel|BaseLanguageModel|BaseLangChain-conversationalAgent_0-conversationalAgent_0-input-model-BaseLanguageModel","data":{"label":""}},{"source":"serpAPI_0","sourceHandle":"serpAPI_0-output-serpAPI-SerpAPI|Tool","target":"conversationalAgent_0","targetHandle":"conversationalAgent_0-input-tools-Tool","type":"buttonedge","id":"serpAPI_0-serpAPI_0-output-serpAPI-SerpAPI|Tool-conversationalAgent_0-conversationalAgent_0-input-tools-Tool","data":{"label":""}},{"source":"chainTool_0","sourceHandle":"chainTool_0-output-chainTool-ChainTool|Tool","target":"conversationalAgent_0","targetHandle":"conversationalAgent_0-input-tools-Tool","type":"buttonedge","id":"chainTool_0-chainTool_0-output-chainTool-ChainTool|Tool-conversationalAgent_0-conversationalAgent_0-input-tools-Tool","data":{"label":""}},{"source":"calculator_0","sourceHandle":"calculator_0-output-calculator-Calculator|Tool","target":"conversationalAgent_0","targetHandle":"conversationalAgent_0-input-tools-Tool","type":"buttonedge","id":"calculator_0-calculator_0-output-calculator-Calculator|Tool-conversationalAgent_0-conversationalAgent_0-input-tools-Tool","data":{"label":""}},{"source":"bufferMemory_0","sourceHandle":"bufferMemory_0-output-bufferMemory-BufferMemory|BaseChatMemory|BaseMemory","target":"conversationalAgent_0","targetHandle":"conversationalAgent_0-input-memory-BaseChatMemory","type":"buttonedge","id":"bufferMemory_0-bufferMemory_0-output-bufferMemory-BufferMemory|BaseChatMemory|BaseMemory-conversationalAgent_0-conversationalAgent_0-input-memory-BaseChatMemory","data":{"label":""}}]}

[therobhayes]

This is very helpful - appreciate the visual examples of how to combine the component parts of a chain here

[urbanscribe]

hey. first thanks a bunch. I now understand the chain tool. 2. the fact you can paste in a schema is really nicely done. thanks again!

[urbanscribe]

Probably a simple question but was hoping to replace the PINECONE LOAD EXISTING INDEX from vector store to retriever in order to narrow down to a namespace (extra field in retriever as opposed to vector store). that won't connect to vectorDB chain. how would I connect that namespace only pinecone retriever into the chain?

[HenryHengZJ]

you can try to use RetrievalQAChain for that, VectorDBQAChain is soon to be deprecated