Text files Comparion - Need Help

[mraguth]

Project Description: Text-Based Task Comparison with Creative Enhancement

I have two text files containing task-related content.

File 1 lists high-level tasks, providing brief descriptions.
File 2 contains more detailed content, covering descriptions not only for the tasks in File 1 but also for hundreds of other tasks.
Example:

File 1: “Langchain work”
File 2: “Build a Langchain application using OpenAI technology for enhanced functionality and performance”
Objective: I want to establish a relationship between the high-level tasks in File 1 and the relevant, detailed content from File 2. Specifically, the goal is to:

Match each high-level task in File 1 with its most relevant, detailed counterpart in File 2.
Generate an enriched and creative output using both files. For each matched task, the output should:
Draw content from File 2 as a primary reference.
Use an LLM to add depth, creativity, and a comprehensive explanation to the task description, going beyond what’s in File 2.
Requirements: The LLM, using Langchain, should systematically:

Pick each task from File 1.
Identify the closest matching or related content in File 2.
Produce a detailed and creatively enhanced description, using information from File 2 as a foundation and further enriched with LLM capabilities.
The code I’ve been using so far is below. Could you help refine or optimize it to better align with these objectives?

from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import FAISS
from langchain.document_loaders import TextLoader
from langchain.llms import OpenAI

# Load documents
loader1 = TextLoader("C:/Users/file1.txt")
docs1 = loader1.load()
loader2 = TextLoader("C:/Users/file2.txt")
docs2 = loader2.load()

# Create embeddings
embeddings = embeddings

# Create vector store
db1 = FAISS.from_documents(docs1, embeddings)
db2 = FAISS.from_documents(docs2, embeddings)

[dosubot[bot]]

To refine and optimize your code for matching tasks from File 1 with detailed content from File 2 and generating enriched outputs using LangChain, you can follow these steps:

1. Fix the Embeddings Initialization: Ensure that the embeddings are properly initialized. It seems like there's a placeholder embeddings = embeddings which should be replaced with an actual embeddings model.

2. Separate Vector Stores: You should create separate vector stores for each file to maintain their distinct content and then perform similarity searches.

3. Perform Similarity Search: Use the vector store to find the most relevant content in File 2 for each task in File 1.

4. Generate Enhanced Descriptions: Use the LLM to generate enriched descriptions based on the matched content.

Here's a refined version of your code:

from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import FAISS
from langchain.document_loaders import TextLoader
from langchain.llms import OpenAI

# Load documents
loader1 = TextLoader("C:/Users/file1.txt")
docs1 = loader1.load()
loader2 = TextLoader("C:/Users/file2.txt")
docs2 = loader2.load()

# Create embeddings
embeddings = HuggingFaceEmbeddings()

# Create vector stores
db1 = FAISS.from_documents(docs1, embeddings)
db2 = FAISS.from_documents(docs2, embeddings)

# Initialize the LLM
llm = OpenAI()

# Match and enhance tasks
for doc1 in docs1:
    # Find the most relevant document in File 2
    similar_docs = db2.similarity_search(doc1.content, k=1)
    if similar_docs:
        detailed_content = similar_docs[0].content
        # Generate enriched description
        enriched_description = llm.generate(
            prompt=f"Enhance the following task description: {detailed_content}",
            max_tokens=150
        )
        print(f"Task: {doc1.content}\nEnhanced Description: {enriched_description}\n")

Key Changes:

• Embeddings Initialization: Properly initialize the embeddings using HuggingFaceEmbeddings().
• Separate Vector Stores: Create separate vector stores for docs1 and docs2.
• Similarity Search: Use similarity_search to find the closest match in docs2 for each task in docs1.
• LLM Generation: Use the LLM to generate an enhanced description based on the matched content.

This should help you achieve the objective of matching and enhancing task descriptions using LangChain.

To continue talking to Dosu, mention @dosu.

[mraguth]

both are .txt files (free text) and loading via 'textloader' and loading into FAISS document store. Do I need to split and load? Code is below

# Load documents
loader1 = TextLoader("C:/Users/file1.txt")
docs1 = loader1.load()
loader2 = TextLoader("C:/Users/file2.txt")
docs2 = loader2.load()

# Create embeddings
embeddings = embeddings

# Create vector store
db1 = FAISS.from_documents(docs1, embeddings)
db2 = FAISS.from_documents(docs2, embeddings)

[mraguth]

@dosu... this code is working chaged to page_content

# Match and enhance tasks
for doc1 in docs1:
    # Find the most relevant document in File 2
    similar_docs = db2.similarity_search(doc1.page_content, k=1)
    if similar_docs:
        detailed_content = similar_docs[0].page_content

        # Summarize detailed_content if it's too long
        if len(detailed_content.split()) > 1000:  # Adjust word limit as needed
            detailed_content_summary = llm(prompt=f"Summarize the following text: {detailed_content[:2000]}", max_tokens=150)
        else:
            detailed_content_summary = detailed_content

        # Generate enriched description
        enriched_description = llm(prompt=f"Enhance the following task description: {detailed_content_summary}", max_tokens=150)
        print(f"Task: {doc1.page_content}\nEnhanced Description: {enriched_description}\n")

[dosubot[bot]]

Hey @mraguth! It looks like you're diving deep into some code magic. However, I'm just a bot designed to help with bugs, technical questions, and engineering knowledge. If you think I should have tackled this request, feel free to reach out to the Dosu Team. They'll be happy to hear your thoughts!

To continue talking to Dosu, mention @dosu.

[mraguth]

@dosu the code is working fine. However, Enhanced Description is not picking any relavent content from file2.... can you help?

[dosubot[bot]]

To ensure that the enhanced description is relevant and effectively uses content from File 2, you can try the following steps:

1. Verify Similarity Search: Make sure the similarity search is correctly identifying the most relevant content from File 2. You can print out the detailed_content before generating the enhanced description to verify that it matches your expectations.

2. Adjust Embeddings and Model Parameters: Ensure that the embeddings are properly initialized and suitable for your text data. You might need to experiment with different embeddings models or parameters to improve the relevance of the matches.

3. Refine the LLM Prompt: The prompt used for the LLM can significantly affect the output. Make sure the prompt clearly instructs the LLM to incorporate specific details from the matched content. You might try something like:

   enriched_description = llm.generate(
       prompt=f"Using the following detailed content as a reference, enhance the task description: {detailed_content_summary}. Make sure to include key details and insights.",
       max_tokens=150
   )

4. Check Token Limits: Ensure that the max_tokens parameter is set appropriately to allow the LLM to generate a sufficiently detailed response.

5. Debugging Output: Add print statements to debug and understand what content is being used at each step. For example:

   print(f"Task: {doc1.page_content}")
   print(f"Matched Content: {detailed_content_summary}")
   print(f"Enhanced Description: {enriched_description}\n")

By following these steps, you should be able to improve the relevance of the enhanced descriptions generated by the LLM.

To continue talking to Dosu, mention @dosu.