From b71dcde8780fb49dd60a970ac64eec920b8231ce Mon Sep 17 00:00:00 2001 From: ivanmilevtues Date: Sun, 8 Jun 2025 20:42:50 +0200 Subject: [PATCH] Added high-level diagrams --- .codeboarding/Data Management.md | 48 +++++++++++++++ .../Evaluation and Performance Monitoring.md | 36 +++++++++++ .codeboarding/Knowledge Integration.md | 54 +++++++++++++++++ .codeboarding/LM Interaction Management.md | 57 ++++++++++++++++++ .../Module Composition and Execution.md | 37 ++++++++++++ .codeboarding/Prediction Strategies.md | 58 ++++++++++++++++++ .../Prompt Optimization and Teleprompting.md | 54 +++++++++++++++++ .codeboarding/on_boarding.md | 59 +++++++++++++++++++ 8 files changed, 403 insertions(+) create mode 100644 .codeboarding/Data Management.md create mode 100644 .codeboarding/Evaluation and Performance Monitoring.md create mode 100644 .codeboarding/Knowledge Integration.md create mode 100644 .codeboarding/LM Interaction Management.md create mode 100644 .codeboarding/Module Composition and Execution.md create mode 100644 .codeboarding/Prediction Strategies.md create mode 100644 .codeboarding/Prompt Optimization and Teleprompting.md create mode 100644 .codeboarding/on_boarding.md diff --git a/.codeboarding/Data Management.md b/.codeboarding/Data Management.md new file mode 100644 index 0000000000..125d322598 --- /dev/null +++ b/.codeboarding/Data Management.md @@ -0,0 +1,48 @@ +```mermaid +graph LR + Dataset["Dataset"] + DataLoader["DataLoader"] + MATH["MATH"] + GSM8K["GSM8K"] + Colors["Colors"] + HotPotQA["HotPotQA"] + AlfWorld["AlfWorld"] + MATH -- "inherits from" --> Dataset + GSM8K -- "inherits from" --> Dataset + Colors -- "inherits from" --> Dataset + HotPotQA -- "inherits from" --> Dataset + AlfWorld -- "inherits from" --> Dataset + DataLoader -- "loads data into" --> Dataset +``` + +## Component Details + +The Data Management component in DSPy provides a set of tools for loading, processing, and managing datasets used for training and evaluating DSPy programs. It offers a unified interface for accessing data from various sources, including local files and Hugging Face datasets. The core functionality revolves around the `Dataset` class, which serves as the base class for all datasets, and the `DataLoader` class, which handles the loading and conversion of data into `Dataset` objects. Specific dataset classes like `MATH`, `GSM8K`, `Colors`, `HotPotQA`, and `AlfWorld` provide standardized access to commonly used datasets, each tailored to its specific data format and evaluation metrics. This component streamlines the data preparation process, making it easier for users to work with different datasets in a consistent manner. + +### Dataset +The `Dataset` class serves as the foundation for all datasets within DSPy. It provides core functionalities for data manipulation, including splitting the dataset into training, validation, and testing subsets, shuffling the data to ensure randomness, and preparing the data for consumption by DSPy modules. All specific dataset implementations inherit from this base class, ensuring a consistent interface for data access and manipulation. +- **Related Classes/Methods**: `dspy.datasets.dataset.Dataset` + +### DataLoader +The `DataLoader` class is responsible for loading datasets from various sources, such as CSV files, JSON files, Pandas DataFrames, Hugging Face datasets, Parquet files, and RM datasets. It transforms the loaded data into a `Dataset` object, making it compatible with DSPy's data processing pipeline. The `DataLoader` abstracts away the complexities of data loading and formatting, providing a simple and unified interface for accessing data from different sources. +- **Related Classes/Methods**: `dspy.datasets.dataloader.DataLoader` + +### MATH +The `MATH` dataset class provides a standardized way to access and utilize the MATH dataset within DSPy. It inherits from the base `Dataset` class and offers specific functionalities or configurations tailored to the MATH dataset's structure and requirements. This ensures that the MATH dataset can be seamlessly integrated into DSPy programs. +- **Related Classes/Methods**: `dspy.datasets.math.MATH` + +### GSM8K +The `GSM8K` dataset class provides a standardized way to access and utilize the GSM8K dataset within DSPy. It inherits from the base `Dataset` class and includes a metric function (`gsm8k_metric`) for evaluating performance on this dataset. This ensures that the GSM8K dataset can be seamlessly integrated into DSPy programs and evaluated effectively. +- **Related Classes/Methods**: `dspy.datasets.gsm8k.GSM8K`, `dspy.datasets.gsm8k.gsm8k_metric` + +### Colors +The `Colors` dataset class provides a standardized way to access and utilize the Colors dataset within DSPy. It inherits from the base `Dataset` class and offers specific functionalities or configurations tailored to the Colors dataset's structure and requirements. This ensures that the Colors dataset can be seamlessly integrated into DSPy programs. +- **Related Classes/Methods**: `dspy.datasets.colors.Colors` + +### HotPotQA +The `HotPotQA` dataset class provides a standardized way to access and utilize the HotPotQA dataset within DSPy. It inherits from the base `Dataset` class and offers specific functionalities or configurations tailored to the HotPotQA dataset's structure and requirements. This ensures that the HotPotQA dataset can be seamlessly integrated into DSPy programs. +- **Related Classes/Methods**: `dspy.datasets.hotpotqa.HotPotQA` + +### AlfWorld +The `AlfWorld` dataset class provides a standardized way to access and utilize the AlfWorld dataset within DSPy. It inherits from the base `Dataset` class and includes environment interaction functionalities, allowing DSPy programs to interact with the AlfWorld environment. This enables the development of DSPy programs that can reason and act within the AlfWorld environment. +- **Related Classes/Methods**: `dspy.datasets.alfworld.alfworld.AlfWorld`, `dspy.datasets.alfworld.alfworld.EnvPool` diff --git a/.codeboarding/Evaluation and Performance Monitoring.md b/.codeboarding/Evaluation and Performance Monitoring.md new file mode 100644 index 0000000000..e753283be3 --- /dev/null +++ b/.codeboarding/Evaluation and Performance Monitoring.md @@ -0,0 +1,36 @@ +```mermaid +graph LR + Evaluate["Evaluate"] + Metric_Functions_answer_exact_match_answer_passage_match_["Metric Functions (answer_exact_match, answer_passage_match)"] + SemanticF1["SemanticF1"] + CompleteAndGrounded["CompleteAndGrounded"] + display_dataframe["display_dataframe"] + Evaluate -- "orchestrates evaluation using" --> Metric_Functions_answer_exact_match_answer_passage_match_ + Evaluate -- "orchestrates evaluation using" --> SemanticF1 + Evaluate -- "orchestrates evaluation using" --> CompleteAndGrounded + Evaluate -- "presents results using" --> display_dataframe +``` + +## Component Details + +The Evaluation and Performance Monitoring component in DSPy provides a comprehensive suite of tools for assessing the quality and effectiveness of DSPy programs. It encompasses functionalities for defining custom metrics, executing evaluations on datasets, and leveraging automated LM-based evaluation methods. The primary goal is to ensure program reliability and offer actionable insights for continuous improvement and optimization. + +### Evaluate +The Evaluate class serves as the central orchestrator for the evaluation process. It takes a DSPy predictor (module), a dataset, and a set of evaluation metrics as input. It then iterates through the dataset, applies the predictor to each data point, calculates the specified metrics by comparing the predictor's output to the ground truth, and presents the results in a structured table format. It relies on various metric components and the display_dataframe function to present the results. +- **Related Classes/Methods**: `dspy.evaluate.evaluate.Evaluate:__call__`, `dspy.evaluate.evaluate.Evaluate:_construct_result_table`, `dspy.evaluate.evaluate.Evaluate:_display_result_table` + +### Metric Functions (answer_exact_match, answer_passage_match) +These functions define fundamental metrics for evaluating answer quality. `answer_exact_match` checks if the predicted answer exactly matches the ground truth, while `answer_passage_match` determines if the predicted answer is present within a retrieved context passage. These metrics provide a basic level of assessment for answer correctness and relevance and are used by the Evaluate class. +- **Related Classes/Methods**: `dspy.evaluate.metrics:answer_exact_match`, `dspy.evaluate.metrics:answer_passage_match`, `dspy.evaluate.metrics:_answer_match`, `dspy.evaluate.metrics:_passage_match` + +### SemanticF1 +The SemanticF1 class implements an automated evaluation metric that leverages semantic similarity to assess answer quality. It compares the predicted answer with the ground truth using a semantic similarity model, calculating an F1 score to quantify the degree of semantic overlap. This metric offers a more nuanced evaluation compared to exact match, capturing semantic equivalence and is used by the Evaluate class. +- **Related Classes/Methods**: `dspy.evaluate.auto_evaluation.SemanticF1:__init__`, `dspy.evaluate.auto_evaluation.SemanticF1:forward` + +### CompleteAndGrounded +The CompleteAndGrounded class implements an automated evaluation metric that assesses the completeness and groundedness of generated answers. It checks whether the answer covers all relevant aspects of the question and whether it is adequately supported by the retrieved context. This metric is particularly useful for evaluating the quality of answers generated by retrieval-augmented language models and is used by the Evaluate class. +- **Related Classes/Methods**: `dspy.evaluate.auto_evaluation.CompleteAndGrounded:__init__`, `dspy.evaluate.auto_evaluation.CompleteAndGrounded:forward` + +### display_dataframe +The `display_dataframe` function is responsible for presenting the evaluation results in a user-friendly, tabular format using a Pandas DataFrame. It takes the calculated metrics and corresponding data points as input and generates a visually appealing table that facilitates easy interpretation and analysis of the evaluation results. It is used by the Evaluate class to present the results. +- **Related Classes/Methods**: `dspy.evaluate.evaluate:display_dataframe` diff --git a/.codeboarding/Knowledge Integration.md b/.codeboarding/Knowledge Integration.md new file mode 100644 index 0000000000..b5910e7dd8 --- /dev/null +++ b/.codeboarding/Knowledge Integration.md @@ -0,0 +1,54 @@ +```mermaid +graph LR + Retrieve["Retrieve"] + AzureAISearchRM["AzureAISearchRM"] + MarqoRM["MarqoRM"] + MilvusRM["MilvusRM"] + MongoDBAtlasRM["MongoDBAtlasRM"] + Neo4jRM["Neo4jRM"] + ClarifaiRM["ClarifaiRM"] + PgVectorRM["PgVectorRM"] + Retrieve -- "uses" --> AzureAISearchRM + Retrieve -- "uses" --> MarqoRM + Retrieve -- "uses" --> MilvusRM + Retrieve -- "uses" --> MongoDBAtlasRM + Retrieve -- "uses" --> Neo4jRM + Retrieve -- "uses" --> ClarifaiRM + Retrieve -- "uses" --> PgVectorRM +``` + +## Component Details + +The Knowledge Integration component in DSPy focuses on retrieving relevant information from external sources to enhance program capabilities. The core of this component is the `Retrieve` class, which acts as an orchestrator, taking a query and using various Retriever Modules (RMs) to fetch relevant passages. Each RM is responsible for interacting with a specific external knowledge source, such as vector databases or search engines. The retrieved passages are then returned to the program for further processing, enabling the integration of external knowledge into the program's reasoning and decision-making processes. + +### Retrieve +The `Retrieve` class is an abstract component responsible for managing the retrieval process. It takes a query as input and uses one or more Retriever modules to fetch relevant passages. It orchestrates the retrieval process and returns the retrieved passages. +- **Related Classes/Methods**: `dspy.retrieve.retrieve.Retrieve`, `dspy.retrieve.retrieve.Retrieve:__call__`, `dspy.retrieve.retrieve.Retrieve:forward` + +### AzureAISearchRM +The `AzureAISearchRM` class is a Retriever module that interacts with Azure AI Search to retrieve relevant passages. It takes a query as input, constructs a search request, and returns the retrieved passages from Azure AI Search. +- **Related Classes/Methods**: `dspy.retrieve.azureaisearch_rm.AzureAISearchRM`, `dspy.retrieve.azureaisearch_rm.AzureAISearchRM:__init__`, `dspy.retrieve.azureaisearch_rm.AzureAISearchRM:azure_search_request`, `dspy.retrieve.azureaisearch_rm.AzureAISearchRM:forward` + +### MarqoRM +The `MarqoRM` class is a Retriever module that interacts with Marqo to retrieve relevant passages. It takes a query as input and returns the retrieved passages from Marqo. +- **Related Classes/Methods**: `dspy.retrieve.marqo_rm.MarqoRM`, `dspy.retrieve.marqo_rm.MarqoRM:__init__`, `dspy.retrieve.marqo_rm.MarqoRM:forward` + +### MilvusRM +The `MilvusRM` class is a Retriever module that interacts with Milvus to retrieve relevant passages. It takes a query as input and returns the retrieved passages from Milvus. +- **Related Classes/Methods**: `dspy.retrieve.milvus_rm.MilvusRM`, `dspy.retrieve.milvus_rm.MilvusRM:forward` + +### MongoDBAtlasRM +The `MongoDBAtlasRM` class is a Retriever module that interacts with MongoDB Atlas Vector Search to retrieve relevant passages. It takes a query as input and returns the retrieved passages from MongoDB Atlas. +- **Related Classes/Methods**: `dspy.retrieve.mongodb_atlas_rm.MongoDBAtlasRM`, `dspy.retrieve.mongodb_atlas_rm.MongoDBAtlasRM:__init__`, `dspy.retrieve.mongodb_atlas_rm.MongoDBAtlasRM:forward` + +### Neo4jRM +The `Neo4jRM` class is a Retriever module that interacts with Neo4j to retrieve relevant passages. It takes a query as input and returns the retrieved passages from Neo4j. +- **Related Classes/Methods**: `dspy.retrieve.neo4j_rm.Neo4jRM`, `dspy.retrieve.neo4j_rm.Neo4jRM:__init__`, `dspy.retrieve.neo4j_rm.Neo4jRM:forward` + +### ClarifaiRM +The `ClarifaiRM` class is a Retriever module that interacts with Clarifai to retrieve relevant passages. It takes a query as input and returns the retrieved passages from Clarifai. +- **Related Classes/Methods**: `dspy.retrieve.clarifai_rm.ClarifaiRM`, `dspy.retrieve.clarifai_rm.ClarifaiRM:__init__`, `dspy.retrieve.clarifai_rm.ClarifaiRM:forward` + +### PgVectorRM +The `PgVectorRM` class is a Retriever module that interacts with PgVector to retrieve relevant passages. It takes a query as input and returns the retrieved passages from PgVector. +- **Related Classes/Methods**: `dspy.retrieve.pgvector_rm.PgVectorRM`, `dspy.retrieve.pgvector_rm.PgVectorRM:__init__`, `dspy.retrieve.pgvector_rm.PgVectorRM:forward` diff --git a/.codeboarding/LM Interaction Management.md b/.codeboarding/LM Interaction Management.md new file mode 100644 index 0000000000..a283dbd3d5 --- /dev/null +++ b/.codeboarding/LM Interaction Management.md @@ -0,0 +1,57 @@ +```mermaid +graph LR + Signature["Signature"] + Adapter["Adapter"] + BaseLM["BaseLM"] + LM["LM"] + Cache["Cache"] + OpenAIProvider["OpenAIProvider"] + ChatAdapter["ChatAdapter"] + JSONAdapter["JSONAdapter"] + Signature -- "defines structure for" --> LM_calls + Adapter -- "uses" --> Signature + LM -- "inherits from" --> BaseLM + LM -- "uses" --> Cache + LM -- "uses" --> OpenAIProvider + ChatAdapter -- "inherits from" --> Adapter + JSONAdapter -- "inherits from" --> Adapter + ChatAdapter -- "formats prompts and parses responses using" --> Signature + JSONAdapter -- "formats prompts and parses responses using" --> Signature + Adapter -- "calls" --> LM +``` + +## Component Details + +The LM Interaction Management component in DSPy orchestrates communication with various Language Models (LMs). It defines signatures for LM calls using `Signature` and its related classes (`InputField`, `OutputField`), handles API requests through `BaseLM` and its subclasses (`LM`, `OpenAIProvider`, `ArborProvider`, `LocalProvider`), caches responses using `Cache`, and adapts input/output formats using `Adapter` and its subclasses (`ChatAdapter`, `JSONAdapter`, `TwoStepAdapter`). This system abstracts away the complexities of different LM providers, ensuring consistent and reliable LM behavior across diverse platforms. + +### Signature +Defines the structure of LM calls, specifying input and output fields. It acts as a contract between DSPy modules and LMs, ensuring that data is formatted correctly for both input and output. +- **Related Classes/Methods**: `dspy.signatures.signature`, `dspy.signatures.field` + +### Adapter +Abstract base class responsible for adapting between DSPy Signatures and LM APIs. It handles formatting prompts, parsing responses, and managing conversation history, bridging the gap between DSPy's structured data and the LM's API. +- **Related Classes/Methods**: `dspy.adapters.base` + +### BaseLM +Abstract base class for language model clients. It provides a unified interface for calling different LMs, abstracting away the specific API details of each LM provider. +- **Related Classes/Methods**: `dspy.clients.base_lm` + +### LM +Base class for language model clients, inheriting from BaseLM. It handles calling LMs, caching responses, and fine-tuning, providing a concrete implementation for interacting with LMs. +- **Related Classes/Methods**: `dspy.clients.lm` + +### Cache +Caches LM responses to improve efficiency and reduce API costs. It stores and retrieves LM responses based on the input prompt, avoiding redundant API calls. +- **Related Classes/Methods**: `dspy.clients.cache` + +### OpenAIProvider +Client for interacting with the OpenAI API. It implements the BaseLM interface and provides specific methods for calling OpenAI's LMs. +- **Related Classes/Methods**: `dspy.clients.openai` + +### ChatAdapter +Adapter specialized for chat-based LM APIs. It formats prompts as chat messages and parses responses accordingly, tailoring the interaction for chat-style LMs. +- **Related Classes/Methods**: `dspy.adapters.chat_adapter` + +### JSONAdapter +Adapter that formats prompts and parses responses as JSON objects. It's useful for LMs that work well with structured JSON data. +- **Related Classes/Methods**: `dspy.adapters.json_adapter` diff --git a/.codeboarding/Module Composition and Execution.md b/.codeboarding/Module Composition and Execution.md new file mode 100644 index 0000000000..91f9257da2 --- /dev/null +++ b/.codeboarding/Module Composition and Execution.md @@ -0,0 +1,37 @@ +```mermaid +graph LR + BaseModule["BaseModule"] + Module["Module"] + ModuleGraph["ModuleGraph"] + Parameter["Parameter"] + Predictor["Predictor"] + BaseModule -- "is base class of" --> Module + ModuleGraph -- "manages" --> Module + BaseModule -- "is base class of" --> ModuleGraph + BaseModule -- "manages" --> Parameter + Predictor -- "is a type of" --> Module +``` + +## Component Details + +This graph represents the module composition and execution functionality in DSPy. It allows users to define reusable components (Modules) that encapsulate LM calls and other operations. These modules can be composed into complex data processing pipelines, managed by the ModuleGraph. The BaseModule provides the foundation for all modules, handling parameters and state. The flow involves defining Modules, composing them within a ModuleGraph, and then executing the graph to perform the desired data processing tasks. + +### BaseModule +The BaseModule class serves as the foundation for all modules within DSPy. It offers core functionalities for managing parameters, submodules, and saving/loading module state. It provides a consistent interface for all module types, enabling modular and reusable component design. +- **Related Classes/Methods**: `dspy.primitives.module.BaseModule` + +### Module +The Module class, a subclass of BaseModule, represents a program or a collection of predictors. It provides methods for managing predictors, setting/getting the language model, and defining the forward pass. It is the primary way to define a reusable program in DSPy, encapsulating the logic and parameters for a specific task. +- **Related Classes/Methods**: `dspy.primitives.module.Module` + +### ModuleGraph +The ModuleGraph class is designed for managing and organizing modules in a graph structure. It allows adding modules, processing submodules, and defining the connections between them. This enables the creation of complex, modular programs by connecting individual modules and managing the data flow between them. +- **Related Classes/Methods**: `dspy.experimental.module_graph.ModuleGraph` + +### Parameter +The Parameter class represents a configurable parameter within a module. It allows users to define and manage the parameters that control the behavior of a module. Parameters can be set, retrieved, and updated, enabling dynamic configuration of modules. +- **Related Classes/Methods**: `dspy.primitives.module.Parameter` + +### Predictor +The Predictor class is a specialized type of Module that is designed to make predictions. It typically encapsulates a language model call and provides methods for formatting the input and output. Predictors are used as building blocks for more complex programs. +- **Related Classes/Methods**: `dspy.primitives.module.Predictor` diff --git a/.codeboarding/Prediction Strategies.md b/.codeboarding/Prediction Strategies.md new file mode 100644 index 0000000000..571056a9fa --- /dev/null +++ b/.codeboarding/Prediction Strategies.md @@ -0,0 +1,58 @@ +```mermaid +graph LR + Predict["Predict"] + ChainOfThought["ChainOfThought"] + ReAct["ReAct"] + Predict___call__["Predict.__call__"] + Predict_forward["Predict.forward"] + Predict__forward_preprocess["Predict._forward_preprocess"] + Predict__forward_postprocess["Predict._forward_postprocess"] + ReAct__format_trajectory["ReAct._format_trajectory"] + ReAct_forward["ReAct.forward"] + ChainOfThought -- "is a subclass of" --> Predict + ReAct -- "is a subclass of" --> Predict + Predict___call__ -- "calls" --> Predict__forward_preprocess + Predict___call__ -- "calls" --> Predict_forward + Predict___call__ -- "calls" --> Predict__forward_postprocess + ReAct_forward -- "calls" --> ReAct__format_trajectory +``` + +## Component Details + +The Prediction Strategies component in DSPy provides a flexible framework for generating predictions using language models. It revolves around the `Predict` class, which serves as a base for different prediction strategies. Subclasses like `ChainOfThought` and `ReAct` implement specific prompting and reasoning techniques. The core prediction logic is encapsulated in the `forward` method, which is customized by subclasses. The `__call__` method orchestrates the prediction process, including preprocessing and postprocessing. This system allows DSPy to easily incorporate and experiment with various LM-based prediction approaches. + +### Predict +The `Predict` class is the foundation for all prediction strategies in DSPy. It defines the basic structure for making predictions, including methods for preprocessing, the forward pass (prediction logic), and postprocessing. It also handles state management, allowing predictors to be saved and loaded. Subclasses override the `forward` method to implement specific prediction strategies. +- **Related Classes/Methods**: `dspy.predict.predict.Predict` + +### ChainOfThought +The `ChainOfThought` class extends `Predict` to implement chain-of-thought reasoning. It generates intermediate reasoning steps to improve prediction accuracy. This class provides a structured way to guide the language model through a series of logical steps before arriving at the final answer. +- **Related Classes/Methods**: `dspy.predict.chain_of_thought.ChainOfThought` + +### ReAct +The `ReAct` class extends `Predict` to implement the ReAct (Reasoning and Acting) paradigm. It enables the language model to interact with an environment by alternating between reasoning and action steps. This allows the model to gather information and refine its predictions based on real-world feedback. +- **Related Classes/Methods**: `dspy.predict.react.ReAct` + +### Predict.__call__ +The `__call__` method is the main entry point for making predictions using a `Predict` object. It orchestrates the entire prediction process, calling the `_forward_preprocess`, `forward`, and `_forward_postprocess` methods in sequence. This method ensures that the input is properly prepared, the prediction logic is executed, and the output is properly formatted. +- **Related Classes/Methods**: `dspy.predict.predict.Predict:__call__` + +### Predict.forward +The `forward` method contains the core prediction logic. This method is designed to be overridden by subclasses to implement specific prediction strategies. It takes the preprocessed input and generates the prediction output. +- **Related Classes/Methods**: `dspy.predict.predict.Predict:forward` + +### Predict._forward_preprocess +The `_forward_preprocess` method preprocesses the input before the forward pass. This may involve formatting the input or applying other transformations to prepare it for the prediction logic. +- **Related Classes/Methods**: `dspy.predict.predict.Predict:_forward_preprocess` + +### Predict._forward_postprocess +The `_forward_postprocess` method postprocesses the output after the forward pass. This may involve parsing the output or applying other transformations to format it for the user. +- **Related Classes/Methods**: `dspy.predict.predict.Predict:_forward_postprocess` + +### ReAct._format_trajectory +The `_format_trajectory` method in `ReAct` formats the trajectory of ReAct steps into a string representation. This is useful for logging and debugging the ReAct process. +- **Related Classes/Methods**: `dspy.predict.react.ReAct:_format_trajectory` + +### ReAct.forward +The `forward` method in `ReAct` implements the forward pass for the ReAct predictor. It manages the interaction loop between reasoning and acting, calling the language model to generate both reasoning and action steps. +- **Related Classes/Methods**: `dspy.predict.react.ReAct:forward` diff --git a/.codeboarding/Prompt Optimization and Teleprompting.md b/.codeboarding/Prompt Optimization and Teleprompting.md new file mode 100644 index 0000000000..e50b5fba93 --- /dev/null +++ b/.codeboarding/Prompt Optimization and Teleprompting.md @@ -0,0 +1,54 @@ +```mermaid +graph LR + BootstrapFewShot["BootstrapFewShot"] + MIPROv2["MIPROv2"] + COPRO["COPRO"] + AvatarOptimizer["AvatarOptimizer"] + GRPO["GRPO"] + BetterTogether["BetterTogether"] + SignatureOptimizer["SignatureOptimizer"] + BootstrapFinetune["BootstrapFinetune"] + BootstrapFewShot -- "uses" --> create_n_fewshot_demo_sets + MIPROv2 -- "uses" --> create_n_fewshot_demo_sets + AvatarOptimizer -- "uses" --> eval_candidate_program + BetterTogether -- "uses" --> AvatarOptimizer + BetterTogether -- "uses" --> GRPO + BootstrapFinetune -- "uses" --> bootstrap_trace_data + BootstrapFinetune -- "uses" --> prepare_teacher +``` + +## Component Details + +The Prompt Optimization and Teleprompting component in DSPy focuses on automating the process of prompt engineering for language models. It encompasses various strategies to discover effective prompts and demonstrations, thereby improving the performance of programs that utilize LMs. The core idea is to optimize the input to the LM, which includes not only the prompt itself but also few-shot examples, to achieve better accuracy and robustness. This involves techniques like generating positive and negative examples, using gradient-based optimization, and employing meta-instruction prompt optimization. + +### BootstrapFewShot +This component is a teleprompter that uses a bootstrap method to generate few-shot examples for prompting. It automatically creates demonstrations to improve the performance of language models on specific tasks. It leverages the `create_n_fewshot_demo_sets` utility to generate these demonstrations. +- **Related Classes/Methods**: `dspy.teleprompt.bootstrap.BootstrapFewShot`, `dspy.teleprompt.utils.create_n_fewshot_demo_sets` + +### MIPROv2 +MIPROv2 is a teleprompter that employs a meta-instruction prompt optimization approach to discover optimal prompts. It also uses `create_n_fewshot_demo_sets` to generate demonstrations, similar to BootstrapFewShot, but focuses on optimizing the meta-instructions that guide the LM. +- **Related Classes/Methods**: `dspy.teleprompt.mipro_optimizer_v2.MIPROv2`, `dspy.teleprompt.utils.create_n_fewshot_demo_sets` + +### COPRO +COPRO is a teleprompter that uses a contrastive prompt optimization approach to find optimal prompts. It contrasts different prompts to identify the most effective ones for a given task. It optimizes prompts by contrasting positive and negative examples. +- **Related Classes/Methods**: `dspy.teleprompt.copro_optimizer.COPRO` + +### AvatarOptimizer +AvatarOptimizer optimizes prompts by generating positive and negative examples and using them to refine the prompt. It leverages an evaluation function to assess the quality of generated prompts and examples. +- **Related Classes/Methods**: `dspy.teleprompt.avatar_optimizer.AvatarOptimizer`, `dspy.teleprompt.utils.eval_candidate_program` + +### GRPO +GRPO is a gradient-based prompt optimization technique that uses validation metrics to guide the prompt refinement process. It refines prompts based on gradient descent, using validation metrics to guide the optimization. +- **Related Classes/Methods**: `dspy.teleprompt.grpo.GRPO` + +### BetterTogether +BetterTogether combines multiple prompt optimization strategies to achieve better performance than any single strategy alone. It integrates AvatarOptimizer and GRPO to leverage their complementary strengths. +- **Related Classes/Methods**: `dspy.teleprompt.bettertogether.BetterTogether` + +### SignatureOptimizer +SignatureOptimizer optimizes the signature of a program to improve its performance. It focuses on refining the input and output specifications of a program to enhance its interaction with language models. +- **Related Classes/Methods**: `dspy.teleprompt.signature_opt.SignatureOptimizer` + +### BootstrapFinetune +BootstrapFinetune is a teleprompter that uses a bootstrap method to generate training data and then finetunes a language model on that data. It leverages `bootstrap_trace_data` to generate training data and `prepare_teacher` to prepare the teacher model for finetuning. +- **Related Classes/Methods**: `dspy.teleprompt.bootstrap_finetune.BootstrapFinetune` diff --git a/.codeboarding/on_boarding.md b/.codeboarding/on_boarding.md new file mode 100644 index 0000000000..5cb5e023cb --- /dev/null +++ b/.codeboarding/on_boarding.md @@ -0,0 +1,59 @@ +```mermaid +graph LR + LM_Interaction_Management["LM Interaction Management"] + Module_Composition_and_Execution["Module Composition and Execution"] + Prompt_Optimization_and_Teleprompting["Prompt Optimization and Teleprompting"] + Knowledge_Integration["Knowledge Integration"] + Evaluation_and_Performance_Monitoring["Evaluation and Performance Monitoring"] + Data_Management["Data Management"] + Prediction_Strategies["Prediction Strategies"] + LM_Interaction_Management -- "defines interface for" --> Prediction_Strategies + Prediction_Strategies -- "uses" --> LM_Interaction_Management + Module_Composition_and_Execution -- "encapsulates" --> Prediction_Strategies + Module_Composition_and_Execution -- "uses" --> LM_Interaction_Management + Prompt_Optimization_and_Teleprompting -- "integrates" --> Module_Composition_and_Execution + Prompt_Optimization_and_Teleprompting -- "optimizes" --> LM_Interaction_Management + Evaluation_and_Performance_Monitoring -- "evaluates" --> Module_Composition_and_Execution + Knowledge_Integration -- "retrieves" --> Module_Composition_and_Execution + Data_Management -- "uses" --> Prompt_Optimization_and_Teleprompting + Data_Management -- "uses" --> Evaluation_and_Performance_Monitoring + click LM_Interaction_Management href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/dspy/LM Interaction Management.md" "Details" + click Module_Composition_and_Execution href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/dspy/Module Composition and Execution.md" "Details" + click Prompt_Optimization_and_Teleprompting href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/dspy/Prompt Optimization and Teleprompting.md" "Details" + click Knowledge_Integration href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/dspy/Knowledge Integration.md" "Details" + click Evaluation_and_Performance_Monitoring href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/dspy/Evaluation and Performance Monitoring.md" "Details" + click Data_Management href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/dspy/Data Management.md" "Details" + click Prediction_Strategies href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/dspy/Prediction Strategies.md" "Details" +``` + +## Component Details + +DSPy is a framework for programming with language models (LMs) to solve advanced tasks. It provides a high-level interface for composing LMs, managing prompts, and optimizing performance. The core flow involves defining signatures for LM calls, using modules to encapsulate LM interactions, and employing teleprompters to automatically optimize prompts and few-shot examples. DSPy also includes tools for retrieval, evaluation, and data handling, enabling developers to build and deploy robust LM-powered applications. + +### LM Interaction Management +This component manages all interactions with Language Models. It includes defining signatures for LM calls, handling API requests, caching responses, and adapting input/output formats. It abstracts away the complexities of different LM providers and ensures consistent and reliable LM behavior. +- **Related Classes/Methods**: `dspy.signatures.signature`, `dspy.signatures.field`, `dspy.clients.lm`, `dspy.clients.base_lm`, `dspy.clients.openai`, `dspy.clients.cache`, `dspy.adapters.base`, `dspy.adapters.chat_adapter`, `dspy.adapters.json_adapter` + +### Module Composition and Execution +This component provides the base class `BaseModule` and `Module` for creating reusable components that encapsulate LM calls and other operations. Modules can be composed to build complex data processing pipelines. They manage parameters, state, and LM configurations, enabling modular and scalable program design. +- **Related Classes/Methods**: `dspy.primitives.module`, `dspy.primitives.program` + +### Prompt Optimization and Teleprompting +This component includes classes for optimizing prompts and few-shot examples for LMs. It offers various teleprompting strategies like `BootstrapFewShot`, `MIPRO`, and `COPRO`, which automatically discover effective prompts and demonstrations. It improves program performance by optimizing LM inputs. +- **Related Classes/Methods**: `dspy.teleprompt.bootstrap`, `dspy.teleprompt.mipro_optimizer_v2`, `dspy.teleprompt.copro_optimizer` + +### Knowledge Integration +This component provides modules for retrieving relevant information from external sources. It includes a base `Retrieve` class and implementations for different retrieval methods, such as vector databases and search engines. It enhances program capabilities by integrating external knowledge. +- **Related Classes/Methods**: `dspy.retrieve.retrieve`, `dspy.retrieve.chromadb_rm`, `dspy.retrieve.pinecone_rm` + +### Evaluation and Performance Monitoring +This component provides tools for evaluating the performance of DSPy programs. It includes classes for defining metrics and running evaluations on datasets. It also offers automatic evaluation methods using LMs. It ensures program quality and provides insights for improvement. +- **Related Classes/Methods**: `dspy.evaluate.evaluate`, `dspy.evaluate.metrics`, `dspy.evaluate.auto_evaluation` + +### Data Management +This component provides tools for loading and managing datasets. It includes classes for loading data from various formats, such as CSV, JSON, and Hugging Face datasets. It also offers utilities for splitting and sampling datasets. It supports the data needs for training and evaluating DSPy programs. +- **Related Classes/Methods**: `dspy.datasets.dataset`, `dspy.datasets.dataloader` + +### Prediction Strategies +This component includes modules that use LMs to make predictions based on defined signatures. It offers various prediction strategies like `Predict`, `ChainOfThought`, and `ReAct`, each implementing a different prompting and reasoning approach. It is responsible for generating predictions from LMs based on program logic. +- **Related Classes/Methods**: `dspy.predict.predict`, `dspy.predict.chain_of_thought`, `dspy.predict.react` \ No newline at end of file