Remove pandas and datasets from core dependencies (#8274)

* move pandas, datasets and optuna as optional deps

* update test deps

* update docs

* update notebooks

* add warning

* remove optimize optional dependency group

* avoid broken version

* add back optuna

* fix notebooks

* revert readme change

* fix test_save_and_load_with_pkl

Author: TomeHirata

docs/docs/index.md

@@ -23,7 +23,7 @@ Instead of wrangling prompts or training jobs, DSPy (Declarative Self-improving
 !!! info "Getting Started I: Install DSPy and set up your LM"
 
     ```bash
-    > pip install -U dspy
+    > pip install -U dspy datasets
     ```
 
     === "OpenAI"

docs/docs/tutorials/agents/index.ipynb

@@ -8,7 +8,7 @@
     "\n",
     "Let's walk through a quick example of setting up a `dspy.ReAct` agent with a couple of tools and optimizing it to conduct advanced browsing for multi-hop search.\n",
     "\n",
-    "Install the latest DSPy via `pip install -U dspy` and follow along.\n",
+    "Install the latest DSPy via `pip install -U dspy` and follow along. You also need to run `pip install datasets`.\n",
     "\n",
     "<details>\n",
     "<summary>Recommended: Set up MLflow Tracing to understand what's happening under the hood.</summary>\n",

docs/docs/tutorials/audio/index.ipynb

@@ -24,7 +24,7 @@
     "To handle audio data, install the following dependencies:\n",
     "\n",
     "```shell\n",
-    "pip install soundfile torch==2.0.1+cu118 torchaudio==2.0.2+cu118\n",
+    "pip install datasets soundfile torch==2.0.1+cu118 torchaudio==2.0.2+cu118\n",
     "```\n"
    ]
   },

docs/docs/tutorials/classification_finetuning/index.ipynb

@@ -12,7 +12,7 @@
     "\n",
     "### Install dependencies and download data\n",
     "\n",
-    "Install the latest DSPy via `pip install -U dspy>=2.6.0` and follow along (or `uv pip`, if you prefer). This tutorial depends on DSPy >= 2.6.0.\n",
+    "Install the latest DSPy via `pip install -U dspy` and follow along (or `uv pip`, if you prefer). This tutorial depends on DSPy >= 2.6.0. You also need to run `pip install datasets`.\n",
     "\n",
     "This tutorial requires a local GPU at the moment for inference, though we plan to support ollama serving for finetuned models as well.\n",
     "\n",

docs/docs/tutorials/entity_extraction/index.ipynb

@@ -24,7 +24,7 @@
       "outputs": [],
       "source": [
         "# Install the latest version of DSPy\n",
-        "%pip install -U dspy-ai\n",
+        "%pip install -U dspy\n",
         "# Install the Hugging Face datasets library to load the CoNLL-2003 dataset\n",
         "%pip install datasets"
       ]

docs/docs/tutorials/math/index.ipynb

@@ -8,7 +8,7 @@
     "\n",
     "Let's walk through a quick example of setting up a `dspy.ChainOfThought` module and optimizing it for answering algebra questions.\n",
     "\n",
-    "Install the latest DSPy via `pip install -U dspy` and follow along.\n",
+    "Install the latest DSPy via `pip install -U dspy` and follow along. You also need to run `pip install datasets`.\n",
     "\n",
     "<details>\n",
     "<summary>Recommended: Set up MLflow Tracing to understand what's happening under the hood.</summary>\n",

docs/docs/tutorials/mcp/index.md

@@ -20,7 +20,7 @@ to [MCP servers built by the community](https://modelcontextprotocol.io/examples
 Before starting, let's install the required dependencies:
 
 ```shell
-pip install -U dspy mcp
+pip install -U dspy[mcp]
 ```
 
 ## MCP Server Setup

docs/docs/tutorials/multihop_search/index.ipynb

@@ -8,7 +8,7 @@
     "\n",
     "Let's walk through a quick example of building a `dspy.Module` with multiple sub-modules. We'll do this for the task for multi-hop search.\n",
     "\n",
-    "Install the latest DSPy via `pip install -U dspy` and follow along."
+    "Install the latest DSPy via `pip install -U dspy` and follow along. You also need to run `pip install datasets`."
    ]
   },
   {

docs/docs/tutorials/rag/index.ipynb

@@ -8,7 +8,7 @@
     "\n",
     "Let's walk through a quick example of **basic question answering** with and without **retrieval-augmented generation** (RAG) in DSPy. Specifically, let's build **a system for answering Tech questions**, e.g. about Linux or iPhone apps.\n",
     "\n",
-    "Install the latest DSPy via `pip install -U dspy` and follow along. If you're looking instead for a conceptual overview of DSPy, this [recent lecture](https://www.youtube.com/live/JEMYuzrKLUw) is a good place to start."
+    "Install the latest DSPy via `pip install -U dspy` and follow along. If you're looking instead for a conceptual overview of DSPy, this [recent lecture](https://www.youtube.com/live/JEMYuzrKLUw) is a good place to start. You also need to run `pip install datasets`."
    ]
   },
   {

docs/docs/tutorials/tool_use/index.ipynb

@@ -8,7 +8,7 @@
     "\n",
     "Let's walk through a quick example of building and prompt-optimizing a DSPy agent for advanced tool use. We'll do this for the challenging task [ToolHop](https://arxiv.org/abs/2501.02506) but with an even stricter evaluation criteria.\n",
     "\n",
-    "Install the latest DSPy via `pip install -U dspy` and follow along. You will also need to `pip install func_timeout`."
+    "Install the latest DSPy via `pip install -U dspy` and follow along. You will also need to `pip install func_timeout datasets`."
    ]
   },
   {