Merge branch 'pre/beta' of https://github.com/VinciGit00/Scrapegraph-ai into pre/beta

Author: VinciGit00

.gitignore

@@ -23,6 +23,7 @@ docs/source/_static/
 venv/
 .venv/
 .vscode/
+.conda/
 
 # exclude pdf, mp3
 *.pdf
@@ -38,3 +39,6 @@ lib/
 *.html
 .idea
 
+# extras
+cache/
+run_smart_scraper.py

CHANGELOG.md

@@ -1,3 +1,34 @@
+## [1.7.0-beta.4](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.7.0-beta.3...v1.7.0-beta.4) (2024-06-12)
+
+
+### Bug Fixes
+
+* common params ([6b4cdf9](https://github.com/VinciGit00/Scrapegraph-ai/commit/6b4cdf92b82fa143e4217a2e5da46d04f2585de8))
+
+## [1.7.0-beta.3](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.7.0-beta.2...v1.7.0-beta.3) (2024-06-11)
+
+
+### Features
+
+* add caching ([d790361](https://github.com/VinciGit00/Scrapegraph-ai/commit/d79036149a3197a385b73553f29df66d36480c38))
+* add dynamic caching ([7ed2fe8](https://github.com/VinciGit00/Scrapegraph-ai/commit/7ed2fe8ef0d16fd93cb2ff88840bcaa643349e33))
+* add new chunking function ([e1f045b](https://github.com/VinciGit00/Scrapegraph-ai/commit/e1f045b2809fc7db0c252f4c6f2f9a435c66ba91))
+* **merge:** add scriptcreatormulti, rag cache and semchunk ([15421ef](https://github.com/VinciGit00/Scrapegraph-ai/commit/15421eff7009b80293f7d84df5086d22944dfb99))
+* **schema:** merge scripts to follow pydantic schema ([5d692bf](https://github.com/VinciGit00/Scrapegraph-ai/commit/5d692bff9e4f124146dd37e573f7c3c0aa8d9a23))
+* refactoring of rag node ([7a13a68](https://github.com/VinciGit00/Scrapegraph-ai/commit/7a13a6819ff35a6f6197ee837d0eb8ea65e31776))
+
+
+### Bug Fixes
+
+* **cache:** correctly pass the node arguments and logging ([c881f64](https://github.com/VinciGit00/Scrapegraph-ai/commit/c881f64209a86a69ddd3105f5d0360d9ed183490))
+* **node:** fixed generate answer node pydantic schema ([ab00f23](https://github.com/VinciGit00/Scrapegraph-ai/commit/ab00f23d859c64995ccfe329b24379cf3c14d73c))
+
+
+### Docs
+
+* **cache:** added cache_path param ([edddb68](https://github.com/VinciGit00/Scrapegraph-ai/commit/edddb682d06262088885e340b7b73cc70adf9583))
+* **scriptcreator:** enhance documentation ([650c3aa](https://github.com/VinciGit00/Scrapegraph-ai/commit/650c3aaa60dab169358c2c04bfca9dee8d1a5d68))
+
 ## [1.7.0-beta.2](https://github.com/VinciGit00/Scrapegraph-ai/compare/v1.7.0-beta.1...v1.7.0-beta.2) (2024-06-10)
 
 

README.md

@@ -43,11 +43,14 @@ The documentation for ScrapeGraphAI can be found [here](https://scrapegraph-ai.r
 Check out also the Docusaurus [here](https://scrapegraph-doc.onrender.com/).
 
 ## 💻 Usage
-There are three main scraping pipelines that can be used to extract information from a website (or local file):
+There are multiple standard scraping pipelines that can be used to extract information from a website (or local file):
 - `SmartScraperGraph`: single-page scraper that only needs a user prompt and an input source;
 - `SearchGraph`: multi-page scraper that extracts information from the top n search results of a search engine;
 - `SpeechGraph`: single-page scraper that extracts information from a website and generates an audio file.
-- `SmartScraperMultiGraph`: multiple page scraper given a single prompt
+- `ScriptCreatorGraph`: single-page scraper that extracts information from a website and generates a Python script.
+
+- `SmartScraperMultiGraph`: multi-page scraper that extracts information from multiple pages given a single prompt and a list of sources;
+- `ScriptCreatorMultiGraph`: multi-page scraper that generates a Python script for extracting information from multiple pages given a single prompt and a list of sources.
 
 It is possible to use different LLM through APIs, such as **OpenAI**, **Groq**, **Azure** and **Gemini**, or local models using **Ollama**.
 

docs/assets/scriptcreatorgraph.png

@@ -13,6 +13,7 @@ Some interesting ones are:
 - `loader_kwargs`: A dictionary with additional parameters to be passed to the `Loader` class, such as `proxy`.
 - `burr_kwargs`: A dictionary with additional parameters to enable `Burr` graphical user interface.
 - `max_images`: The maximum number of images to be analyzed. Useful in `OmniScraperGraph` and `OmniSearchGraph`.
+- `cache_path`: The path where the cache files will be saved. If already exists, the cache will be loaded from this path.
 
 .. _Burr:
 

docs/source/scrapers/graph_config.rst

@@ -6,11 +6,15 @@ Graphs are scraping pipelines aimed at solving specific tasks. They are composed
 There are several types of graphs available in the library, each with its own purpose and functionality. The most common ones are:
 
 - **SmartScraperGraph**: one-page scraper that requires a user-defined prompt and a URL (or local file) to extract information using LLM.
-- **SmartScraperMultiGraph**: multi-page scraper that requires a user-defined prompt and a list of URLs (or local files) to extract information using LLM. It is built on top of SmartScraperGraph.
 - **SearchGraph**: multi-page scraper that only requires a user-defined prompt to extract information from a search engine using LLM. It is built on top of SmartScraperGraph.
 - **SpeechGraph**: text-to-speech pipeline that generates an answer as well as a requested audio file. It is built on top of SmartScraperGraph and requires a user-defined prompt and a URL (or local file).
 - **ScriptCreatorGraph**: script generator that creates a Python script to scrape a website using the specified library (e.g. BeautifulSoup). It requires a user-defined prompt and a URL (or local file).
 
+There are also two additional graphs that can handle multiple sources:
+
+- **SmartScraperMultiGraph**: similar to `SmartScraperGraph`, but with the ability to handle multiple sources.
+- **ScriptCreatorMultiGraph**: similar to `ScriptCreatorGraph`, but with the ability to handle multiple sources.
+
 With the introduction of `GPT-4o`, two new powerful graphs have been created:
 
 - **OmniScraperGraph**: similar to `SmartScraperGraph`, but with the ability to scrape images and describe them.
@@ -186,4 +190,37 @@ It will fetch the data from the source, extract the information based on the pro
    )
 
    result = speech_graph.run()
-   print(result)
+   print(result)
+
+
+ScriptCreatorGraph & ScriptCreatorMultiGraph
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. image:: ../../assets/scriptcreatorgraph.png
+   :align: center
+   :width: 90%
+   :alt: ScriptCreatorGraph
+
+First we define the graph configuration, which includes the LLM model and other parameters.
+Then we create an instance of the ScriptCreatorGraph class, passing the prompt, source, and configuration as arguments. Finally, we run the graph and print the result.
+
+.. code-block:: python
+
+   from scrapegraphai.graphs import ScriptCreatorGraph
+
+   graph_config = {
+      "llm": {...},
+      "library": "beautifulsoup4"
+   }
+
+   script_creator_graph = ScriptCreatorGraph(
+      prompt="Create a Python script to scrape the projects.",
+      source="https://perinim.github.io/projects/",
+      config=graph_config,
+      schema=schema
+   )
+
+   result = script_creator_graph.run()
+   print(result)
+
+**ScriptCreatorMultiGraph** is similar to ScriptCreatorGraph, but it can handle multiple sources. We define the graph configuration, create an instance of the ScriptCreatorMultiGraph class, and run the graph.

docs/source/scrapers/graphs.rst

@@ -0,0 +1,53 @@
+""" 
+Basic example of scraping pipeline using ScriptCreatorGraph
+"""
+
+import os
+from dotenv import load_dotenv
+from scrapegraphai.graphs import ScriptCreatorMultiGraph
+from scrapegraphai.utils import prettify_exec_info
+
+load_dotenv()
+
+# ************************************************
+# Define the configuration for the graph
+# ************************************************
+
+graph_config = {
+    "llm": {
+        "api_key": os.getenv("ANTHROPIC_API_KEY"),
+        "model": "claude-3-haiku-20240307",
+        "max_tokens": 4000
+        },
+        "library": "beautifulsoup"
+}
+
+# ************************************************
+# Create the ScriptCreatorGraph instance and run it
+# ************************************************
+
+urls=[
+    "https://schultzbergagency.com/emil-raste-karlsen/",
+    "https://schultzbergagency.com/johanna-hedberg/",
+]
+
+# ************************************************
+# Create the ScriptCreatorGraph instance and run it
+# ************************************************
+
+script_creator_graph = ScriptCreatorMultiGraph(
+    prompt="Find information about actors",
+    # also accepts a string with the already downloaded HTML code
+    source=urls,
+    config=graph_config
+)
+
+result = script_creator_graph.run()
+print(result)
+
+# ************************************************
+# Get graph execution info
+# ************************************************
+
+graph_exec_info = script_creator_graph.get_execution_info()
+print(prettify_exec_info(graph_exec_info))

examples/anthropic/script_multi_generator_haiku.py

@@ -12,31 +12,14 @@
 # Define the configuration for the graph
 # ************************************************
 
-openai_key = os.getenv("OPENAI_APIKEY")
-
-""" 
-Basic example of scraping pipeline using SmartScraper
-"""
-
-import os, json
-from dotenv import load_dotenv
-from scrapegraphai.graphs import SmartScraperMultiGraph
-
 load_dotenv()
 
-# ************************************************
-# Define the configuration for the graph
-# ************************************************
-
-openai_key = os.getenv("OPENAI_APIKEY")
-
 graph_config = {
     "llm": {
-        "api_key": openai_key,
-        "model": "gpt-4o",
-    },
-    "verbose": True,
-    "headless": False,
+        "api_key": os.getenv("ANTHROPIC_API_KEY"),
+        "model": "claude-3-haiku-20240307",
+        "max_tokens": 4000
+        },
 }
 
 # *******************************************************

examples/anthropic/smart_scraper_multi_haiku.py

@@ -25,7 +25,8 @@
 )
 graph_config = {
     "llm": {"model_instance": llm_model_instance},
-    "embeddings": {"model_instance": embedder_model_instance}
+    "embeddings": {"model_instance": embedder_model_instance},
+    "library": "beautifulsoup"
 }
 
 # ************************************************

examples/azure/script_generator_azure.py

@@ -0,0 +1,61 @@
+""" 
+Basic example of scraping pipeline using ScriptCreatorGraph
+"""
+
+import os
+from dotenv import load_dotenv
+from scrapegraphai.graphs import ScriptCreatorMultiGraph
+from scrapegraphai.utils import prettify_exec_info
+from langchain_openai import AzureChatOpenAI
+from langchain_openai import AzureOpenAIEmbeddings
+
+load_dotenv()
+
+# ************************************************
+# Define the configuration for the graph
+# ************************************************
+llm_model_instance = AzureChatOpenAI(
+    openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
+    azure_deployment=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT_NAME"]
+)
+
+embedder_model_instance = AzureOpenAIEmbeddings(
+    azure_deployment=os.environ["AZURE_OPENAI_EMBEDDINGS_DEPLOYMENT_NAME"],
+    openai_api_version=os.environ["AZURE_OPENAI_API_VERSION"],
+)
+graph_config = {
+    "llm": {"model_instance": llm_model_instance},
+    "embeddings": {"model_instance": embedder_model_instance},
+    "library": "beautifulsoup"
+}
+
+
+# ************************************************
+# Create the ScriptCreatorGraph instance and run it
+# ************************************************
+
+urls=[
+    "https://schultzbergagency.com/emil-raste-karlsen/",
+    "https://schultzbergagency.com/johanna-hedberg/",
+]
+
+# ************************************************
+# Create the ScriptCreatorGraph instance and run it
+# ************************************************
+
+script_creator_graph = ScriptCreatorMultiGraph(
+    prompt="Find information about actors",
+    # also accepts a string with the already downloaded HTML code
+    source=urls,
+    config=graph_config
+)
+
+result = script_creator_graph.run()
+print(result)
+
+# ************************************************
+# Get graph execution info
+# ************************************************
+
+graph_exec_info = script_creator_graph.get_execution_info()
+print(prettify_exec_info(graph_exec_info))