ScrapeGraphAI · VinciGit00 · Aug 9, 2024 · Aug 9, 2024 · Aug 10, 2024 · Aug 10, 2024
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,76 @@
+## [1.14.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.6...v1.14.0-beta.7) (2024-08-16)
+
+
+### Bug Fixes
+
+* model count ([faef318](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/faef3186f795e950ade14bc8b6d8d1cea3afd327))
+
+## [1.14.0-beta.6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.5...v1.14.0-beta.6) (2024-08-16)
+
+
+### Features
+
+* add integration for new module of gpt4o ([982150e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/982150e81fbaa4241c725aaa9dfcd553f8b86978))
+
+## [1.14.0-beta.5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.4...v1.14.0-beta.5) (2024-08-16)
+
+
+### Features
+
+* Add new feature to support gpt-4o variant models with different pricing ([8551448](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/855144876d796ceebb0930fec45ead6cc3834f14))
+
+## [1.14.0-beta.4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.3...v1.14.0-beta.4) (2024-08-15)
+
+
+### Features
+
+* update abstract graph ([c77231c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c77231c983bd6e154eefd26422cd156da4c8b7bb))
+
+## [1.14.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.2...v1.14.0-beta.3) (2024-08-13)
+
+
+### Bug Fixes
+
+* **models_tokens:** incorrect provider names ([cb6b353](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/cb6b35397e56c6785553480200aa948053d9904b))
+
+
+### chore
+
+* **examples:** add vertex examples, rename genai examples ([1aa9c6e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/1aa9c6e73bfa26b83010cf8d980cdf5f572cde5a))
+* **examples:** update provider names to match tokens dictionary ([ee078cb](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ee078cb102ad922a900228ebe5ea45724712a960))
+
+## [1.14.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.14.0-beta.1...v1.14.0-beta.2) (2024-08-12)
+
+
+### Bug Fixes
+
+* **AbstractGraph:** pass kwargs to Ernie and Nvidia models ([e6bedb6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e6bedb6701601e87a6dff99eabec9c3494280411))
+
+
+### chore
+
+* **examples:** fix import bug in image2text demo ([71438a1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/71438a1e8696aee51d054f9df7243665497fc35c))
+* **requirements:** update requirements.txt ([7fe181f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/7fe181f69b3178d2d9d41a00fd660a98e04b777e))
+
+## [1.14.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.3...v1.14.0-beta.1) (2024-08-11)
+
+
+### Features
+
+* add refactoring of default temperature ([6c3b37a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/6c3b37ab001b80c09ea9ffb56d4c3df338e33a7a))
+
+
+### Bug Fixes
+
+* broken node ([1272273](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/127227349915deeb0dede34aa575ad269ed7cbe3))
+* merge_anwser prompt import ([f17cef9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f17cef94bb39349d40cc520d93b51ac4e629db32))
+
+
+### CI
+
+* **release:** 1.13.0-beta.8 [skip ci] ([b470d97](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b470d974cf3fdb3a75ead46fceb8c21525e2e616))
+* **release:** 1.13.0-beta.9 [skip ci] ([d4c1a1c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d4c1a1c58a54740ff50aa87b1d1d3500b61ea088))
+
 ## [1.13.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.2...v1.13.3) (2024-08-10)
 
 
@@ -25,6 +98,8 @@
 * conditional node ([ce00345](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ce003454953e5785d4746223c252de38cd5d07ea))
 
 ## [1.13.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.12.2...v1.13.0) (2024-08-09)
+## [1.13.0-beta.9](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.0-beta.8...v1.13.0-beta.9) (2024-08-10)
+
 
 
 ### Features
@@ -65,6 +140,15 @@
 * **release:** 1.13.0-beta.5 [skip ci] ([2eba73b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2eba73b784ee443260117e98ab7c943934b3018d)), closes [#513](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/513)
 * **release:** 1.13.0-beta.6 [skip ci] ([e75b574](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e75b574b67040e127599da9ee1b0eee13d234cb9))
 * **release:** 1.13.0-beta.7 [skip ci] ([6e56925](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/6e56925355c424edae290c70fd98646ab5f420ee))
+* add refactoring of default temperature ([6c3b37a](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/6c3b37ab001b80c09ea9ffb56d4c3df338e33a7a))
+
+## [1.13.0-beta.8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.0-beta.7...v1.13.0-beta.8) (2024-08-09)
+
+
+### Bug Fixes
+
+* broken node ([1272273](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/127227349915deeb0dede34aa575ad269ed7cbe3))
+
 
 ## [1.13.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.13.0-beta.6...v1.13.0-beta.7) (2024-08-09)
 

diff --git a/Dockerfile b/Dockerfile
@@ -1,8 +1,9 @@
 FROM python:3.11-slim
 
-RUN apt-get update && apt-get upgrade -y && \
-useradd -m -s /bin/bash app
-
-USER app
+RUN apt-get update && apt-get upgrade -y
 
 RUN pip install scrapegraphai
+RUN pip install scrapegraphai[burr]
+
+RUN python3 -m playwright install-deps
+RUN python3 -m playwright install
diff --git a/examples/anthropic/custom_graph_haiku.py b/examples/anthropic/custom_graph_haiku.py
@@ -6,7 +6,7 @@
 from dotenv import load_dotenv
 
 from langchain_openai import OpenAIEmbeddings
-from scrapegraphai.models import OpenAI
+from langchain_openai import ChatOpenAI
 from scrapegraphai.graphs import BaseGraph
 from scrapegraphai.nodes import FetchNode, ParseNode, RAGNode, GenerateAnswerNode, RobotsNode
 load_dotenv()

diff --git a/examples/azure/csv_scraper_azure.py b/examples/azure/csv_scraper_azure.py
@@ -25,7 +25,7 @@
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/azure/csv_scraper_graph_multi_azure.py b/examples/azure/csv_scraper_graph_multi_azure.py
@@ -25,7 +25,7 @@
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/azure/json_scraper_azure.py b/examples/azure/json_scraper_azure.py
@@ -23,7 +23,7 @@
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/azure/json_scraper_multi_azure.py b/examples/azure/json_scraper_multi_azure.py
@@ -12,7 +12,7 @@
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/azure/pdf_scraper_azure.py b/examples/azure/pdf_scraper_azure.py
@@ -10,7 +10,7 @@
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/azure/scrape_plain_text_azure.py b/examples/azure/scrape_plain_text_azure.py
@@ -28,7 +28,7 @@
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/azure/script_generator_azure.py b/examples/azure/script_generator_azure.py
@@ -15,7 +15,7 @@
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/azure/script_multi_generator_azure.py b/examples/azure/script_multi_generator_azure.py
@@ -16,7 +16,7 @@
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/azure/search_graph_azure.py b/examples/azure/search_graph_azure.py
@@ -22,7 +22,7 @@
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/azure/search_graph_schema_azure.py b/examples/azure/search_graph_schema_azure.py
@@ -30,7 +30,7 @@ class Dishes(BaseModel):
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/azure/search_link_graph_azure.py b/examples/azure/search_link_graph_azure.py
@@ -15,7 +15,7 @@
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/azure/smart_scraper_azure.py b/examples/azure/smart_scraper_azure.py
@@ -26,7 +26,7 @@
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/azure/smart_scraper_multi_azure.py b/examples/azure/smart_scraper_multi_azure.py
@@ -14,7 +14,7 @@
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/azure/smart_scraper_schema_azure.py b/examples/azure/smart_scraper_schema_azure.py
@@ -28,7 +28,7 @@ class Projects(BaseModel):
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/azure/xml_scraper_azure.py b/examples/azure/xml_scraper_azure.py
@@ -24,7 +24,7 @@
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/azure/xml_scraper_graph_multi_azure.py b/examples/azure/xml_scraper_graph_multi_azure.py
@@ -25,7 +25,7 @@
 graph_config = {
     "llm": {
         "api_key": os.environ["AZURE_OPENAI_KEY"],
-        "model": "azure/gpt-3.5-turbo",
+        "model": "azure_openai/gpt-3.5-turbo",
     },
     "verbose": True,
     "headless": False

diff --git a/examples/ernie/custom_graph_ernie.py b/examples/ernie/custom_graph_ernie.py
@@ -6,7 +6,7 @@
 from dotenv import load_dotenv
 
 from langchain_openai import OpenAIEmbeddings
-from scrapegraphai.models import OpenAI
+from langchain_openai import ChatOpenAI
 from scrapegraphai.graphs import BaseGraph
 from scrapegraphai.nodes import FetchNode, ParseNode, RAGNode, GenerateAnswerNode, RobotsNode
 

diff --git a/examples/gemini/.env.example → examples/google_genai/.env.example b/examples/gemini/.env.example → examples/google_genai/.env.example
diff --git a/examples/gemini/csv_scraper_gemini.py → examples/google_genai/csv_scraper_gemini.py b/examples/gemini/csv_scraper_gemini.py → examples/google_genai/csv_scraper_gemini.py
@@ -24,7 +24,7 @@
 graph_config = {
     "llm": {
         "api_key": gemini_key,
-        "model": "gemini-pro",
+        "model": "google_genai/gemini-pro",
     },
 }
 

diff --git a/.../gemini/csv_scraper_graph_multi_gemini.py → ...e_genai/csv_scraper_graph_multi_gemini.py b/.../gemini/csv_scraper_graph_multi_gemini.py → ...e_genai/csv_scraper_graph_multi_gemini.py
@@ -28,7 +28,7 @@
 graph_config = {
     "llm": {
         "api_key": gemini_key,
-        "model": "gemini-pro",
+        "model": "google_genai/gemini-pro",
     },
 }
 

diff --git a/examples/gemini/custom_graph_gemini.py → examples/google_genai/custom_graph_gemini.py b/examples/gemini/custom_graph_gemini.py → examples/google_genai/custom_graph_gemini.py
@@ -18,7 +18,7 @@
 graph_config = {
     "llm": {
         "api_key": gemini_key,
-        "model": "gemini-pro",
+        "model": "google_genai/gemini-pro",
         "temperature": 0,
         "streaming": True
     },

diff --git a/examples/gemini/inputs/books.xml → examples/google_genai/inputs/books.xml b/examples/gemini/inputs/books.xml → examples/google_genai/inputs/books.xml
diff --git a/examples/gemini/inputs/example.json → examples/google_genai/inputs/example.json b/examples/gemini/inputs/example.json → examples/google_genai/inputs/example.json
diff --git a/...ples/gemini/inputs/plain_html_example.txt → ...oogle_genai/inputs/plain_html_example.txt b/...ples/gemini/inputs/plain_html_example.txt → ...oogle_genai/inputs/plain_html_example.txt
diff --git a/examples/gemini/inputs/username.csv → examples/google_genai/inputs/username.csv b/examples/gemini/inputs/username.csv → examples/google_genai/inputs/username.csv
diff --git a/examples/gemini/json_scraper_gemini.py → examples/google_genai/json_scraper_gemini.py b/examples/gemini/json_scraper_gemini.py → examples/google_genai/json_scraper_gemini.py
@@ -28,7 +28,7 @@
 graph_config = {
     "llm": {
         "api_key": gemini_key,
-        "model": "gemini-pro",
+        "model": "google_genai/gemini-pro",
     },
 }
 

diff --git a/examples/gemini/json_scraper_multi_gemini.py → ...google_genai/json_scraper_multi_gemini.py b/examples/gemini/json_scraper_multi_gemini.py → ...google_genai/json_scraper_multi_gemini.py
@@ -13,7 +13,7 @@
 graph_config = {
     "llm": {
         "api_key": gemini_key,
-        "model": "gemini-pro",
+        "model": "google_genai/gemini-pro",
     },
     "library": "beautifulsoup"
 }

diff --git a/examples/gemini/pdf_scraper_graph_gemini.py → .../google_genai/pdf_scraper_graph_gemini.py b/examples/gemini/pdf_scraper_graph_gemini.py → .../google_genai/pdf_scraper_graph_gemini.py
@@ -19,7 +19,7 @@
 graph_config = {
     "llm": {
         "api_key": gemini_key,
-        "model": "gemini-pro",
+        "model": "google_genai/gemini-pro",
     },
 }
 

diff --git a/examples/gemini/pdf_scraper_multi_gemini.py → .../google_genai/pdf_scraper_multi_gemini.py b/examples/gemini/pdf_scraper_multi_gemini.py → .../google_genai/pdf_scraper_multi_gemini.py
@@ -13,7 +13,7 @@
 graph_config = {
     "llm": {
         "api_key": gemini_key,
-        "model": "gemini-pro",
+        "model": "google_genai/gemini-pro",
     },
     "library": "beautifulsoup"
 }

diff --git a/examples/gemini/readme.md → examples/google_genai/readme.md b/examples/gemini/readme.md → examples/google_genai/readme.md
diff --git a/examples/gemini/scrape_plain_text_gemini.py → .../google_genai/scrape_plain_text_gemini.py b/examples/gemini/scrape_plain_text_gemini.py → .../google_genai/scrape_plain_text_gemini.py
@@ -29,7 +29,7 @@
 graph_config = {
     "llm": {
         "api_key": gemini_key,
-        "model": "gemini-pro",
+        "model": "google_genai/gemini-pro",
         "temperature": 0,
         "streaming": True
     },

diff --git a/examples/gemini/scrape_xml_gemini.py → examples/google_genai/scrape_xml_gemini.py b/examples/gemini/scrape_xml_gemini.py → examples/google_genai/scrape_xml_gemini.py
@@ -29,7 +29,7 @@
 graph_config = {
     "llm": {
         "api_key": gemini_key,
-        "model": "gemini-pro",
+        "model": "google_genai/gemini-pro",
         "temperature": 0,
         "streaming": True
     },

diff --git a/examples/gemini/script_generator_gemini.py → ...s/google_genai/script_generator_gemini.py b/examples/gemini/script_generator_gemini.py → ...s/google_genai/script_generator_gemini.py
@@ -19,7 +19,7 @@
 graph_config = {
     "llm": {
         "api_key": gemini_key,
-        "model": "gemini-pro",
+        "model": "google_genai/gemini-pro",
     },
     "library": "beautifoulsoup"
 }

diff --git a/...s/gemini/script_multi_generator_gemini.py → ...le_genai/script_multi_generator_gemini.py b/...s/gemini/script_multi_generator_gemini.py → ...le_genai/script_multi_generator_gemini.py
@@ -18,7 +18,7 @@
 graph_config = {
     "llm": {
         "api_key": gemini_key,
-        "model": "gemini-pro",
+        "model": "google_genai/gemini-pro",
     },
     "library": "beautifoulsoup"
 }

diff --git a/examples/gemini/search_graph_gemini.py → examples/google_genai/search_graph_gemini.py b/examples/gemini/search_graph_gemini.py → examples/google_genai/search_graph_gemini.py
@@ -17,7 +17,7 @@
 graph_config = {
     "llm": {
         "api_key": gemini_key,
-        "model": "gemini-pro",
+        "model": "google_genai/gemini-pro",
         "temperature": 0,
         "streaming": True
     },

diff --git a/...ples/gemini/search_graph_schema_gemini.py → ...oogle_genai/search_graph_schema_gemini.py b/...ples/gemini/search_graph_schema_gemini.py → ...oogle_genai/search_graph_schema_gemini.py
@@ -32,7 +32,7 @@ class Dishes(BaseModel):
 graph_config = {
     "llm": {
         "api_key": gemini_key,
-        "model": "gemini-pro",
+        "model": "google_genai/gemini-pro",
     },
 }