Skip to content

598 - Fix pydantic validation error #663

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
49ae56c
Added screenshot preparation script for screenshot scraping
Santabot123 Aug 23, 2024
e11f0cd
Added text_detection.py and updated screenshot_preparation.py
Santabot123 Aug 24, 2024
b4f8ea4
add __init__.py and docstrings
Santabot123 Aug 26, 2024
0cf7c44
correct the typo and updated select_area_with_ipywidget()
Santabot123 Aug 26, 2024
7e23c3d
correct the typo
Santabot123 Aug 26, 2024
c0a0e69
remove some comments and image
Santabot123 Aug 27, 2024
92bec28
Updated requirements.txt
Santabot123 Aug 28, 2024
aa9e85f
remove some comments and image
Santabot123 Aug 26, 2024
90d7549
updated requirements.txt
Santabot123 Aug 28, 2024
6e9911c
Merge branch 'pre/beta' of https://github.com/Santabot123/Scrapegraph…
Santabot123 Aug 28, 2024
55a7727
Update requirements.txt
Santabot123 Aug 28, 2024
8f615ad
feat: add togheterai
VinciGit00 Aug 28, 2024
34942de
chore(examples): create Together AI examples
f-aguzzi Aug 28, 2024
5f604d1
Merge pull request #605 from ScrapeGraphAI/togheter_ai_integration
f-aguzzi Aug 28, 2024
d7f6036
ci(release): 1.16.0-beta.1 [skip ci]
semantic-release-bot Aug 28, 2024
c348f67
fix: update generate answernode
VinciGit00 Aug 30, 2024
735120d
Merge branch 'screenshot-scraper-fix' into pre/beta
VinciGit00 Aug 30, 2024
405f28e
Merge pull request #606 from Santabot123/pre/beta
VinciGit00 Aug 30, 2024
a0d2113
refactoring of folders
VinciGit00 Aug 30, 2024
388630c
fix: screenshot scraper
VinciGit00 Aug 30, 2024
a73573d
update version
VinciGit00 Aug 31, 2024
9fd6509
Update pyproject.toml
VinciGit00 Aug 31, 2024
9c2aefa
Merge pull request #612 from ScrapeGraphAI/598-1140+-pydantic-validat…
f-aguzzi Aug 31, 2024
1c37d5d
ci(release): 1.16.0-beta.2 [skip ci]
semantic-release-bot Aug 31, 2024
0e0b280
Merge branch 'pre/beta' into temp
VinciGit00 Sep 1, 2024
86f9442
Merge pull request #615 from ScrapeGraphAI/temp
VinciGit00 Sep 1, 2024
886c987
ci(release): 1.16.0-beta.3 [skip ci]
semantic-release-bot Sep 1, 2024
f51b155
add example
VinciGit00 Sep 1, 2024
8422463
feat:expose the search engine params to user
goasleep Sep 2, 2024
a8b0e4a
updated token calculation on parsenode
tm-robinson Sep 2, 2024
3d265a8
change GenerateScraperNode to only use first chunk
tm-robinson Sep 2, 2024
1bcc0bf
Merge pull request #620 from goasleep/feature/export_search_engine
VinciGit00 Sep 2, 2024
ba5c7ad
ci(release): 1.16.0-beta.4 [skip ci]
semantic-release-bot Sep 2, 2024
e741602
Merge branch 'pre/beta' into 543-ScriptCreatorGraph-only-use-first-chunk
VinciGit00 Sep 2, 2024
fd0a902
Merge pull request #619 from tm-robinson/543-ScriptCreatorGraph-only-…
VinciGit00 Sep 2, 2024
13efd4e
ci(release): 1.17.0-beta.1 [skip ci]
semantic-release-bot Sep 2, 2024
ef2db0c
Update pyproject.toml
VinciGit00 Sep 2, 2024
74dfc69
fix(DeepSeek): proper model initialization
f-aguzzi Sep 2, 2024
398b2c5
fix(Ollama): instance model from correct package
f-aguzzi Sep 2, 2024
1e466cd
Merge branch 'pre/beta' into screenshot-scraper-fix
VinciGit00 Sep 2, 2024
3ff69cb
Merge pull request #614 from ScrapeGraphAI/screenshot-scraper-fix
VinciGit00 Sep 2, 2024
89b1f10
Merge pull request #621 from ScrapeGraphAI/609-fix-deepseek-instancing
VinciGit00 Sep 2, 2024
08afc92
ci(release): 1.17.0-beta.2 [skip ci]
semantic-release-bot Sep 2, 2024
66a3b6d
fix: Parse Node scraping link and img urls allowing OmniScraper to work
LorenzoPaleari Sep 2, 2024
57337a0
fix: Removed link_urls and img_ulrs from FetchNode output
LorenzoPaleari Sep 2, 2024
b8ef937
fix(ScreenshotScraper): impose dynamic imports
f-aguzzi Sep 2, 2024
5242166
fix(SmartScraper): pass llm_model to ParseNode
f-aguzzi Sep 2, 2024
aed5452
Merge pull request #624 from ScrapeGraphAI/fix-import-errors
VinciGit00 Sep 2, 2024
fc55418
ci(release): 1.17.0-beta.3 [skip ci]
semantic-release-bot Sep 2, 2024
81af62d
Merge pull request #622 from LorenzoPaleari/pre/beta
VinciGit00 Sep 2, 2024
5e99071
ci(release): 1.17.0-beta.4 [skip ci]
semantic-release-bot Sep 2, 2024
8e74ac5
fix: correctly parsing output when using structured_output
LorenzoPaleari Sep 2, 2024
8442700
Merge pull request #626 from LorenzoPaleari/598-fix-pydantic-errors
VinciGit00 Sep 2, 2024
16ab1bf
ci(release): 1.17.0-beta.5 [skip ci]
semantic-release-bot Sep 2, 2024
52fe441
fix(ScreenShotScraper): static import of optional dependencies
f-aguzzi Sep 4, 2024
e477a44
Merge pull request #631 from ScrapeGraphAI/627-PIL-import-error
VinciGit00 Sep 4, 2024
50c9c6b
ci(release): 1.17.0-beta.6 [skip ci]
semantic-release-bot Sep 4, 2024
bd4b26d
feat: ConcatNode.py added for heavy merge operations
ekinsenler Sep 4, 2024
f83c3d1
add example for gemini
ekinsenler Sep 4, 2024
c0339d9
fix file name
ekinsenler Sep 4, 2024
63a5d18
fix(AbstractGraph): Bedrock init issues
f-aguzzi Sep 5, 2024
31aff6b
Merge pull request #636 from ScrapeGraphAI/633-bedrock-support-fix
VinciGit00 Sep 5, 2024
4347afb
ci(release): 1.17.0-beta.7 [skip ci]
semantic-release-bot Sep 5, 2024
2859fb7
feat(AbstractGraph): add adjustable rate limit
f-aguzzi Sep 5, 2024
c382b9d
Merge pull request #630 from ScrapeGraphAI/595-rate-limit-error
VinciGit00 Sep 6, 2024
85c374e
ci(release): 1.17.0-beta.8 [skip ci]
semantic-release-bot Sep 6, 2024
8b02cb4
Merge pull request #632 from ekinsenler/concat_node
VinciGit00 Sep 6, 2024
77d0fd3
ci(release): 1.17.0-beta.9 [skip ci]
semantic-release-bot Sep 6, 2024
9e9c775
add examples multi concat
VinciGit00 Sep 6, 2024
94e69a0
feat: add scrape_do_integration
VinciGit00 Sep 6, 2024
f5e7a8b
fix of the bug for fetching the code
VinciGit00 Sep 6, 2024
8883bce
asdd proxy integratrion
VinciGit00 Sep 6, 2024
167f970
feat: fetch_node improved
VinciGit00 Sep 7, 2024
afb6eb7
feat: return urls in searchgraph
VinciGit00 Sep 7, 2024
ef7a589
fix: screenshot_scraper
VinciGit00 Sep 7, 2024
af28885
ci(release): 1.17.0-beta.10 [skip ci]
semantic-release-bot Sep 7, 2024
9016bb5
Merge pull request #639 from ScrapeGraphAI/scrape_do_integration
f-aguzzi Sep 7, 2024
a73fec5
ci(release): 1.17.0-beta.11 [skip ci]
semantic-release-bot Sep 7, 2024
fc738ca
Update parse_node.py
VinciGit00 Sep 8, 2024
14c5e6b
Merge branch 'pre/beta' into temp
VinciGit00 Sep 8, 2024
9f52602
Merge pull request #646 from ScrapeGraphAI/temp
VinciGit00 Sep 8, 2024
eddcb79
ci(release): 1.19.0-beta.1 [skip ci]
semantic-release-bot Sep 8, 2024
f2bb22d
fix: temporary fix for parse_node
VinciGit00 Sep 9, 2024
8a0d46b
Merge pull request #641 from ScrapeGraphAI/urls_search_graph
f-aguzzi Sep 9, 2024
32a102a
Merge pull request #648 from ScrapeGraphAI/637-it-can´t-scrape-urls-f…
f-aguzzi Sep 9, 2024
23a260c
ci(release): 1.19.0-beta.2 [skip ci]
semantic-release-bot Sep 9, 2024
947ebd2
fix: parse node
VinciGit00 Sep 10, 2024
4c14fd7
Merge pull request #650 from ScrapeGraphAI/637-it-can´t-scrape-urls-f…
f-aguzzi Sep 10, 2024
38cba96
ci(release): 1.19.0-beta.3 [skip ci]
semantic-release-bot Sep 10, 2024
380174d
add chunking functionn
VinciGit00 Sep 10, 2024
1a7f21f
feat: removed semchunk and used tikton
VinciGit00 Sep 10, 2024
24c38f9
ci(release): 1.19.0-beta.4 [skip ci]
semantic-release-bot Sep 10, 2024
4ee7753
Merge pull request #654 from ScrapeGraphAI/main
VinciGit00 Sep 10, 2024
7621a7c
ci(release): 1.19.0-beta.5 [skip ci]
semantic-release-bot Sep 10, 2024
fe3aa28
refactoring of the code
VinciGit00 Sep 11, 2024
57a58e1
docs: Updated the graph_config in the documentation.
shenghongtw Sep 12, 2024
9eb40e1
Update script_generator_openai.py
VinciGit00 Sep 12, 2024
ca31bd9
Merge pull request #658 from shenghongtw/docs/Updated_the_graph_confi…
VinciGit00 Sep 12, 2024
18277c1
Merge branch 'pre/beta' into temp
VinciGit00 Sep 12, 2024
5ff8cc7
Merge pull request #659 from ScrapeGraphAI/temp
VinciGit00 Sep 12, 2024
ed8e173
ci(release): 1.19.0-beta.6 [skip ci]
semantic-release-bot Sep 12, 2024
da9726f
updates to tokenization for #651 to implement for mistral and ollama
tm-robinson Sep 12, 2024
dc4a76b
use semchunk by default as the other code is causing tokenizers to be…
tm-robinson Sep 12, 2024
c64ce88
refactoting of imports
VinciGit00 Sep 12, 2024
4a16f14
Merge pull request #660 from tm-robinson/651-add-tokenization-for-oll…
VinciGit00 Sep 12, 2024
b805aea
fix: pyproject.toml dependencies
VinciGit00 Sep 12, 2024
4ab26a2
ci(release): 1.19.0-beta.7 [skip ci]
semantic-release-bot Sep 12, 2024
ec6b164
feat: refactoring of the tokenization function
VinciGit00 Sep 12, 2024
88b2c46
ci(release): 1.19.0-beta.8 [skip ci]
semantic-release-bot Sep 12, 2024
039ba2e
fix: Fixed pydantic error on SearchGraphs
LorenzoPaleari Sep 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
204 changes: 202 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,122 @@
## [1.18.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.18.2...v1.18.3) (2024-09-11)
## [1.19.0-beta.8](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.19.0-beta.7...v1.19.0-beta.8) (2024-09-12)


### Features

* refactoring of the tokenization function ([ec6b164](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ec6b164653250fdf01fd4db1454ea7534822f9cf))

## [1.19.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.19.0-beta.6...v1.19.0-beta.7) (2024-09-12)


### Bug Fixes

* pyproject.toml dependencies ([b805aea](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b805aea1deb227e213bb9a027924d49058fefcc1))

## [1.19.0-beta.6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.19.0-beta.5...v1.19.0-beta.6) (2024-09-12)


### Bug Fixes

* models tokens ([039fe3c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/039fe3c6d91978f70baedfef407bda912a285aed))

## [1.18.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.18.1...v1.18.2) (2024-09-10)

### Docs

* Updated the graph_config in the documentation. ([57a58e1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/57a58e162e254828d890e1a110cb5d3d4beb03df))


### CI

* **release:** 1.18.2 [skip ci] ([e1a9caa](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/e1a9caa905f2a62d5b245a0abbcf4d304bd24de3))
* **release:** 1.18.3 [skip ci] ([4bd4659](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4bd4659dc15ae5c7f71702ad6acab200c2a64921))

### Bug Fixes


* models tokens ([039fe3c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/039fe3c6d91978f70baedfef407bda912a285aed))

## [1.18.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.18.1...v1.18.2) (2024-09-10)

* models tokens ([b2be6b7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b2be6b739e0a6b71e16867f751012bc2d95f72c9))

## [1.19.0-beta.4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.19.0-beta.3...v1.19.0-beta.4) (2024-09-10)


### Features

* removed semchunk and used tikton ([1a7f21f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/1a7f21fbf34dc9ef17bca683e2139a88eed70b16))

## [1.19.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.19.0-beta.2...v1.19.0-beta.3) (2024-09-10)


### Bug Fixes


* parse node ([947ebd2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/947ebd2895408c5ebd00b9a3da1b220937553c4a))

## [1.19.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.19.0-beta.1...v1.19.0-beta.2) (2024-09-09)


### Features

* return urls in searchgraph ([afb6eb7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/afb6eb7e4796ab208a050ad04ad96a83406f7fa1))


### Bug Fixes

* temporary fix for parse_node ([f2bb22d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/f2bb22d8e9b3ac5c1560793a6ec09f9ae4f257d3))

## [1.19.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.18.1...v1.19.0-beta.1) (2024-09-08)


### Features

* **AbstractGraph:** add adjustable rate limit ([2859fb7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2859fb72d699f26b617ed2f949cdcfca1671c5c8))
* add scrape_do_integration ([94e69a0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/94e69a051591aeec1e7268bf0d5e0338f90e9539))
* add togheterai ([8f615ad](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8f615adef320dacdd214a184981384dd05df8171))
* ConcatNode.py added for heavy merge operations ([bd4b26d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/bd4b26d7d7c1a7953d1bc9d78b436007880028c9))
* fetch_node improved ([167f970](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/167f97040f081867cecff542c3af8aa122499ce8))


### Bug Fixes

* **AbstractGraph:** Bedrock init issues ([63a5d18](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/63a5d18486789ce1b4a8f5ea661fc83779fceca2)), closes [#633](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/633)
* correctly parsing output when using structured_output ([8e74ac5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8e74ac55a16ca012b52affbc754e4b04130e65db))
* **ScreenshotScraper:** impose dynamic imports ([b8ef937](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b8ef93738ec4ae48c361fe5650df5194e845a2b1))
* **Ollama:** instance model from correct package ([398b2c5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/398b2c556faf518ca28ccc284bc8761a16281cf7))
* Parse Node scraping link and img urls allowing OmniScraper to work ([66a3b6d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/66a3b6d6a3efdf1ee72b802fc9bf8175482c45bd))
* **SmartScraper:** pass llm_model to ParseNode ([5242166](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/52421665759032bcfad80ce540efebe5f47310f6))
* **DeepSeek:** proper model initialization ([74dfc69](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/74dfc693f6e487d20da58704284fe9f492d2b2aa))
* Removed link_urls and img_ulrs from FetchNode output ([57337a0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/57337a0a8c86fb28c9ccbd70d41acfc9abea11f0))
* screenshot scraper ([388630c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/388630c0ffa2850c3d5ea47e62b71b41795203d8))
* screenshot_scraper ([ef7a589](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ef7a5891dcb1b4ed8a97947f5563fa78af917ecb))
* **ScreenShotScraper:** static import of optional dependencies ([52fe441](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/52fe441c5af9c728983a2c3cd880fe9afcb5d428))
* update generate answernode ([c348f67](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c348f674ad0caae4f4dc04e194fae9634e01b621))


### chore

* **examples:** create Together AI examples ([34942de](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/34942deca514df53e8aa1c7f96f812ee78b994bf))


### CI

* **release:** 1.16.0-beta.1 [skip ci] ([d7f6036](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d7f6036f907eda8d1faa0944da4d1d168ca4c40e))
* **release:** 1.16.0-beta.2 [skip ci] ([1c37d5d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/1c37d5db1c637f791133df254838a0deade6d6be))
* **release:** 1.16.0-beta.3 [skip ci] ([886c987](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/886c987172bb57fb59863e4d7b494797bba16980))
* **release:** 1.16.0-beta.4 [skip ci] ([ba5c7ad](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ba5c7adcea138d993005377f4cfe438795e1b124))
* **release:** 1.17.0-beta.1 [skip ci] ([13efd4e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/13efd4e3a4175e85e7c41f5d575a249c27ecbf1d))
* **release:** 1.17.0-beta.10 [skip ci] ([af28885](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/af2888539e4ce83ab5f52b5c605ecc3472b14aff))
* **release:** 1.17.0-beta.11 [skip ci] ([a73fec5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/a73fec5a98f5e646dd8f7d08dfe2dd0dbe067a94))
* **release:** 1.17.0-beta.2 [skip ci] ([08afc92](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/08afc9292ea8ae227b75f640db3d4dd097265482))
* **release:** 1.17.0-beta.3 [skip ci] ([fc55418](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/fc55418a4511389d053e8c6b9a28878a3bc91fe6))
* **release:** 1.17.0-beta.4 [skip ci] ([5e99071](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/5e990719cfc9e063fc2253fc70b3da14fae49360))
* **release:** 1.17.0-beta.5 [skip ci] ([16ab1bf](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/16ab1bf3d920ae8e3dbac372f075e4853200a0e9))
* **release:** 1.17.0-beta.6 [skip ci] ([50c9c6b](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/50c9c6bd8ca67d3d4d83ca3717085042e8a51bc5))
* **release:** 1.17.0-beta.7 [skip ci] ([4347afb](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/4347afb8d4d93f600221d8f77c2701361f0f96a2)), closes [#633](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/633)
* **release:** 1.17.0-beta.8 [skip ci] ([85c374e](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/85c374e4b38f825af20e9e3d095c3a467025fdca))
* **release:** 1.17.0-beta.9 [skip ci] ([77d0fd3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/77d0fd3dba8d52aff8321ab5ff1a1cc8b92b0837))

## [1.18.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.18.0...v1.18.1) (2024-09-08)


Expand All @@ -22,13 +127,15 @@
## [1.18.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.17.0...v1.18.0) (2024-09-08)



### Features

* **browser_base_fetch:** add async_mode to support both synchronous and asynchronous execution ([d56253d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d56253d183969584cacc0cb164daa0152462f21c))

## [1.17.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.16.0...v1.17.0) (2024-09-08)



### Features

* **docloaders:** Enhance browser_base_fetch function flexibility ([57fd01f](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/57fd01f9a76ea8ea69ec04b7238ab58ca72ac8f4))
Expand All @@ -38,9 +145,84 @@

* **sponsor:** 🅱️ Browserbase sponsor 🅱️ ([a540139](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/a5401394cc939d9a5fc58b8a9145141c2f047bab))

* **AbstractGraph:** add adjustable rate limit ([2859fb7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/2859fb72d699f26b617ed2f949cdcfca1671c5c8))

## [1.17.0-beta.7](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.17.0-beta.6...v1.17.0-beta.7) (2024-09-05)


### Bug Fixes

* **AbstractGraph:** Bedrock init issues ([63a5d18](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/63a5d18486789ce1b4a8f5ea661fc83779fceca2)), closes [#633](https://github.com/ScrapeGraphAI/Scrapegraph-ai/issues/633)

## [1.17.0-beta.6](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.17.0-beta.5...v1.17.0-beta.6) (2024-09-04)


### Bug Fixes

* **ScreenShotScraper:** static import of optional dependencies ([52fe441](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/52fe441c5af9c728983a2c3cd880fe9afcb5d428))

## [1.17.0-beta.5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.17.0-beta.4...v1.17.0-beta.5) (2024-09-02)


### Bug Fixes

* correctly parsing output when using structured_output ([8e74ac5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8e74ac55a16ca012b52affbc754e4b04130e65db))

## [1.17.0-beta.4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.17.0-beta.3...v1.17.0-beta.4) (2024-09-02)


### Bug Fixes

* Parse Node scraping link and img urls allowing OmniScraper to work ([66a3b6d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/66a3b6d6a3efdf1ee72b802fc9bf8175482c45bd))
* Removed link_urls and img_ulrs from FetchNode output ([57337a0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/57337a0a8c86fb28c9ccbd70d41acfc9abea11f0))

## [1.17.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.17.0-beta.2...v1.17.0-beta.3) (2024-09-02)


### Bug Fixes

* **ScreenshotScraper:** impose dynamic imports ([b8ef937](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/b8ef93738ec4ae48c361fe5650df5194e845a2b1))
* **SmartScraper:** pass llm_model to ParseNode ([5242166](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/52421665759032bcfad80ce540efebe5f47310f6))

## [1.17.0-beta.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.17.0-beta.1...v1.17.0-beta.2) (2024-09-02)


### Bug Fixes

* **Ollama:** instance model from correct package ([398b2c5](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/398b2c556faf518ca28ccc284bc8761a16281cf7))
* **DeepSeek:** proper model initialization ([74dfc69](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/74dfc693f6e487d20da58704284fe9f492d2b2aa))
* screenshot scraper ([388630c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/388630c0ffa2850c3d5ea47e62b71b41795203d8))

## [1.17.0-beta.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.16.0...v1.17.0-beta.1) (2024-09-02)


### Features

* add togheterai ([8f615ad](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/8f615adef320dacdd214a184981384dd05df8171))


### Bug Fixes

* update generate answernode ([c348f67](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/c348f674ad0caae4f4dc04e194fae9634e01b621))


### chore

* **examples:** create Together AI examples ([34942de](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/34942deca514df53e8aa1c7f96f812ee78b994bf))


### CI

* **release:** 1.16.0-beta.1 [skip ci] ([d7f6036](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d7f6036f907eda8d1faa0944da4d1d168ca4c40e))
* **release:** 1.16.0-beta.2 [skip ci] ([1c37d5d](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/1c37d5db1c637f791133df254838a0deade6d6be))
* **release:** 1.16.0-beta.3 [skip ci] ([886c987](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/886c987172bb57fb59863e4d7b494797bba16980))
* **release:** 1.16.0-beta.4 [skip ci] ([ba5c7ad](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/ba5c7adcea138d993005377f4cfe438795e1b124))


## [1.16.0](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.2...v1.16.0) (2024-09-01)



### Features

* add deepcopy error ([71b22d4](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/71b22d48804c462798109bb47ec792a5a3c70b6e))
Expand All @@ -54,10 +236,28 @@
## [1.15.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.1...v1.15.2) (2024-09-01)


## [1.16.0-beta.3](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.16.0-beta.2...v1.16.0-beta.3) (2024-09-01)


### Bug Fixes

* pyproject.toml ([360ce1c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/360ce1c0e468c959e63555120ac7cecf55563846))


### CI

* **release:** 1.15.2 [skip ci] ([d88730c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/d88730ccc7190d09a54e6c24db1644512b576430))

## [1.15.2](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.1...v1.15.2) (2024-09-01)




### Bug Fixes

* pyproject.toml ([360ce1c](https://github.com/ScrapeGraphAI/Scrapegraph-ai/commit/360ce1c0e468c959e63555120ac7cecf55563846))


## [1.15.1](https://github.com/ScrapeGraphAI/Scrapegraph-ai/compare/v1.15.0...v1.15.1) (2024-08-28)


Expand Down
14 changes: 11 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,10 @@ Additional dependecies can be added while installing the library:

- <b>More Language Models</b>: additional language models are installed, such as Fireworks, Groq, Anthropic, Hugging Face, and Nvidia AI Endpoints.

```bash
pip install scrapegraphai[other-language-models]
```

This group allows you to use additional language models like Fireworks, Groq, Anthropic, Together AI, Hugging Face, and Nvidia AI Endpoints.
```bash
pip install scrapegraphai[other-language-models]

- <b>Semantic Options</b>: this group includes tools for advanced semantic processing, such as Graphviz.

Expand All @@ -58,6 +59,13 @@ Additional dependecies can be added while installing the library:



### Installing "More Browser Options"

This group includes an ocr scraper for websites
```bash
pip install scrapegraphai[screenshot_scraper]
```

## 💻 Usage
There are multiple standard scraping pipelines that can be used to extract information from a website (or local file).

Expand Down
2 changes: 1 addition & 1 deletion docs/chinese.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ from scrapegraphai.graphs import SpeechGraph
graph_config = {
"llm": {
"api_key": "OPENAI_API_KEY",
"model": "gpt-3.5-turbo",
"model": "openai/gpt-3.5-turbo",
},
"tts_model": {
"api_key": "OPENAI_API_KEY",
Expand Down
2 changes: 1 addition & 1 deletion docs/japanese.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ from scrapegraphai.graphs import SpeechGraph
graph_config = {
"llm": {
"api_key": "OPENAI_API_KEY",
"model": "gpt-3.5-turbo",
"model": "openai/gpt-3.5-turbo",
},
"tts_model": {
"api_key": "OPENAI_API_KEY",
Expand Down
2 changes: 1 addition & 1 deletion docs/korean.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ from scrapegraphai.graphs import SpeechGraph
graph_config = {
"llm": {
"api_key": "OPENAI_API_KEY",
"model": "gpt-3.5-turbo",
"model": "openai/gpt-3.5-turbo",
},
"tts_model": {
"api_key": "OPENAI_API_KEY",
Expand Down
2 changes: 1 addition & 1 deletion docs/russian.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ from scrapegraphai.graphs import SpeechGraph
graph_config = {
"llm": {
"api_key": "OPENAI_API_KEY",
"model": "gpt-3.5-turbo",
"model": "openai/gpt-3.5-turbo",
},
"tts_model": {
"api_key": "OPENAI_API_KEY",
Expand Down
2 changes: 1 addition & 1 deletion docs/source/getting_started/examples.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ OpenAI models
graph_config = {
"llm": {
"api_key": openai_key,
"model": "gpt-3.5-turbo",
"model": "openai/gpt-3.5-turbo",
},
}

Expand Down
2 changes: 1 addition & 1 deletion examples/anthropic/custom_graph_haiku.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@

fetch_node = FetchNode(
input="url | local_dir",
output=["doc", "link_urls", "img_urls"],
output=["doc"],
node_config={
"verbose": True,
"headless": True,
Expand Down
48 changes: 48 additions & 0 deletions examples/anthropic/rate_limit_haiku.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
"""
Basic example of scraping pipeline using SmartScraper while setting an API rate limit.
"""

import os
from dotenv import load_dotenv
from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai.utils import prettify_exec_info


# required environment variables in .env
# ANTHROPIC_API_KEY
load_dotenv()

# ************************************************
# Create the SmartScraperGraph instance and run it
# ************************************************

graph_config = {
"llm": {
"api_key": os.getenv("ANTHROPIC_API_KEY"),
"model": "anthropic/claude-3-haiku-20240307",
"rate_limit": {
"requests_per_second": 1
}
},
}

smart_scraper_graph = SmartScraperGraph(
prompt="""Don't say anything else. Output JSON only. List me all the events, with the following fields: company_name, event_name, event_start_date, event_start_time,
event_end_date, event_end_time, location, event_mode, event_category,
third_party_redirect, no_of_days,
time_in_hours, hosted_or_attending, refreshments_type,
registration_available, registration_link""",
# also accepts a string with the already downloaded HTML code
source="https://www.hmhco.com/event",
config=graph_config
)

result = smart_scraper_graph.run()
print(result)

# ************************************************
# Get graph execution info
# ************************************************

graph_exec_info = smart_scraper_graph.get_execution_info()
print(prettify_exec_info(graph_exec_info))
Loading
Loading