feat: Durable web search golem:web-search API across multiple providers in Rust

[SaikiranSurapalli17]

Introduction

This pull request introduces a robust and extensible durable web search API for the Golem platform, implemented in Rust. This web-search component provides a unified interface for performing web searches across multiple providers—including Brave, Google, Serper, and Tavily—enabling users to leverage the strengths of each search engine through a single, consistent API.

Key features include support for durable, replayable search sessions, advanced filtering (such as domain inclusion/exclusion and safe search levels), multilingual and regional queries, and the ability to retrieve both standard and enriched content (like images and HTML snippets). The API is designed for reliability and flexibility, making it suitable for a wide range of use cases, from simple information retrieval to advanced, policy-driven or content-sensitive searches.

---

Tests Added

A comprehensive suite of tests has been included to validate the new web search API, including:

1. Basic Search Test:
   Verifies simple search queries return expected results.

2. Session Pagination Test:
   Ensures multiple pages of results can be fetched in a session.

3. Error Handling Test:
   Confirms graceful handling of invalid queries and provider errors.

4. Domain Filtering Test:
   Tests inclusion and exclusion of specific domains in search results.

5. Image and HTML Content Test:
   Checks correct handling of image results and HTML snippets.

6. Multilingual and Regional Search Test:
   Validates search with different languages and regions.

7. Advanced Search Test:
   Demonstrates advanced search with high safe search and content chunks.
   This test ensures the API can perform advanced queries with strict safe search settings and return results in content chunks, supporting use cases like child safety and trusted content discovery.

   /closes Implement Durable Web Search golem:web-search API Across Different Providers in Rust #34
   /claim Implement Durable Web Search golem:web-search API Across Different Providers in Rust #34

[SaikiranSurapalli17]

Google

google.mp4

[SaikiranSurapalli17]

Brave

brave.mp4

[SaikiranSurapalli17]

serper

serper.mp4

[SaikiranSurapalli17]

tavily

tavily.mp4

[SaikiranSurapalli17]

cc
@jdegoes @vigoo

[SaikiranSurapalli17]

Hi @mschuwalow,
I’ve fixed all Clippy errors and the Ollama build is now working. Could you please rerun the CI workflow when you have a moment? Thanks!

[mschuwalow]

Some more clippy issues. You should be able to just run cargo make fix to get these fixed. Please also remove all commited .DS_STORE files

[mschuwalow]

This looks promising! A few minor changes left

[SaikiranSurapalli17]

@mschuwalow I'd like to clarify the scope for the web search providers. Are the existing providers sufficient for this task, or is there an expectation to implement any new ones beyond those detailed in the issue?

[Rutik7066]

@mschuwalow I'd like to clarify the scope for the web search providers. Are the existing providers sufficient for this task, or is there an expectation to implement any new ones beyond those detailed in the issue?

I was using exa in my personal side hustle project. I found it more usefull. So thought why not add it to golem.

They have this https://docs.exa.ai/reference/search#body-type feature that lets you decide to get search results like google which will be seo ranked or more relevant result where seo does not matter.

[mschuwalow]

@mschuwalow I'd like to clarify the scope for the web search providers. Are the existing providers sufficient for this task, or is there an expectation to implement any new ones beyond those detailed in the issue?

I was using exa in my personal side hustle project. I found it more usefull. So thought why not add it to golem.

They have this https://docs.exa.ai/reference/search#body-type feature that lets you decide to get search results like google which will be seo ranked or more relevant result where seo does not matter.

Adding more is of course appreciated, but the current set is enough for me to merge when all comments are addressed.

[mschuwalow]

Some cleanup left

[Rutik7066]

@mschuwalow This is really wrong.