[Websearch] update (#427)

* Fix reuqest body

* update webSearchQueryPromptTemplate

* update generate google query parser

* Add today's date to google search query creator

* crawl top stories if exts; remove answer_box & knowledgeGraph

* Create paragraph chunks from top articles

* flattened paragprah chunks

* update status texts

* add gradio client

* call gradio app for RAG

* Web scrape only "p, li, span" els

* add MAX_N_CHUNKS

* gradio result typing

* parse only <p> elements

* rm dev change

* update typing WebSearch

* buld RAG prompt

* Rm dev change

* change websearch context msg from user to assisntat type

* use hosted gradio app

* fix lint

* prompt engineering

* more prompt engineering

* MAX_N_PAGES_SCRAPE = 10

* better error msg

* more prompt engineering

* revert websearch prompt to previous

* rm `top_stories` from websearch as the results are not good

* Stop using gradio client, use regular fetch

* chore

* Rm websearchsummary references as it is no longer used

* update readme

* Apply suggestions from code review

Co-authored-by: Julien Chaumond <julien@huggingface.co>

* Use tfjs to do embeddings in server node

* fix websearch component disapperar after finishing generation

* Show sources of closest embeddings used in RAG

* fix prompting and also add current date

* add comment

* comment for search query

* sources

* hide www

* using hostname direclty

* Show successful web pages instead of failed ones

* rm noisy messages

* google query generation using previous messaages as context

* handle falcon generation

* bring back Browsing webpage msg

---------

Co-authored-by: Julien Chaumond <julien@huggingface.co>
Co-authored-by: Victor Mustar <victor.mustar@gmail.com>

Author: 3 people

README.md

@@ -155,7 +155,7 @@ You can change things like the parameters, or customize the preprompt to better
 
 By default the prompt is constructed using `userMessageToken`, `assistantMessageToken`, `userMessageEndToken`, `assistantMessageEndToken`, `preprompt` parameters and a series of default templates.
 
-However, these templates can be modified by setting the `chatPromptTemplate`, `webSearchSummaryPromptTemplate`, and `webSearchQueryPromptTemplate` parameters. Note that if WebSearch is not enabled, only `chatPromptTemplate` needs to be set. The template language is https://handlebarsjs.com. The templates have access to the model's prompt parameters (`preprompt`, etc.). However, if the templates are specified it is recommended to inline the prompt parameters, as using the references (`{{preprompt}}`) is deprecated.
+However, these templates can be modified by setting the `chatPromptTemplate` and `webSearchQueryPromptTemplate` parameters. Note that if WebSearch is not enabled, only `chatPromptTemplate` needs to be set. The template language is https://handlebarsjs.com. The templates have access to the model's prompt parameters (`preprompt`, etc.). However, if the templates are specified it is recommended to inline the prompt parameters, as using the references (`{{preprompt}}`) is deprecated.
 
 For example:
 
@@ -187,33 +187,14 @@ The following is the default `chatPromptTemplate`, although newlines and indenti
 
 When performing a websearch, the search query is constructed using the `webSearchQueryPromptTemplate` template. It is recommended that that the prompt instructs the chat model to only return a few keywords.
 
-The following is the default `webSearchQueryPromptTemplate`. Note that not all models supports consecutive user-messages which this template uses.
+The following is the default `webSearchQueryPromptTemplate`.
 
 ```
 {{userMessageToken}}
-  The following messages were written by a user, trying to answer a question.
+  My question is: {{message.content}}.
+  Based on the conversation history (my previous questions are: {{previousMessages}}), give me an appropriate query to answer my question for google search. You should not say more than query. You should not say any words except the query. For the context, today is {{currentDate}}
 {{userMessageEndToken}}
-{{#each messages}}
-  {{#ifUser}}{{@root.userMessageToken}}{{content}}{{@root.userMessageEndToken}}{{/ifUser}}
-{{/each}}
-{{userMessageToken}}
-  What plain-text english sentence would you input into Google to answer the last question? Answer with a short (10 words max) simple sentence.
-{{userMessageEndToken}}
-{{assistantMessageToken}}Query:
-```
-
-**webSearchSummaryPromptTemplate**
-
-The search-engine response (`answer`) is summarized using the following prompt template. However, when `HF_ACCESS_TOKEN` is provided, a dedicated summary model is used instead. Additionally, the model's `query` response to `webSearchQueryPromptTemplate` is also available to this template.
-
-The following is the default `webSearchSummaryPromptTemplate`. Note that not all models supports consecutive user-messages which this template uses.
-
-```
-{{userMessageToken}}{{answer}}{{userMessageEndToken}}
-{{userMessageToken}}
-  The text above should be summarized to best answer the query: {{query}}.
-{{userMessageEndToken}}
-{{assistantMessageToken}}Summary:
+{{assistantMessageToken}}
 ```
 
 #### Running your own models using a custom endpoint