how to customize http headers when fetch html page ?

[lamysafari]

For example, I want to set UA to 'GoogleBot' to get same result like search engine's crawling .
Thank you!

[B4nan]

You can set the user agent via crawler options, e.g. https://crawlee.dev/api/playwright-crawler/interface/PlaywrightLaunchContext#userAgent

[lamysafari]

thank you. I found that javascript rendering crawler has this option, but not found in sample http crawler like cheerioCrawler that I need.

[B4nan]

there is no browser with cheerio, but you can set it as a header. to do that, define a preNavigationHandler and adjust the second parameter - the gotOptions, which is the options objects for got, the HTTP client we use to run requests

https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions#preNavigationHooks

[lamysafari]

What's the difference between gotOptions.headers in preVavigationHooks and request.headers in requestHandler? This is my code and how to verify if it really works?

const cheerioCrawl  = new CheerioCrawler({
    preNavigationHooks: [
        async(_, gotOptions)=> {
            gotOptions.headers = {
                'user-agent': 'GoogleBot'
            }
        }
    ],
    async requestHandler(inputs) {
        const { enqueueLinks, request} = inputs;
        console.log(request.headers); // output: {}
        await enqueueLinks();
    }
})

[B4nan]

When you get to request handler, the request has been already made.

[alexbs]

or like this:
(both tested - works - ok)

const cheerioCrawl  = new CheerioCrawler({
  preNavigationHooks: [
    async ({ request }) => {
     request.headers = {
       'User-Agent': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
     };
   }
  ]
})