Comparison with got-scraping

[garysassano]

I'd like to better understand how ky compares against got-sraping. The HeaderGeneratorOptions seems a pretty trivial feature to me, since it just sets very few headers you could manually define yourself. I guess ky could do the same?

[sholladay]

Ky users generally wouldn't need to "generate realistic browser headers" since, unlike Got, Ky can run directly in an actual browser. Of course, if you're using Ky in a Node/Deno environment to do web scraping, you'll still need to generate those headers. But then, if you need the features of got-scraping, why not just use that?

Someone could certainly make an equivalent Ky-scraping module that uses .extend() and a hook to add the headers and then use it with Ky on the server in the same way. In fact, it would probably be fairly easy to make such a module that supports both Ky and Got, either as a peer dependency or with dependency injection.

[garysassano]

I tend to prefer native fetch(), or something that builds upon it while improving the DX like Ky does. I think it would be nice to have a ky-scraping module indeed with built-in proxy support and fingerprint spoofing (like fingerprint-suite or CycleTLS).

[sholladay]

Sure, feel free to make one. It sounds useful, though probably not something we will add to Ky itself. We could potentially link to it from the readme. I would like to see more such third party extensions. It encourages experimenting with different approaches and helps gauge interest in particular features.