Request made with reqwest blocked by Cloudflare's protection but succeeds with curl

[filips123]

A request made with reqwest to https://www.crunchyroll.com/manifest.json is blocked by Cloudflare's protection/challenge. However, a request to the same URL with the same headers using curl works.

This is a minimal reproducible example for this issue:

use anyhow::Result;
use reqwest::blocking::Client;
use reqwest::header::{HeaderMap, HeaderValue};

pub fn create_client() -> reqwest::Result<Client> {
    let mut headers = HeaderMap::new();
    headers.insert("Sec-Fetch-Site", HeaderValue::from_static("none"));
    headers.insert("Sec-Fetch-Dest", HeaderValue::from_static("manifest"));

    let builder = Client::builder()
        .user_agent("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0")
        .default_headers(headers);

    builder.build()
}

fn main() -> Result<()> {
    let url = "https://www.crunchyroll.com/manifest.json";

    let response = create_client()?
        .get(url.to_owned())
        .header(reqwest::header::REFERER, url.to_owned())
        .send()?
        .text()?;

    println!("{}", response);

    Ok(())
}

The client is configured with some additional headers, but removing them does not fix the issue. This example has been tested with these dependencies:

[dependencies]
anyhow = "1.0.81"
reqwest = { version = "0.12.2", features = ["blocking", "socks", "gzip", "brotli", "deflate", "native-tls"] }

Note that the exact versions do not seem to matter, as this issue also happens in my actual project with older dependencies (and reqwest 0.11.23).

This code should download the URL and print it. When tested, it downloads an HTML page which seems to be Cloudflare's challenge, instead of the actual JSON manifest.

However, when I try to download the URL with the same headers using curl from the command line, it works:

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0" -H "Sec-Fetch-Site: none" -H "Sec-Fetch-Dest: manifest" --referer "https://www.crunchyroll.com/manifest.json" "https://www.crunchyroll.com/manifest.json"

{
  "background_color": "#23252b",
  "display": "minimal-ui",
  "name": "Crunchyroll - Watch Popular Anime",
  "orientation": "any",
  "scope": "/",
  "short_name": "Crunchyroll",
  "start_url": "/?utm_medium=pwa_app_launch&utm_source=pwa",
  "theme_color": "#f47521",
  "description": "Stream the world’s largest anime library. Watch over 1,000 titles—from past seasons to new episodes fresh from Japan",
  "categories": ["entertainment", "video"],
  "icons": [
    {
      "src": "https://crunchyroll.com/build/assets/img/pwa/512.png",
      "sizes": "512x512",
      "type": "image/png",
      "purpose": "any"
    },
    {
      "src": "https://crunchyroll.com/build/assets/img/pwa/512-maskable.png",
      "sizes": "512x512",
      "type": "image/png",
      "purpose": "maskable"
    }
  ],
  "scope_extensions": [
    {
      "origin": "www.crunchyroll.com"
    },
    {
      "origin": "sso.crunchyroll.com"
    }
  ],
  "shortcuts": [
    {
      "name": "View Watchlist",
      "short_name": "Watchlist",
      "url": "/watchlist?utm_medium=pwa_app_launch&utm_source=pwa_shortcut",
      "description": "Pick up where you left off with your favorite anime series and movies",
      "icons": [
        {
          "src": "https://crunchyroll.com/build/assets/img/pwa/icons/watchlist-icon.png",
          "sizes": "96x96",
          "type": "image/png",
          "purpose": "any"
        }
      ]
    },
    {
      "name": "Browse Anime",
      "short_name": "Browse",
      "url": "/videos/popular?utm_medium=pwa_app_launch&utm_source=pwa_shortcut",
      "description": "See all anime Crunchyroll has to offer by popularity, freshness, and genre",
      "icons": [
        {
          "src": "https://crunchyroll.com/build/assets/img/pwa/icons/browse-icon.png",
          "sizes": "96x96",
          "type": "image/png",
          "purpose": "any"
        }
      ]
    },
    {
      "name": "Search",
      "short_name": "Search",
      "url": "/search?utm_medium=pwa_app_launch&utm_source=pwa_shortcut",
      "description": "Search for anime on Crunchyroll",
      "icons": [
        {
          "src": "https://crunchyroll.com/build/assets/img/pwa/icons/search-icon.png",
          "sizes": "96x96",
          "type": "image/png",
          "purpose": "any"
        }
      ]
    },
    {
      "name": "View Crunchylists",
      "short_name": "Crunchylists",
      "url": "/crunchylists?utm_medium=pwa_app_launch&utm_source=pwa_shortcut",
      "description": "Create and manage your custom Crunchylists",
      "icons": [
        {
          "src": "https://crunchyroll.com/build/assets/img/pwa/icons/crunchylists-icon.png",
          "sizes": "96x96",
          "type": "image/png",
          "purpose": "any"
        }
      ]
    },
    {
      "name": "View History",
      "short_name": "History",
      "url": "/history?utm_medium=pwa_app_launch&utm_source=pwa_shortcut",
      "description": "See your most recently watched videos on Crunchyroll",
      "icons": [
        {
          "src": "https://crunchyroll.com/build/assets/img/pwa/icons/history-icon.png",
          "sizes": "96x96",
          "type": "image/png",
          "purpose": "any"
        }
      ]
    }
  ],
  "id": "com.crunchyroll.pwa",
  "dir": "auto"
}

I tested both ways multiple times (from the same computer), and the results were always the same. This issue also happens to some users of my project (filips123/PWAsForFirefox#482). It also seems to be specific to Crunchyroll, as it works fine with other Cloudflare-hosted websites.

Does reqwest send some different hidden headers than curl that make Cloudflare block the request? Or does it maybe do some advanced detection that can differentiate between curl and reqwest? So, is it possible to somehow configure reqwest client so it is not detected and blocked by Cloudflare?

[whispersilk]

I'm experiencing a similar issue. When I copy Cloudflare advanced protection cookies from my browser and plug them into a curl request, the request succeeds. When I use those same credentials with reqwest, I get a 403. For example, suppose that __cf_bm=x and cf_clearance=y. When I plug these values into a Client::get, printing the resulting RequestBuilder shows me the following headers:

headers: {
    "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0",
    "accept": "*/*"
    "cookie": "__cf_bm=x; cookies=yes; cf_clearance=y",
}

and sending the request returns 403 forbidden.

When I then plug those into curl like so:

curl $URL \
    -H "accept: */*" \
    -H "user-agent: Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0" \
    -H "cookie: __cf_bm=x; cookies=yes; cf_clearance=y"

I get a 200 success. curl -vv shows me that HTTP/2 is being used and that no extra headers are being added. I'm not sure how to get additional information about what reqwest is sending out (if there's any additional information to get) so I don't know why curl is succeeding and reqwest isn't. From what I can view, they appear to be identical.

[paolobarbolini]

I can't see what reqwest is doing as far as ALPN is concerned, but it looks like not offering both http/1.1 and http/2 as options is what's causing CloudFlare to complain even though cookies are provided.

Could you try enabling the http2 feature and also native-tls-alpn if you're using native-tls?

[whispersilk]

http2 was enabled (I'm using default features), but I added native-tls-alpn and it looks like reqwest is using HTTP/2 now! Unfortunately it doesn't fix the issue, but it's probably one step closer. This is what my client construction and get looks like:

let cookies: Arc<Jar> = Arc::new(Default::default());
let client = ReqwestClient::builder()
    .connection_verbose(true)
    .cookie_provider(cookies.clone())
    .build()
    .expect("Failed to build client");

// Load cookies.

let user_agent = "Mozilla/5.0 (X11; Linux x86_64; rv:124.0) Gecko/20100101 Firefox/124.0";
let get = client.client.get(url.as_ref()).header("User-Agent", user_agent);

if let Some(cookie) = client.cookies.cookies(&Url::from_str(url.as_ref())?) {
    println!("curl {url} -H \"user-agent: {user_agent}\" -H \"cookie: {}\"", cookie.to_str().unwrap());
};

When I run the program, I still get a 403, and then when I copy the printed curl command and paste it into my terminal without changes it runs successfully.

Thank you for helping me look into this!

[filips123]

In my case, adding native-tls-alpn fixed the issue.

Without native-tls-alpn, reqwest is also using HTTP/1.1 and is blocked. However, interestingly, in my case, curl uses HTTP/1.1 (because curl on Windows does not support HTTP/2) but still succeeds.

[filips123]

And I found something even more weird... If I set the user-agent in curl with -A option, it works, but if I set it with -H User-Agent header, it doesn't:

$ curl -vvvv -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0" --referer "https://www.crunchyroll.com/manifest.json" "https://www.crunchyroll.com/manifest.json"

* Host www.crunchyroll.com:443 was resolved.
* IPv6: (none)
* IPv4: 104.18.34.202, 172.64.153.54
*   Trying 104.18.34.202:443...
* Connected to www.crunchyroll.com (104.18.34.202) port 443
* schannel: disabled automatic use of client certificate
* using HTTP/1.x
> GET /manifest.json HTTP/1.1
> Host: www.crunchyroll.com
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0
> Accept: */*
> Referer: https://www.crunchyroll.com/manifest.json
>
* Request completely sent off
< HTTP/1.1 200 OK
...

$ curl -vvvv -H "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0" --referer "https://www.crunchyroll.com/manifest.json" "https://www.crunchyroll.com/manifest.json"

* Host www.crunchyroll.com:443 was resolved.
* IPv6: (none)
* IPv4: 104.18.34.202, 172.64.153.54
*   Trying 104.18.34.202:443...
* Connected to www.crunchyroll.com (104.18.34.202) port 443
* schannel: disabled automatic use of client certificate
* using HTTP/1.x
> GET /manifest.json HTTP/1.1
> Host: www.crunchyroll.com
> Accept: */*
> Referer: https://www.crunchyroll.com/manifest.json
> User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0
>
* Request completely sent off
< HTTP/1.1 403 Forbidden
...

The only difference I can see is the order of headers.

request, when using HTTP/1.1, sends headers in this order:

GET /manifest.json HTTP/1.1
referer: https://www.crunchyroll.com/manifest.json
accept: */*
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0
sec-fetch-site: none
sec-fetch-dest: manifest
accept-encoding: gzip, br, zstd, deflate
host: www.crunchyroll.com

So, maybe they do detection based on the order (and capitalization) of headers, and only for on HTTP/1.1? But I don't know why ALPN didn' fix the issue for @whispersilk.

[whispersilk]

@filips123 I think the difference is that crunchyroll.com is just behind standard CloudFlare. I'm trying to access www.fanfiction.net story pages (e.g. this), which appear to be permanently under CloudFlare's "I'm Under Attack" mode. So I've got the relevant cookies and am attaching them to the request, and for whatever reason CloudFlare is calling the request suspicious anyway — but only for reqwest, not for curl.

[TnZzZHlp]

I encountered a similar issue, but the error Cloudflare gave me was

his website is using a security service to protect itself from online attacks. The action you just performed triggered the security solution. There are several actions that could trigger this block including submitting a certain word or phrase, a SQL command or malformed data.

        let res = match self
            .client
            .get("https://api.bgm.tv/v0/subjects/12")
            .send()
            .await
        {
            Ok(res) => res,
            Err(e) => {
                return Err(e.to_string());
            }
        };

Even if I write it this way, the same error still occurs, but it works fine with curl. This is really strange.

[TnZzZHlp]

After adding the native-tls-alpn feature, it works fine.

[skairunner]

I encountered this issue on a Cloudflare site too, including the part where other ways of sending the request like Postman or Curl worked fine. Enabling HTTP/2 and adding native-tls-alpn resolved the issue for me.

[kornelski]

Does it help if you use User-Agent that correctly indicates it's from reqwest?

If you make a request from something that isn't Firefox while sending User-Agent saying you are Firefox, this looks very much like a spambot.

Requests from curl use curl's own User-Agent string and don't pretend to be something else.

[TnZzZHlp]

Useless. This isn't an issue with the User-Agent, it's likely caused by Cloudflare's internal implementation

[kornelski]

I mean that Cloudflare's bot detection looks at the User-Agent string, as well as many other properties of the request, and can tell when the request was made by a real Firefox browser, and when it came some other client pretending to be Firefox.

Faking a browser makes your requests similar to requests from spambots that routinely try to hide behind fake User-Agent headers. This may make your requests judged more harshly than requests from software that doesn't hide its identity.

[filips123]

Well, at least in my case, setting the "correct" user-agent also won't work in general, as some other websites just block all non-browser user-agents... Setting the browser user-agent at least bypasses those trivial checks, and seems to work on most websites, apart from those using more advanced detections like Cloudflare.

[blabra]

I have got the same problem. I still got blocked by the website even with the native-tls-alpn feature enabled. But it works fine when i just switch to ureq. The link is https://yande.re/post.json.

#![allow(unused_imports)]
use reqwest::{self, tls};
use std::collections::HashMap;
use ureq;
static URL: &str = "https://yande.re/post.json";

// This will respond correctly
fn main() -> Result<(), ureq::Error> {
    let res = ureq::get(URL).call()?.body_mut().read_to_string()?;
    println!("{}", res);
    Ok(())
}
// And this will always respond 404 Error
#[tokio::main]
async fn main() -> Result<(), reqwest::Error> {
    let cli = reqwest::Client::new();
    let res = cli.get(URL).send().await?;
    dbg!(&res);
    dbg!(&res.headers());
    dbg!(res.text().await?);
    Ok(())
}

[seanmonstar]

This looks like a case of not setting a user-agent. Testing your code fails for me too, but when I add a .user_agent(some_value), it passes.

ureq provides a default user-agent, but reqwest decided for itself that it's not the right place to pick a user-agent, since a library isn't a user-agent, the program you build is the agent.