Restore requests support

[ValueRaider]

Thanks to @deeleeramone #2486 (comment), learnt that requests can work if it has a curl_adapter.

Confirmed working with rate limiter. Also works with requests_cache, but probably always misses cache because of crumb changing.

[ValueRaider]

Bad news. Works fine for a few requests but after ~10 I get error:

requests.exceptions.HTTPError: 401 Client Error: for url: https://query2.finance.yahoo.com/v10/finance/quoteSummary...

Tried with and without rate limiting. Must be a problem with curl_adapter. @deeleeramone what is your experience?

[ValueRaider]

@el1s7 hope you don't mind me tagging you as owner of curl_adapter. yfinance works fine with curl_cffi, not so well with your adapter (see previous message). Can you provide any insight into differences between them?

[el1s7]

@el1s7 hope you don't mind me tagging you as owner of curl_adapter. yfinance works fine with curl_cffi, not so well with your adapter (see previous message). Can you provide any insight into differences between them?

Hi @ValueRaider ,

It's unclear if the 401 unauthorized error that you're getting is related to curl_adapter. I recommend doing some additional testing and debugging, and let me know if you find any issue related to the library.

The current version of the library passes all the tests and it works the same as requests library. I'm also about to release a new version soon with further stability & performance improvements.

[deeleeramone]

Bad news. Works fine for a few requests but after ~10 I get error:

requests.exceptions.HTTPError: 401 Client Error: for url: https://query2.finance.yahoo.com/v10/finance/quoteSummary...

Tried with and without rate limiting. Must be a problem with curl_adapter. @deeleeramone what is your experience?

So far it's been alright, although I am pinned to 0.2.58. I'm not getting rate-limiting issues, and doing Ticker.info for a 200 symbols in one shot does work without dropping out; although, the request itself seems to be slower than before switching. Do you know if this change gets rid of connection pooling?

Had some busy weeks, so haven't been able to give you a decent answer here, sorry!

[ValueRaider]

Do you know if this change gets rid of connection pooling?

Don't know. Put a @utils.log_indent_decorator on YfData._set_session and run with debug mode, see if something keeps resetting it.

[deeleeramone]

It seems to be doing a retry effort, but TBH, I'm not sure what I'm looking for. I do see it toggling between cookie strategies, so retrying until it hits, and doesn't seem to fail. It could explain the reduction in speed, which is not detrimental.

Anecdotally, yf.download with ~30 tickers & 3 years of daily has been reliable as ever, but I haven't checked in on to see how it goes via TradingView UDF with search/quote/historical happening all at once. I'll let you know!

In [28]: yf.Ticker("BND", session=session).get_info()
DEBUG    get_raw_json(): https://query2.finance.yahoo.com/v10/finance/quoteSummary/BND
DEBUG    Entering get()
DEBUG     Entering _make_request()
DEBUG      url=https://query2.finance.yahoo.com/v10/finance/quoteSummary/BND
DEBUG      params={'modules': 'financialData,quoteType,defaultKeyStatistics,assetProfile,summaryDetail', 'corsDomain': 'finance.yahoo.com', 'formatted': 'false', 'symbol': 'BND'}
DEBUG      Entering _get_cookie_and_crumb()
DEBUG       cookie_mode = 'basic'
DEBUG       Entering _get_cookie_and_crumb_basic()
DEBUG        reusing crumb
DEBUG       Exiting _get_cookie_and_crumb_basic()
DEBUG      Exiting _get_cookie_and_crumb()
DEBUG      response code=401
DEBUG      toggling cookie strategy basic -> csrf
DEBUG      Entering _get_cookie_and_crumb()
DEBUG       cookie_mode = 'csrf'
DEBUG       Entering _get_crumb_csrf()
DEBUG        Failed to find "csrfToken" in response
DEBUG       Exiting _get_crumb_csrf()
DEBUG       toggling cookie strategy csrf -> basic
DEBUG       Entering _get_cookie_and_crumb_basic()
DEBUG        crumb = 'NPyL9jTes66'
DEBUG       Exiting _get_cookie_and_crumb_basic()
DEBUG      Exiting _get_cookie_and_crumb()
DEBUG      response code=200
DEBUG     Exiting _make_request()
DEBUG    Exiting get()
DEBUG    get_raw_json(): https://query1.finance.yahoo.com/v7/finance/quote?
DEBUG    Entering get()
DEBUG     Entering _make_request()
DEBUG      url=https://query1.finance.yahoo.com/v7/finance/quote?
DEBUG      params={'symbols': 'BND', 'formatted': 'false'}
DEBUG      Entering _get_cookie_and_crumb()
DEBUG       cookie_mode = 'basic'
DEBUG       Entering _get_cookie_and_crumb_basic()
DEBUG        reusing crumb
DEBUG       Exiting _get_cookie_and_crumb_basic()
DEBUG      Exiting _get_cookie_and_crumb()
DEBUG      response code=200
DEBUG     Exiting _make_request()
DEBUG    Exiting get()

[ValueRaider]

I'm not sure what I'm looking for

If session object is not overwritten then connection pooling should work.

[jasonvogel]

Just trying this library for the first time today.

Immediately hit:

Failed to perform, curl: (60) SSL certificate problem: self signed certificate in certificate chain. See https://curl.se/libcurl/c/libcurl-errors.html first for more details.

In your Session usage, please add:

session.verify = False  # Disable SSL verification