Investigating: arXiv API flakiness

[lukasschwab]

Description

A clear and concise description of what the bug is.

The arXiv API seems to be degraded. I expect to see more bug reports about this until the underlying issue is resolved.

Behavior identified in #43 seems to have intensified or changed in character (e.g. increased clustering, such that retries are more likely to re-fail, perhaps because of cached bad responses).

Why can't you fix the API?
: I'm not affiliated with arXiv — I maintain a wrapper library for an API I don't administer. I've written the arxiv-api Google Group about this issue.

Why aren't you merging bug fixes?
: Some of the proposed changes here (e.g. consolidating on HTTPS, pinning a specific feedparser version, etc.) are probably good changes regardless of the API's stability. I'm hesitant to rush merging and releasing changes without having a strong sense, through integration tests, that they don't damage this library's behavior. That judgment is subject to change, esp. if this issue persists.

Steps to reproduce

Steps to reproduce the behavior; ideally, include a code snippet.

• Differing results between HTTPS and HTTP calls.
• Unexpectedly empty pages of results, per Unreliable results: pages from API are unexpectedly empty #43.
• This is reflected in integration test CI instability, both on master and on various feature branches.

Versions

• python version: independent.

• arxiv.py version: 1.*.*.

Additional context

Add any other context about the problem here.

PRs directly addressing the instability:

• Debug flaky CI integration tests #127
• Use HTTPS in API Endpoint #128

[lukasschwab]

Good example of flakiness between identical versions/protocols: #132 (comment)

[liyucheng09]

Good diagnosis for this issue. I guess there is not too much we can do unless they fix the backend.

[liyucheng09]

BTW I found arxiv treats requests differently for programatic clients and real browsers. I suspect this flakiness is on purpose.

[lukasschwab]

@liyucheng09 can you share any details on that investigation? In #127 I tried tweaking the user-agent.

[liyucheng09]

I tried about 300 attempts hourly today. More than 3000 in total. 0 out of 3000 suceeded.
By sending a user-agent to the feedparser, 28 out of 100 suceeded.
I suppose we could safely say arxiv is declining requests from programmatic clients.

[Ar4ikov]

Hello! feedparser that needs to arxiv lib works contains that... I really can't describe my emotions, when I'd seen that first time. (feedparser/init.py)

USER_AGENT = "feedparser/%s +https://github.com/kurtmckee/feedparser/" % __version__

Does the developer find out this funny?

Instead of using normally worked application, I need to cp -r /path/to/site-packages/feedparser /path/to/my-project-dir/, change USER_AGENT to my real and finally! ArXiv API works 100 times of 100.

It will be much MUCH better, if feedparser will use something like that:

from os import environ

# <...>
USER_AGENT = environ.get('PYTHON_FEEDPARSER_USER_AGENT', "feedparser/%s +https://github.com/kurtmckee/feedparser/" % __version__)  # thank you for you joke, I I throw to the garbage myself and my 2 days for running my project that use langchain and ArXiVLoader

[lukasschwab]

@Ar4ikov I believe all currently-released versions of feedparser support specifying the User-Agent header through a named parameter (agent) to feedparser.parse, but — to your point — this package neither overrides the default nor exposes a way to set it.

I think the most robust change is to make the HTTP calls from arxiv (e.g. with requests), then pass the body to feedparser for parsing.

Nonetheless, my testing hasn't shown that updating the user agent makes the tests pass 100% of the time. Still searching, but I'll investigate this angle more.

Update: I published the major version release.

• pypi: arxiv 2.0.0
• Release notes

If you find any issues with the new version unrelated to the API instability, please open separate issues for those! I rolled this release in a hurry.