Error handling / retry

[obra]

Per the airtable API docs:

RATE LIMITS
The API is limited to 5 requests per second per base. If you exceed this rate, you will receive a 429 status code and will need to wait 30 seconds before subsequent requests will succeed.

The official JavaScript client has built-in retry logic.

If you anticipate a higher read volume, we recommend using a caching proxy. This rate limit is the same for all plans and increased limits are not currently available.

Possibly related, but possibly unrelated, I've been seeing occasional Error: The read operation timed out in my GitHub Actions airtable backup jobs.

I don't know if there is already internal retry logic and we're just blowing past the retry count or if we're running into something not currently contemplated by the code.

[simonw]

The tool currently sleeps for 0.2 seconds between requests to try and stay within the 5 seconds per base limit, but it doesn't do anything smarter than that:

airtable-export/airtable_export/cli.py

Lines 74 to 90 in 6e27758

	def all_records(base_id, table, api_key, sleep=0.2):
	first = True
	offset = None
	while first or offset:
	first = False
	url = "https://api.airtable.com/v0/{}/{}".format(base_id, quote(table))
	if offset:
	url += "?" + urlencode({"offset": offset})
	response = httpx.get(
	url, headers={"Authorization": "Bearer {}".format(api_key)}
	)
	response.raise_for_status()
	data = response.json()
	offset = data.get("offset")
	yield from data["records"]
	if offset and sleep:
	time.sleep(sleep)

Ideally it would catch those 429 status codes, delay and then retry - potentially with exponential backoff of some kind.

[simonw]

Also worth noting:

Iteration may timeout due to client inactivity or server restarts. In that case, the client will receive a 422 response with error message LIST_RECORDS_ITERATOR_NOT_AVAILABLE. It may then restart iteration from the beginning.