to_zarr to s3 with asynchronous=False

[raybellwaves]

I'm using https://aws.amazon.com/batch/ to run multiple jobs of to_zarr. One issue i'm having is with hitting a request rate to s3. with to_zarr("s3://bucket/file.zarr", mode="w")

botocore.exceptions.ClientError: An error occurred (SlowDown) when calling the DeleteObjects operation (reached max retries: 4): Please reduce your request rate.

File “/opt/conda/lib/python3.9/site-packages/s3fs/core.py”, line 1662, in _bulk_delete
await self._call_s3(“delete_objects”, kwargs, Bucket=bucket, Delete=delete_keys) 

I would like to pass the asynchronous=False to the s3fs config. I'm guessing I could do this with backend_kwargs but not sure on the syntax.

For reference, here's the default behaviour or to_zarr when storing to s3. Note I ran this locally and not on AWS batch which may add an extra layer of debugging.

to_zarr.py:

import xarray as xr

ds = xr.tutorial.open_dataset("air_temperature")
ds.to_zarr("s3://bucket/file.zarr", mode="w", consolidated=True)

viztracer to_zarr.py

i.e. I see calls to fsspec/asyn.py

cc. @martindurant

[martindurant]

I would like to pass the asynchronous=False to the s3fs config

I'm afraid this misunderstands the asynchronous argument. The implementation is always asynchronous, and this argument specifies whether the instance will be called from asyn coroutines (True) or from normal code (False), with the latter being the default.

The story around the appropriate values for retries is fraught, trying to find a default value that satisfies most people. For the moment, the retries value is a class attribute, so you could change it using client.run, but not simply by adding to storage_option. In addition, in #537 we discuss the right number of simultaneous requests to dispatch; it's unlikely that in your scenario you are encountering this.

[raybellwaves]

Thanks @martindurant. Am I right in thinking you meant to link an issue from s3fs? fsspec/s3fs#537

[raybellwaves]

From reading https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html#botocore-config

I see

‘’’
retries (dict) --
A dictionary for retry specific configurations. Valid keys are:

'max_attempts' -- An integer representing the maximum number of retry attempts that will be made on a single request. For example, setting this value to 2 will result in the request being retried at most two times after the initial request. Setting this value to 0 will result in no retries ever being attempted on the initial request. If not provided, the number of retries will default to whatever is modeled, which is typically four retries.
‘’’

[martindurant]

PS: better documentation of asyncronous= should go in fsspec, and exposing configuration of retries should go in s3fs, although gcsfs has the same issue (and http doesn't retry at all).

[RichardScottOZ]

Just hit this now - @raybellwaves - did you find what a good rate was?

[raybellwaves]

Had daily netcdf files. Found was well behaved for a month of days. Added sleeps in my code between months

[RichardScottOZ]

Thanks - I was writing with a 100 machine with 4 cpu each cluster in same s3 region.
When I used 50 machines instead was ok.