parcels running on MacOS Sonoma 14.5

[carolaferronato]

Question

Question

Hi everyone, I am trying to run the parcels_tutorial, and I am encountering the following error when executing:

Supporting code/error messages

# pset.execute(
    parcels.AdvectionRK4,  # the kernel (which defines how particles move)
    runtime=timedelta(days=6),  # the total length of the run
    dt=timedelta(minutes=5),  # the timestep of the kernel
    output_file=output_file,
)

# ---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[14], line 1
----> 1 pset.execute(
      2     parcels.AdvectionRK4,  # the kernel (which defines how particles move)
      3     runtime=timedelta(days=6),  # the total length of the run
      4     dt=timedelta(minutes=5),  # the timestep of the kernel
      5     output_file=output_file,
      6 )

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/parcels/particleset.py:1180, in ParticleSet.execute(self, pyfunc, pyfunc_inter, endtime, runtime, dt, output_file, verbose_progress, postIterationCallbacks, callbackdt, delete_cfiles)
   1178 if self._interaction_kernel is None:
   1179     if not skip_kernel:
-> 1180         res = self._kernel.execute(self, endtime=next_time, dt=dt)
   1181         if res == StatusCode.StopAllExecution:
   1182             return StatusCode.StopAllExecution

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/parcels/kernel.py:628, in Kernel.execute(self, pset, endtime, dt)
    626 # Execute the kernel over the particle set
    627 if self.ptype.uses_jit:
--> 628     self.execute_jit(pset, endtime, dt)
    629 else:
    630     self.execute_python(pset, endtime, dt)

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/parcels/kernel.py:589, in Kernel.execute_jit(self, pset, endtime, dt)
    587 fargs += [c_double(f) for f in self.const_args.values()]
    588 particle_data = byref(pset.ctypes_struct)
--> 589 return self._function(c_int(len(pset)), particle_data, c_double(endtime), c_double(dt), *fargs)

AttributeError: 'Kernel' object has no attribute '_function'

I understand that there is a problem in

particleset.py:1180

and in

kernel.py:628

But I am new on this so I am kind of lost :(

[VeckoTheGecko]

Hi @carolaferronato , can you provide the following:

• the output of print(parcels.__version__)
• is this part of a larger script? Would you be able to provide the larger script so we have more context here? I think that this would be related to prior calls of pset.execute() so having full context would be nice here

[carolaferronato]

Hi Nick

the parcels version is 3.1.2

and the script start as this:

example_dataset_folder = parcels.download_example_dataset("MovingEddies_data")

fieldset = parcels.FieldSet.from_parcels(f"{example_dataset_folder}/moving_eddies")

print(fieldset)

fieldset.computeTimeChunk(time=86400.0, dt=86400.0)

plt.pcolormesh(fieldset.U.grid.lon, fieldset.U.grid.lat, fieldset.U.data[0, :, :])
plt.xlabel("Zonal distance [m]")
plt.ylabel("Meridional distance [m]")
plt.colorbar()
plt.show()

pset = parcels.ParticleSet.from_list(
    fieldset=fieldset,  # the fields on which the particles are advected
    pclass=parcels.JITParticle,  # the type of particles (JITParticle or ScipyParticle)
    lon=[3.3e5, 3.3e5],  # a vector of release longitudes
    lat=[1e5, 2.8e5],  # a vector of release latitudes
)

print(pset)
plt.pcolormesh(fieldset.U.grid.lon, fieldset.U.grid.lat, fieldset.U.data[0, :, :])
plt.xlabel("Zonal distance [m]")
plt.ylabel("Meridional distance [m]")
plt.colorbar()

plt.plot(pset.lon, pset.lat, "ko")
plt.show()

output_file = pset.ParticleFile(
    name="EddyParticles.zarr",  # the file name
    outputdt=timedelta(hours=1),  # the time step of the outputs
)

print(output_file)

pset.execute(
    parcels.AdvectionRK4,  # the kernel (which defines how particles move)
    runtime=timedelta(days=6),  # the total length of the run
    dt=timedelta(minutes=5),  # the timestep of the kernel
    output_file=output_file,
)

[carolaferronato]

could be that my problem is related to this?

#1080

[erikvansebille]

Thanks for reporting, @carolaferronato. I haven't seen this error before, but I don't think it relates to #1080 (that is very old and resolved, plus deals with the compilation).

What happens when you change the pclass when you create your pset to ScipyParticle? Do you get the same error?

[carolaferronato]

Hi Erik

I changed to ScipyParticle and now I have another error:

AttributeError                            Traceback (most recent call last)
Cell In[21], line 1
----> 1 pset.execute(
      2     parcels.AdvectionRK4,  # the kernel (which defines how particles move)
      3     runtime=timedelta(days=6),  # the total length of the run
      4     dt=timedelta(minutes=5),  # the timestep of the kernel
      5     output_file=output_file,
      6 )

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/parcels/particleset.py:1218, in ParticleSet.execute(self, pyfunc, pyfunc_inter, endtime, runtime, dt, output_file, verbose_progress, postIterationCallbacks, callbackdt, delete_cfiles)
   1216         output_file.write_latest_locations(self, time)
   1217     else:
-> 1218         output_file.write(self, time_at_startofloop)
   1219 if np.isfinite(outputdt):
   1220     next_output += outputdt * np.sign(dt)

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/parcels/particlefile.py:347, in ParticleFile.write(self, pset, time, indices)
    345             ds[varout] = xr.DataArray(data=data, dims=dims, attrs=attrs[varout])
    346             ds[varout].encoding["chunks"] = self.chunks[0] if self._write_once(var) else self.chunks  # type: ignore[index]
--> 347     ds.to_zarr(self.fname, mode="w")
    348     self._create_new_zarrfile = False
    349 else:
    350     # Either use the store that was provided directly or create a DirectoryStore:

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/xarray/core/dataset.py:2629, in Dataset.to_zarr(self, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, storage_options, zarr_version, zarr_format, write_empty_chunks, chunkmanager_store_kwargs)
   2461 """Write dataset contents to a zarr group.
   2462 
   2463 Zarr chunks are determined in the following way:
   (...)
   2625     The I/O user guide, with more details and examples.
   2626 """
   2627 from xarray.backends.api import to_zarr
-> 2629 return to_zarr(  # type: ignore[call-overload,misc]
   2630     self,
   2631     store=store,
   2632     chunk_store=chunk_store,
   2633     storage_options=storage_options,
   2634     mode=mode,
   2635     synchronizer=synchronizer,
   2636     group=group,
   2637     encoding=encoding,
   2638     compute=compute,
   2639     consolidated=consolidated,
   2640     append_dim=append_dim,
   2641     region=region,
   2642     safe_chunks=safe_chunks,
   2643     zarr_version=zarr_version,
   2644     zarr_format=zarr_format,
   2645     write_empty_chunks=write_empty_chunks,
   2646     chunkmanager_store_kwargs=chunkmanager_store_kwargs,
   2647 )

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/xarray/backends/api.py:2239, in to_zarr(dataset, store, chunk_store, mode, synchronizer, group, encoding, compute, consolidated, append_dim, region, safe_chunks, storage_options, zarr_version, zarr_format, write_empty_chunks, chunkmanager_store_kwargs)
   2237 writer = ArrayWriter()
   2238 # TODO: figure out how to properly handle unlimited_dims
-> 2239 dump_to_store(dataset, zstore, writer, encoding=encoding)
   2240 writes = writer.sync(
   2241     compute=compute, chunkmanager_store_kwargs=chunkmanager_store_kwargs
   2242 )
   2244 if compute:

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/xarray/backends/api.py:1975, in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims)
   1972 if encoder:
   1973     variables, attrs = encoder(variables, attrs)
-> 1975 store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/xarray/backends/zarr.py:1020, in ZarrStore.store(self, variables, attributes, check_encoding_set, writer, unlimited_dims)
   1017 else:
   1018     variables_to_set = variables_encoded
-> 1020 self.set_variables(
   1021     variables_to_set, check_encoding_set, writer, unlimited_dims=unlimited_dims
   1022 )
   1023 if self._consolidate_on_close:
   1024     kwargs = {}

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/xarray/backends/zarr.py:1193, in ZarrStore.set_variables(self, variables, check_encoding_set, writer, unlimited_dims)
   1189         encoded_attrs[DIMENSION_KEY] = dims
   1191     encoding["overwrite"] = True if self._mode == "w" else False
-> 1193     zarr_array = self._create_new_array(
   1194         name=name,
   1195         dtype=dtype,
   1196         shape=shape,
   1197         fill_value=fill_value,
   1198         encoding=encoding,
   1199         attrs=encoded_attrs,
   1200     )
   1202 writer.add(v.data, zarr_array, region)

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/xarray/backends/zarr.py:1088, in ZarrStore._create_new_array(self, name, shape, dtype, fill_value, encoding, attrs)
   1085     else:
   1086         encoding["write_empty_chunks"] = self._write_empty
-> 1088 zarr_array = self.zarr_group.create(
   1089     name,
   1090     shape=shape,
   1091     dtype=dtype,
   1092     fill_value=fill_value,
   1093     **encoding,
   1094 )
   1095 zarr_array = _put_attrs(zarr_array, attrs)
   1096 return zarr_array

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/zarr/hierarchy.py:1060, in Group.create(self, name, **kwargs)
   1057 def create(self, name, **kwargs):
   1058     """Create an array. Keyword arguments as per
   1059     :func:zarr.creation.create."""
-> 1060     return self._write_op(self._create_nosync, name, **kwargs)

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/zarr/hierarchy.py:827, in Group._write_op(self, f, *args, **kwargs)
    824     lock = self._synchronizer[group_meta_key]
    826 with lock:
--> 827     return f(*args, **kwargs)

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/zarr/hierarchy.py:1066, in Group._create_nosync(self, name, **kwargs)
   1064 kwargs.setdefault('synchronizer', self._synchronizer)
   1065 kwargs.setdefault('cache_attrs', self.attrs.cache)
-> 1066 return create(store=self._store, path=path, chunk_store=self._chunk_store,
   1067               **kwargs)

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/zarr/creation.py:169, in create(shape, chunks, dtype, compressor, fill_value, order, store, synchronizer, overwrite, path, chunk_store, filters, cache_metadata, cache_attrs, read_only, object_codec, dimension_separator, write_empty_chunks, zarr_version, meta_array, **kwargs)
    166     path = '/'
    168 # initialize array metadata
--> 169 init_array(store, shape=shape, chunks=chunks, dtype=dtype, compressor=compressor,
    170            fill_value=fill_value, order=order, overwrite=overwrite, path=path,
    171            chunk_store=chunk_store, filters=filters, object_codec=object_codec,
    172            dimension_separator=dimension_separator)
    174 # instantiate array
    175 z = Array(store, path=path, chunk_store=chunk_store, synchronizer=synchronizer,
    176           cache_metadata=cache_metadata, cache_attrs=cache_attrs, read_only=read_only,
    177           write_empty_chunks=write_empty_chunks, meta_array=meta_array)

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/zarr/storage.py:426, in init_array(store, shape, chunks, dtype, compressor, fill_value, order, overwrite, path, chunk_store, filters, object_codec, dimension_separator)
    423 if not compressor:
    424     # compatibility with legacy tests using compressor=[]
    425     compressor = None
--> 426 _init_array_metadata(store, shape=shape, chunks=chunks, dtype=dtype,
    427                      compressor=compressor, fill_value=fill_value,
    428                      order=order, overwrite=overwrite, path=path,
    429                      chunk_store=chunk_store, filters=filters,
    430                      object_codec=object_codec,
    431                      dimension_separator=dimension_separator)

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/zarr/storage.py:499, in _init_array_metadata(store, shape, chunks, dtype, compressor, fill_value, order, overwrite, path, chunk_store, filters, object_codec, dimension_separator)
    497 shape = normalize_shape(shape) + dtype.shape
    498 dtype = dtype.base
--> 499 chunks = normalize_chunks(chunks, shape, dtype.itemsize)
    500 order = normalize_order(order)
    501 fill_value = normalize_fill_value(fill_value, dtype)

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/zarr/util.py:137, in normalize_chunks(chunks, shape, typesize)
    133 # N.B., expect shape already normalized
    134 
    135 # handle auto-chunking
    136 if chunks is None or chunks is True:
--> 137     return guess_chunks(shape, typesize)
    139 # handle no chunking
    140 if chunks is False:

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/zarr/util.py:96, in guess_chunks(shape, typesize)
     92 chunks = np.maximum(np.array(shape, dtype='=f8'), 1)
     94 # Determine the optimal chunk size in bytes using a PyTables expression.
     95 # This is kept as a float.
---> 96 dset_size = np.product(chunks)*typesize
     97 target_size = CHUNK_BASE * (2**np.log10(dset_size/(1024.*1024)))
     99 if target_size > CHUNK_MAX:

File ~/anaconda3/envs/parcels/lib/python3.11/site-packages/numpy/__init__.py:414, in __getattr__(attr)
    411     import numpy.char as char
    412     return char.chararray
--> 414 raise AttributeError("module {!r} has no attribute "
    415                      "{!r}".format(__name__, attr))

AttributeError: module 'numpy' has no attribute 'product'`

I have parcels parcels            3.1.2
xarray                    2025.1.2
zarr                      2.13.3
numpy                     2.2.3

I thinks is not ideal but I replaced

np.product = np.prod

and now it works fine!

[VeckoTheGecko]

zarr 2.13.3

this is quite an old zarr version (Oct 9 2022). If you do conda install zarr<3 you should get 2.18.2 (May 26, 2024) fixing this AttributeError: module 'numpy' has no attribute 'product' problem