Merge branch 'main' into zarr-dtype-tests

Author: dcherian

.github/workflows/benchmarks.yml

@@ -15,7 +15,7 @@ jobs:
     runs-on: ubuntu-latest
     env:
       ASV_DIR: "./asv_bench"
-      CONDA_ENV_FILE: ci/requirements/environment.yml
+      CONDA_ENV_FILE: ci/requirements/environment-benchmark.yml
 
     steps:
       # We need the full repo to avoid this issue
@@ -29,7 +29,7 @@ jobs:
         with:
           micromamba-version: "1.5.10-0"
           environment-file: ${{env.CONDA_ENV_FILE}}
-          environment-name: xarray-tests
+          environment-name: xarray-benchmark
           cache-environment: true
           cache-environment-key: "${{runner.os}}-${{runner.arch}}-py${{env.PYTHON_VERSION}}-${{env.TODAY}}-${{hashFiles(env.CONDA_ENV_FILE)}}-benchmark"
           # add "build" because of https://github.com/airspeed-velocity/asv/issues/1385

.pre-commit-config.yaml

@@ -24,24 +24,24 @@ repos:
       - id: rst-inline-touching-normal
       - id: text-unicode-replacement-char
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.12.1
+    rev: v0.12.2
     hooks:
-      - id: ruff-format
-      - id: ruff
+      - id: ruff-check
         args: ["--fix", "--show-fixes"]
+      - id: ruff-format
   - repo: https://github.com/keewis/blackdoc
-    rev: v0.3.9
+    rev: v0.4.1
     hooks:
       - id: blackdoc
         exclude: "generate_aggregations.py"
         additional_dependencies: ["black==24.8.0"]
   - repo: https://github.com/rbubley/mirrors-prettier
-    rev: v3.5.3
+    rev: v3.6.2
     hooks:
       - id: prettier
         args: [--cache-location=.prettier_cache/cache]
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.16.0
+    rev: v1.16.1
     hooks:
       - id: mypy
         # Copied from setup.cfg

HOW_TO_RELEASE.md

@@ -52,6 +52,7 @@ upstream        https://github.com/pydata/xarray (push)
 
 6.  After merging, again ensure your main branch is synced to upstream:
     ```sh
+    git switch main
     git pull upstream main
     ```
 7.  If you have any doubts, run the full test suite one final time!
@@ -98,17 +99,17 @@ upstream        https://github.com/pydata/xarray (push)
 
     ```
 
-12. Commit your changes and push to main again:
+12. Make a PR with these changes and merge it:
 
     ```sh
-    git commit -am 'New whatsnew section'
-    git push upstream main
+    git checkout -b empty-whatsnew-YYYY.MM.X+1
+    git commit -am "empty whatsnew"
+    git push
     ```
 
-    You're done pushing to main!
+    (Note that repo branch restrictions prevent pushing to `main`, so you have to just-self-merge this.)
 
 13. Update the version available on pyodide:
-
     - Open the PyPI page for [Xarray downloads](https://pypi.org/project/xarray/#files)
     - Edit [`pyodide/packages/xarray/meta.yaml`](https://github.com/pyodide/pyodide/blob/main/packages/xarray/meta.yaml) to update the
       - version number
@@ -119,7 +120,6 @@ upstream        https://github.com/pydata/xarray (push)
 14. Issue the release announcement to mailing lists & Twitter (X). For bug fix releases, I
     usually only email xarray@googlegroups.com. For major/feature releases, I will email a broader
     list (no more than once every 3-6 months):
-
     - pydata@googlegroups.com
     - xarray@googlegroups.com
     - numpy-discussion@scipy.org

asv_bench/asv.conf.json

@@ -60,7 +60,7 @@
   // },
   "matrix": {
     "setuptools_scm": [""], // GH6609
-    "numpy": [""],
+    "numpy": ["2.2"],
     "pandas": [""],
     "netcdf4": [""],
     "scipy": [""],

asv_bench/benchmarks/README_CI.md

@@ -115,8 +115,10 @@ To minimize the time required to run the full suite, we trimmed the parameter ma
 ```python
 from . import _skip_slow  # this function is defined in benchmarks.__init__
 
+
 def time_something_slow():
     pass
 
+
 time_something.setup = _skip_slow
 ```

asv_bench/benchmarks/repr.py

@@ -57,3 +57,31 @@ def time_repr(self):
 
     def time_repr_html(self):
         self.da._repr_html_()
+
+
+class ReprPandasRangeIndex:
+    # display a memory-saving pandas.RangeIndex shouldn't trigger memory
+    # expensive conversion into a numpy array
+    def setup(self):
+        index = xr.indexes.PandasIndex(pd.RangeIndex(1_000_000), "x")
+        self.ds = xr.Dataset(coords=xr.Coordinates.from_xindex(index))
+
+    def time_repr(self):
+        repr(self.ds.x)
+
+    def time_repr_html(self):
+        self.ds.x._repr_html_()
+
+
+class ReprXarrayRangeIndex:
+    # display an Xarray RangeIndex shouldn't trigger memory expensive conversion
+    # of its lazy coordinate into a numpy array
+    def setup(self):
+        index = xr.indexes.RangeIndex.arange(1_000_000, dim="x")
+        self.ds = xr.Dataset(coords=xr.Coordinates.from_xindex(index))
+
+    def time_repr(self):
+        repr(self.ds.x)
+
+    def time_repr_html(self):
+        self.ds.x._repr_html_()

ci/requirements/environment-benchmark.yml

@@ -0,0 +1,23 @@
+name: xarray-benchmark
+channels:
+  - conda-forge
+  - nodefaults
+dependencies:
+  - bottleneck
+  - cftime
+  - dask-core
+  - distributed
+  - flox
+  - netcdf4
+  - numba
+  - numbagg
+  - numexpr
+  - numpy>=2.2,<2.3 # https://github.com/numba/numba/issues/10105
+  - opt_einsum
+  - packaging
+  - pandas
+  - pyarrow # pandas raises a deprecation warning without this, breaking doctests
+  - sparse
+  - scipy
+  - toolz
+  - zarr

design_notes/flexible_indexes_notes.md

@@ -97,12 +97,12 @@ The new `indexes` argument of Dataset/DataArray constructors may be used to spec
 ```python
 >>> da = xr.DataArray(
 ...     data=[[275.2, 273.5], [270.8, 278.6]],
-...     dims=('x', 'y'),
+...     dims=("x", "y"),
 ...     coords={
-...         'lat': (('x', 'y'), [[45.6, 46.5], [50.2, 51.6]]),
-...         'lon': (('x', 'y'), [[5.7, 10.5], [6.2, 12.8]]),
+...         "lat": (("x", "y"), [[45.6, 46.5], [50.2, 51.6]]),
+...         "lon": (("x", "y"), [[5.7, 10.5], [6.2, 12.8]]),
 ...     },
-...     indexes={('lat', 'lon'): SpatialIndex},
+...     indexes={("lat", "lon"): SpatialIndex},
 ... )
 <xarray.DataArray (x: 2, y: 2)>
 array([[275.2, 273.5],
@@ -120,7 +120,7 @@ More formally, `indexes` would accept `Mapping[CoordinateNames, IndexSpec]` wher
 Currently index objects like `pandas.MultiIndex` can be passed directly to `coords`, which in this specific case results in the implicit creation of virtual coordinates. With the new `indexes` argument this behavior may become even more confusing than it currently is. For the sake of clarity, it would be appropriate to eventually drop support for this specific behavior and treat any given mapping value given in `coords` as an array that can be wrapped into an Xarray variable, i.e., in the case of a multi-index:
 
 ```python
->>> xr.DataArray([1.0, 2.0], dims='x', coords={'x': midx})
+>>> xr.DataArray([1.0, 2.0], dims="x", coords={"x": midx})
 <xarray.DataArray (x: 2)>
 array([1., 2.])
 Coordinates:
@@ -169,8 +169,8 @@ Like for the indexes, explicit coordinate creation should be preferred over impl
 For example, it is currently possible to pass a `pandas.MultiIndex` object as a coordinate to the Dataset/DataArray constructor:
 
 ```python
->>> midx = pd.MultiIndex.from_arrays([['a', 'b'], [0, 1]], names=['lvl1', 'lvl2'])
->>> da = xr.DataArray([1.0, 2.0], dims='x', coords={'x': midx})
+>>> midx = pd.MultiIndex.from_arrays([["a", "b"], [0, 1]], names=["lvl1", "lvl2"])
+>>> da = xr.DataArray([1.0, 2.0], dims="x", coords={"x": midx})
 >>> da
 <xarray.DataArray (x: 2)>
 array([1., 2.])
@@ -201,7 +201,9 @@ Besides `pandas.MultiIndex`, there may be other situations where we would like t
 The example given here is quite confusing, though: this is not an easily predictable behavior. We could entirely avoid the implicit creation of coordinates, e.g., using a helper function that generates coordinate + index dictionaries that we could then pass directly to the DataArray/Dataset constructor:
 
 ```python
->>> coords_dict, index_dict = create_coords_from_index(midx, dims='x', include_dim_coord=True)
+>>> coords_dict, index_dict = create_coords_from_index(
+...     midx, dims="x", include_dim_coord=True
+... )
 >>> coords_dict
 {'x': <xarray.Variable (x: 2)>
  array([('a', 0), ('b', 1)], dtype=object),
@@ -211,7 +213,7 @@ The example given here is quite confusing, though: this is not an easily predict
  array([0, 1])}
 >>> index_dict
 {('lvl1', 'lvl2'): midx}
->>> xr.DataArray([1.0, 2.0], dims='x', coords=coords_dict, indexes=index_dict)
+>>> xr.DataArray([1.0, 2.0], dims="x", coords=coords_dict, indexes=index_dict)
 <xarray.DataArray (x: 2)>
 array([1., 2.])
 Coordinates:

design_notes/grouper_objects.md

@@ -8,7 +8,7 @@
 I propose the addition of Grouper objects to Xarray's public API so that
 
 ```python
-Dataset.groupby(x=BinGrouper(bins=np.arange(10, 2))))
+Dataset.groupby(x=BinGrouper(bins=np.arange(10, 2)))
 ```
 
 is identical to today's syntax:
@@ -27,7 +27,7 @@ results = []
 for element in unique_labels:
     subset = ds.sel(x=(ds.x == element))  # split
     # subset = ds.where(ds.x == element, drop=True)  # alternative
-    result = subset.mean() # apply
+    result = subset.mean()  # apply
     results.append(result)
 
 xr.concat(results)  # combine
@@ -36,7 +36,7 @@ xr.concat(results)  # combine
 to
 
 ```python
-ds.groupby('x').mean()  # splits, applies, and combines
+ds.groupby("x").mean()  # splits, applies, and combines
 ```
 
 Efficient vectorized implementations of this pattern are implemented in numpy's [`ufunc.at`](https://numpy.org/doc/stable/reference/generated/numpy.ufunc.at.html), [`ufunc.reduceat`](https://numpy.org/doc/stable/reference/generated/numpy.ufunc.reduceat.html), [`numbagg.grouped`](https://github.com/numbagg/numbagg/blob/main/numbagg/grouped.py), [`numpy_groupies`](https://github.com/ml31415/numpy-groupies), and probably more.
@@ -110,11 +110,13 @@ All Grouper objects will subclass from a Grouper object
 ```python
 import abc
 
+
 class Grouper(abc.ABC):
     @abc.abstractmethod
     def factorize(self, by: DataArray):
         raise NotImplementedError
 
+
 class CustomGrouper(Grouper):
     def factorize(self, by: DataArray):
         ...