duplicated indices when slicing dense backed view lead to .to_memory() TypeError #2064
Description
Please make sure these conditions are met
- I have checked that this issue has not already been reported.
- I have confirmed this bug exists on the latest version of anndata.
- (optional) I have confirmed this bug exists on the main branch of anndata.
Report
Thanks for the great package!
I encountered something unexpected and I think it may be considered a bug. If not a bug, then an "enhancement request" :)
Here is what I observe in words:
I want to slice an anndata using indices that may not be in strictly increasing numerical order. This seems to work in almost all cases, but I found a strange edge case where an error is thrown and I think it may be a bug. To recreate the bug, you need to save an anndata with a dense X (not CSR). Then you need to load the file in backed mode. Then you need to slice it (in some order other than strictly numerically increasing) and call .to_memory().
Here is an example that demonstrates the bug.
Code:
import anndata
import numpy as np
ad = anndata.AnnData(X=np.ones((5, 10)))
ad.write('tmp.h5ad')
ad_backed = anndata.read_h5ad('tmp.h5ad', backed='r')
# ad_backed.X is <HDF5 dataset "X": shape (5, 10), type "<f8">
ad_backed[np.array([0, 2, 2])].to_memory()Traceback:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/sfleming/miniconda3/envs/cellarium/lib/python3.10/site-packages/anndata/_core/anndata.py", line 1414, in to_memory
attr = getattr(self, attr_name, None)
File "/Users/sfleming/miniconda3/envs/cellarium/lib/python3.10/site-packages/anndata/_core/anndata.py", line 556, in X
X = _subset(X, (self._oidx, self._vidx))
File "/Users/sfleming/miniconda3/envs/cellarium/lib/python3.10/functools.py", line 889, in wrapper
return dispatch(args[0].__class__)(*args, **kw)
File "/Users/sfleming/miniconda3/envs/cellarium/lib/python3.10/site-packages/anndata/_core/index.py", line 209, in _subset_dataset
return d[tuple(ordered)][tuple(rev_order)]
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/Users/sfleming/miniconda3/envs/cellarium/lib/python3.10/site-packages/h5py/_hl/dataset.py", line 831, in __getitem__
selection = sel.select(self.shape, args, dataset=self)
File "/Users/sfleming/miniconda3/envs/cellarium/lib/python3.10/site-packages/h5py/_hl/selections.py", line 82, in select
return selector.make_selection(args)
File "h5py/_selector.pyx", line 282, in h5py._selector.Selector.make_selection
File "h5py/_selector.pyx", line 215, in h5py._selector.Selector.apply_args
TypeError: Indexing elements must be in increasing orderLots of other combinations work as expected. For example, if the X is a CSR, then this works fine. It also all works in non-backed mode. Example with a sparse X:
Code:
import anndata
import numpy as np
import scipy.sparse as sp
ad = anndata.AnnData(X=sp.csr_matrix(np.ones((5, 10))))
ad.write('tmp2.h5ad')
ad_backed = anndata.read_h5ad('tmp2.h5ad', backed='r')
# ad_backed.X is CSRDataset: backend hdf5, shape (5, 10), data_dtype float64
ad_backed[np.array([0, 2, 2])].to_memory()Output:
/Users/sfleming/miniconda3/envs/cellarium/lib/python3.10/site-packages/anndata/_core/anndata.py:1756: UserWarning: Observation names are not unique. To make them unique, call `.obs_names_make_unique`.
utils.warn_names_duplicates("obs")
AnnData object with n_obs × n_vars = 3 × 10Versions
| Package | Version |
| ------- | ------- |
| anndata | 0.12.1 |
| Dependency | Version |
| ----------------------------- | -------------------- |
| sphinxcontrib-devhelp | 2.0.0 |
| h5py | 3.13.0 |
| ipython | 9.0.0 |
| prompt_toolkit | 3.0.50 |
| stack-data | 0.6.3 |
| wrapt | 1.17.2 |
| hatchling | 1.27.0 |
| numcodecs | 0.15.1 |
| Deprecated | 1.2.18 |
| sphinxcontrib-htmlhelp | 2.1.0 |
| msgpack | 1.1.0 |
| donfig | 0.8.1.post1 |
| six | 1.16.0 |
| PyYAML | 6.0.1 |
| sphinxcontrib-applehelp | 2.0.0 |
| regex | 2023.12.25 (2.5.140) |
| setuptools | 75.8.0 |
| comm | 0.2.3 |
| awkward_cpp | 44 |
| typing_extensions | 4.12.2 |
| zipp | 3.17.0 |
| awkward | 2.7.4 |
| pure_eval | 0.2.3 |
| legacy-api-wrap | 1.4.1 |
| python-dateutil | 2.9.0.post0 |
| sphinxcontrib-qthelp | 2.0.0 |
| MarkupSafe | 3.0.2 |
| pandas | 2.2.3 |
| charset-normalizer | 3.4.1 |
| numpy | 2.2.3 |
| fsspec | 2025.2.0 |
| scipy | 1.15.2 |
| pytz | 2025.1 |
| jedi | 0.19.2 |
| traitlets | 5.14.3 |
| natsort | 8.4.0 |
| asttokens | 3.0.0 |
| executing | 2.2.0 |
| sphinxcontrib-serializinghtml | 2.0.0 |
| session-info2 | 0.2 |
| wcwidth | 0.2.13 |
| ipywidgets | 8.1.7 |
| crc32c | 2.7.1 |
| zarr | 3.1.1 |
| pluggy | 1.5.0 |
| sphinx-jinja2-compat | 0.3.0 |
| Pygments | 2.19.1 |
| sphinxcontrib-jsmath | 1.0.1 |
| parso | 0.8.4 |
| Jinja2 | 3.1.5 |
| decorator | 5.2.1 |
| packaging | 24.2 |
| importlib-metadata | 7.0.1 |
| Component | Info |
| --------- | ----------------------------------------------------- |
| Python | 3.11.11 (main, Dec 11 2024, 10:25:04) [Clang 14.0.6 ] |
| OS | macOS-15.5-arm64-arm-64bit |
| Updated | 2025-07-30 14:36 |