Incompatible with python 3.12

[clbarnes]

Some dependencies cannot be installed on python 3.12, mainly because those packages do not list their build-time dependencies. Listing here as I hit them:

• compresso
  • fixed by seung-lab/compresso@97d2462
• dracopy
  • pinned to <v2, which doesn't include a pyproject.toml and so does not specify that it needs skbuild
• pysimdjson

[william-silversmith]

Hmm, for compresso it seems that I have released wheels. I think you meant that this is fixed by the checkmark?

Yes, DracoPy needs a new release. Thanks for reminding me.

[clbarnes]

Yes, compresso's fixed, thanks for doing that!

[william-silversmith]

forgot to mention, new DracoPy was released last week!

[clbarnes]

Unfortunately pysimdjson won't install either! Do you have a CI setup where the 3.12 build could be tested?

[william-silversmith]

It looks like there's a PR for that with a workaround while it waits to be merged: TkTech/pysimdjson#113

I think I need to overhaul the CloudVolume tests. They've only been able to be run locally for a while.

[clbarnes]

By these benchmarks, orjson beats pysimdjson by some margin, has wheels up to and including 3.12, and if the wheels don't exist, the build story is "have a rust compiler released in the last year" rather than the nightmare that building C++ packages from source often is.

I can put together a github actions workflow if that would be helpful, if the limiting factor is just travis - but if it's about the test configuration itself I won't be of much use.

[william-silversmith]

We use both orjson and pysimdjson unfortunately... We use orjson for decoding JSON rapidly as a whole, but pysimdjson has special methods for partial decoding that are necessary for managing very large and numerous json files. This is necessary for processing the spatial index, which we use for creating sharded mesh and skeleton types. It saves hours or days of computation.

[william-silversmith]

It's also worth noting that the decoder amounts to about 5% of the compute time. Most of the time is spent marshaling the decoded objects into python objects in my experience.

[william-silversmith]

I posted asking the maintainer of pysimdjson to update their repo. In the meantime, there may be a workaround. I haven't quite gotten this to work yet, but looking at the setup.py file, it seems that if you have cython installed and set BUILD_WITH_CYTHON=1 in your environment, you can force recythonization (which should fix the issue if your cython is up-to-date).

https://github.com/TkTech/pysimdjson/blob/master/setup.py#L32

[william-silversmith]

Looks like py3.12 wheels are available for pysimdjson. https://pypi.org/project/pysimdjson/#files

[william-silversmith]

Afaik, this problem is solved.