BUG: ABI incompatibilities with `ewah-bool-utils`

[neutrinoceros]

It was informally reported to me that in some contexts¹, there was an ABI incompatibility between yt and ewah-bool-utils, which manifested as an exception, preventing importing yt altogether. I don't have the exact tracebacks but here's the sort of thing we've seen:

ValueError: ewah_bool_utils.ewah_bool_wrap.SparseUnorderedBitmaskSet size changed, may indicate binary incompatibility. Expected 72 from C header, got 48 from PyObject

I suspect this class of problems are caused by ewah-bool-utils being both a compile-time and a runtime dependency and partially written in Cython, meaning that the exact layout of the class SparseUnorderedBitmaskSet that yt is compiled against is generated and may differ from the one that was used to compile ewah-bool-utils itself. In short, compilation and runtime do not necessarily use a single copy of ewah-bool-utils (which is especially true when building in isolation, or when retrieving pre-compiled wheels from PyPI), which may have different ABIs that we do not control...

I don't know yet whether the problem should be addressed in yt or ewah-bool-utils, but I suspect it might be possible to eliminate the build-time dependency relation, and in any case this issue will get much better visibility on this side.

Tentatively marking this as a blocker as the bug is critical to users who encounter it, and the workaround (building both packages from source, without isolation) is far from trivial if one doesn't already know a lot about Python packaging.

Footnotes

1. at least once an old mac-intel (@jordidj) as well as a mac-arm M1 (@brittonsmith) ↩

[neutrinoceros]

I suspect it might be possible to eliminate the build-time dependency relation

Looking more closely I don't think that it's as straight forward as I was hoping.
I was suspecting that we might replace cimport statements with runtime import ones, but this approach cannot work because some C types from ewah are used at compile-time in yt.geometry.particle_oct_container.pyx.
Meanwhile, yt.geometry.particle_oct_container is only ever imported from at runtime in yt, and only two classes from it are actually used (namely ParticleBitmap and ParticleOctreeContainer), so I was hoping maybe they could be moved to ewah, which would allow eliminating ewah as a build time dependency, but alas, these classes heavily relies on other yt modules that are themselves needed at build time...

So my current conclusion is that splitting out ewah-bool-utils as a separate package was probably a mistake and should be reverted; the boundary between it and yt is too blurry to be maintainable in the current state, I think.

for reference, I'm essentially proposing we revert #2711
Thoughts, @matthewturk ?

[matthewturk]

I don't think reverting is the right course of action. I can engage more deeply once I'm back from vacation but I think we can continue as-is although I need to investigate further. I believe I have both encountered and fixed these specific issues in the past.

If need be we can try to adjust the API to minimize issues, which may also be possible.

[neutrinoceros]

Alright, then I'll tentatively assign you, essentially as a signal to myself that this isn't my department any more (I can't work on this unless I'm convinced that the separation can actually be done cleanly and without a huge refactoring effort. At the moment, I am not.)

[neutrinoceros]

(also, please do consider the possibility of temporarily reverting to mitigate an immediate issue if sufficient work force isn't available quickly)