Interface and 'do'

[jonas-eschle]

Hi, this is an interesting, yet difficult library! While it seems intriguing, I wonder thought about the interface: why is there always a 'do'? Why does the library not just wrap the API directly? Writing do("command") seems quite cumbersome compared to command.

Is that API a future plan?

[jcmgray]

Hi @jonas-eschle, thanks for the issue! Firstly note you can do:

from autoray import numpy as np

x = np.random.uniform(size=(2, 3, 4), like='numpy')
y = np.cos(x)

With regards to the do command, the reasoning is just the following:

1. It makes the abstraction explicit
2. Since do performs a cached look-up of fn this approach means the set of supported functions is everything any library provides and autoray doesn't have to list them in a specific API. The fake numpy object above overrides __getattr__ to get around this.

For example

x = do('random.normal', size=(10), like='numpy')
do('fft.ifftshift', x)
# array([-1.03934721,  0.20498712,  2.19194394,  0.3072858 , -1.27799811,
#        -1.41765394, -0.33233833, -0.24674171,  1.20681194,  0.86175232])

works even though we haven't listed anywhere all the fft routines numpy has.

That being said however I wouldn't be against having explicit functions listed like

tensordot = functools.partial(do, fn='tensordot')
...

if people thought that was a improvement over the two above methods.

[jonas-eschle]

I see, I was indeed not aware of the numpy interface, thanks for pointing me to it! I simply overlooked it

What about the gradients? I have not seen anything mentioned here? Or control-flow in general, cond, while_loop etc?

And is there a possibility to set the backend global (I didn't see this either, just in a context manager)?

[jcmgray]

Gradient-wise, autoray functions very work with autograd, tensorflow, pytorch, jax and probably others, the interface is just not abstracted out yet to a autoray.grad function or something, maybe that would be nice! quimb (which is mainly what this project is used for) does have a general interface to these (e.g. the torch one https://github.com/jcmgray/quimb/blob/01350dbdaf0879c989923455be8195fcc56e12a6/quimb/tensor/optimize.py#L471-L522) that could be maybe be upstreamed at some point.

There's not any support for non-python control-flow since this is not something I have ever used. I suppose it's only necessary for dynamic computations where one is also compiling or taking the gradient? One could just translate them directly to python for numpy, cupy and torch etc but would need to be careful about tracing through these making them accidentally static in certain situations where they would be dynamic for tensorflow / jax.

Adding functions to set the backend permanently could easily be added - but are currently not there.

[jcmgray]

I've added ar.set_backend and ar.get_backend.

[jonas-eschle]

I've added ar.set_backend and ar.get_backend.

Ah good, many thanks!

Gradient-wise, autoray functions very work with autograd, tensorflow, pytorch, jax and probably others, the interface is just not abstracted out yet to a autoray.grad function or something, maybe that would be nice!

Indeed, that would be helpful, but it's not urgent, as we also have abstracted this away currently.

There's not any support for non-python control-flow since this is not something I have ever used.

Indeed, this is for any compiled computation where you want to have a while loop or vectorize or execute a function conditionally only. But it's a new level, and easier to wrap as there are only a few functions.

So in general the library looks quite interesting. Is there any overview which numpy functions are actually supported?

Also, just looked again at the README: I would very strongly encourage to advertise the numpy API! Having a numpy API with switchable backend that works well is a treasure! But the do(...) is really not something anyone would want to use. I can't think of a single use-case where the do(....) syntax would be preferred (except some stuff that is not yet there, as a dynamic extension it is good). I guess you can advertise the library way better with this

[jcmgray]

Is there any overview which numpy functions are actually supported?

It really is any backend function that can be imported, so the entirety of numpy for numpy arrays. With a few caveats:

1. Some functions don't always dispatch on the first argument - e.g. stack and einsum, these need to have custom dispatchers so that autoray knows which arg to automatically infer the backend from.
2. For other libraries, if the library doesn't have the numpy equivalent then its ofc not supported for that backend.
3. For other libraries, if the library syntax doesn't match numpy, (e.g. to an extent tensorflow and torch, to much less of an extent cupy and dask etc.), a specific translator needs to be in place. See for example the torch section of autoray.py.

The idea is that it should find the correct function automatically most of the time, and then we just cover the remaining cases with explicit 'translations' which are very quick to add when they are needed.

I would very strongly encourage to advertise the numpy API!

It is in the readme, just a little down the page. It used to be more prominent and maybe should be made so again...