global power spectra

[nicholasloveday]

Spectral calculations are being widely used at the moment in the evaluation of data-driven models. We should add this capability to scores.

The solution should ideally support the calculation of:

• kinetic power spectrum (e.g., horizontal winds)
• variance spectrum (e.g., temperature)

There are further measures that we should ideally support (https://arxiv.org/abs/2407.06100).

• the transient spectral amplitude ratio
• the spectral coherence

One thing to consider is if there is something in scipy that we can leverage or if we should use numpy.fft instead

[Thomas-Moore-Creative]

One thing to consider is if there is something in scipy that we can leverage or if we should use numpy.fft instead

a possibly naive suggestion is to have a look at xrft:

• https://github.com/xgcm/xrft
• https://xrft.readthedocs.io/en/latest/

I've dabbled with xrft for xarray-ready time series analysis but I note that scores contributor @dougiesquire is a xrft contributor and he might have an informed POV?

[nicholasloveday]

Thanks @Thomas-Moore-Creative - I hadn't come across xrft. It looks useful! Definitely keen to hear @dougiesquire 's view.
I initially just focused on numpy/scipy since we're trying to limit dependencies as much as possible (although it looks like xrft's dependencies are minimal - xarray + dask).

@dougiesquire do you know how actively maintained this package is? I note that it hasn't had a pull request merged in since July last year.

ping @tennlee

[dougiesquire]

Hi @nicholasloveday, I haven't had anything to do with xrft for years, but my impression, like yours, is that it's not very actively maintained. The most recent seven issues for example have not received any replies.

It's straightforward to calculate the spectra and coherence with xrft. I'm not familiar with the transient spectral amplitude ratio, so can't really comment on that.

Not very helpful sorry.

[nicholasloveday]

Thanks @dougiesquire. It sounds to me like we might be best off producing our independent, maintained implementation.

[nikeethr]

I'll preface by saying that - while I have worked on Fourier analysis quite a bit in a past life, I do not have sufficient expertise in the spherical domain.

Despite this, and having lurked at ML forums I cannot confidently say that there is a standard implementation when it comes to PSD of spherical harmonics. I imagine this is not a trivial thing - especially on a global scale - it begs for some approximations.

Fourier transforms are generally used in Euclidian coords e.g. $R^N$, not polar/spherical. They can be approximated locally, but at a global scale taking an fft is bound to require more drastic approximations. I am not knowledgeable of whether the propositions in measurement theory by which Fourier analysis is partially based on, are transferable to spherical coordinates. My intuition says however, that this is not a trivial exercise. Further, spherical harmonic analysis from what I understand is commonly addressed by deriving orthonormal bases, wavelets or utilizing discrete cosine transform (DCT) and the like.

I am not implying that current implementations are inaccurate perse (conversely, I am not sure that there is sufficient evidence of their accuracy either - as far as ML literature is concerned anyway). Rather, its that I can't see a clear standard/consistent implementation in ML literature or otherwise. Consistency is paramount when doing any sort of comparison.

Having said that one of the common denominators/foundational implementations I often see is:
Hauser, M G, & Peebles, P J.E. (1973). Statistical analysis of catalogs of extragalactic objects. II. The Abell catalog of rich clusters. Astrophys. J., v. 185, no. 3, pp. 757-785. (actually has 12 parts?, this is just one of them)

I haven't looked closely at xrft but the code does exhibit certain keywords in this paper. There maybe more up to date research and implementation, but whatever we decide it has to be done with some rigour.

[nikeethr]

Some literature on algorithms/implementations/approximations/speed:

• https://arxiv.org/pdf/1202.6522
• https://www.sciencedirect.com/science/article/pii/S0196885884710086
• https://www.sciencedirect.com/science/article/pii/S0377042703005466

[dougiesquire]

Years ago, when I was first learning Python, I wrote some xarray/dask wrappers for pyspharm: https://github.com/dougiesquire/xspharm.

They may be useful, though, as mentioned, they were written when my Python was even worse than it is now...

[dougiesquire]

Looks like I even included a notebook calculating the power spectra: https://github.com/dougiesquire/xspharm/blob/master/notebooks%2Fpyspharm.ipynb