Dealing with missing values

[Oisin-M]

What happened?

We have some inconsistent behaviour when dealing with nan inputs that I think we need to try and standardise. Some functions just ignore or impute nans (e.g. see #43) whilst some others do not e.g.

arr1 = np.random.rand(500)
arr1[0] = np.nan
arr2 = np.random.rand(500)
pearson(arr1, arr2)
>> np.float64(nan)

I think this is confusing and would like a consistent treatment across functions. What should we be doing with missing values?

What are the steps to reproduce the bug?

Version

Platform (OS and architecture)

Relevant log output

Accompanying data

No response

Organisation

No response

[oiffrig]

In my opinion, missing values should propagate by default: if any of the input values used for one of the outputs is missing, then the output should be missing as well. But keep it constrained: if we apply the same operation in bulk over some dimensions of the input, only the individual output values corresponding to missing inputs should be affected.

score([[1, 2, 3], [4, 5, nan], [7, 8, 9]], [[0, 3, nan], [1, 4, 7], [2, 5, 8]], axis=1)
# should return something like
[nan, nan, 0.42]

We could then add an argument describing what to do with missing values if needed. For the record, that's what scipy does:

nan_policy: {‘propagate’, ‘omit’, ‘raise’}
Defines how to handle input NaNs.
* propagate: if a NaN is present in the axis slice (e.g. row) along which the statistic is computed, the corresponding entry of the output will be NaN.
* omit: NaNs will be omitted when performing the calculation. If insufficient data remains in the axis slice along which the statistic is computed, the corresponding entry of the output will be NaN.
* raise: if a NaN is present, a ValueError will be raised.

With the default being propagate.

[Oisin-M]

I agree, this would be much better. I am happy to follow scipy's approach laid out in https://docs.scipy.org/doc/scipy/dev/api-dev/nan_policy.html