HIRSBestLineFilter.finalise fails on numpy 1.14 with invalid type promotion #75
Description
Realised through FCDR_HIRS but the problem lies in typhon. On numpy 1.13:
$ plot_hirs_field_timeseries --plot_noise_with_other --without_gain --without_rself --temp_fields --count_fields space --with_corr above --corr_count 6 --corr_timeres 30min --sat noaa15 --channel 1 --from_date 1999-01-01 --to_date 1999-01-2
WARNING 2017-12-18 11:00:34,532 registry._define_single_adder:324: Redefining 'micro' (<class
'pint.definitions.PrefixDefinition'>)
WARNING 2017-12-18 11:00:34,532 registry._define_single_adder:324: Redefining 'µ' (<class 'pin
t.definitions.PrefixDefinition'>)
INFO 2017-12-18 11:00:36,346 dataset.read_period:373: Reading fcdr_hirs {'satname': 'noaa15
'} for period 1999-01-01 00:00:00 – 1999-01-02 00:00:00
[ ] N/A% (ETA: --:--:--) /
dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/numpy/ma/core.py:3008: FutureWarning
: Numpy has detected that you (may be) writing to an array returned
by numpy.diagonal or by selecting multiple fields in a structured
array. This code will likely break in a future numpy release --
see numpy.diagonal or arrays.indexing reference docs for details.
The quick fix is to make an explicit copy (e.g., do
arr.diagonal().copy() or arr[['f0','f1']].copy()).
if (obj.__array_interface__["data"][0]
/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/numpy/ma/core.py:3009: FutureWarnin
g: Numpy has detected that you (may be) writing to an array returned
by numpy.diagonal or by selecting multiple fields in a structured
array. This code will likely break in a future numpy release --
see numpy.diagonal or arrays.indexing reference docs for details.
The quick fix is to make an explicit copy (e.g., do
arr.diagonal().copy() or arr[['f0','f1']].copy()).
!= self.__array_interface__["data"][0]):
[=================== ] 27% (ETA: 0:00:11) W
ARNING 2017-12-18 11:00:41,122 filters.filter:151: 1 scanlines are out of order, resorting
WARNING 2017-12-18 11:00:41,132 filters.filter:159: 24 duplicate scanlines (judging from scanl
ine number), removing
[======================================================================] 100% (Time: 0:00:13)
Continues happily beyond this point. But on numpy 1.14, it fails at this point with:
Traceback (most recent call last):
File "/dev/shm/gerrit/venv/stable-3.6/bin/plot_hirs_field_timeseries", line 11, in <module>
load_entry_point('FCDR-HIRS==0.1.15+dev23.g827117e', 'console_scripts', 'plot_hirs_field_ti
meseries')()
File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/FCDR_HIRS/analysis/timeseri
es.py", line 1370, in main
temp_fields=p.temp_fields)
File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/FCDR_HIRS/analysis/timeseri
es.py", line 337, in __init__
NO_CACHE=True, **hrsargs)
File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/utils/cache.py", lin
e 54, in wrapper
return user_function(*args, **kwds)
File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/datasets/dataset.py"
, line 477, in read_period
arr = of.finalise(arr)
File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/datasets/filters.py"
, line 509, in finalise
fields_notclose = {nm for nm in rep.dtype.names
File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/typhon/datasets/filters.py"
, line 513, in <setcomp>
else numpy.isclose(rep[nm][0, ...], rep[nm])
File "/dev/shm/gerrit/venv/stable-3.6/lib/python3.6/site-packages/numpy/core/numeric.py", lin
e 2329, in isclose
dt = multiarray.result_type(y, 1.)
TypeError: invalid type promotion
There may be other places where typhon functionality fails on numpy 1.14. I suspect this is due to me doing something inappropriate to which numpy has become less tolerant, but this remains to be seen.