Replies: 1 comment
-
My calculation is essentially
|
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi all. I have two questions related to the following calculation. I have a function
opt(vector, parameter)
that is not a gufunc and takes time on each "atom". Let's say its signature isopt: [["dim1"], []] -> [["dim1"]]
, think of the first argument as the intial vector guess, the second is a parameter and the output is the solution vector to a scipy optimisation routine. xarray'sapply_ufunc
is extremely convenient to run this over a DataArray ofparameter
specs and possibly different initialisationvector
values, mingled as appropriate with each parameter.I obviously need the vectorize as
opt
can only work atomically. Then to make the slow optimisation faster overall, I chunk the inputschunks={"dim2":1,...}
before this line on all exterior dimensions, and finally I.compute
. This is a sketch, I have wrapped all these in classes and it is working fine. The example is essentially: use vectorize and parallelize to the maximum, ie the minimum chunk size of exterior dims.My first question is general.
apply_ufunc
is extremely convenient in taking care of reassembling exterior dimensions. I keep thinking though if there would be more mileage out of using dask tasks and notapply_ufunc
. The reason for saying so is I feel I am abusingapply_ufunc
however convenient it is.My second question is specific. I have a lot of logging for each individual
opt
and the messages come from each chunk as they are made, and the logging stream is impossible to read. Are there ways to mitigate this problem? Ideally I could retrieve each individual opt log messages like the log messages from a celery task, somehow. Or be able to organize them and name them by worker id? Can they be written to the root logger delayed, flushed if an exception is raised? Any examples using any idea out there?Thank you!
Beta Was this translation helpful? Give feedback.
All reactions