Experiences using multiprocessing.Pool?

[shortydutchie]

Hi

I wrapped the functionality used by qsartuna-optimize in a function and call that using python's multiprocessing (using Pool) to run some model builds in parallel. It all seems to work (great!) but I noticed that there is a warning

"/home/xxxx/.venv/env_qsartuna314/lib/python3.10/site-packages/optunaz/objective.py:200: UserWarning: Loky-backed parallel loops cannot be called in a multiprocessing, setting n_jobs=1"

which is likely because children cannot spawn children. Is there a way to suppress that warning eg. by having qsartuna specify njobs=1 at that point?

Thank you!

[lewismervin1]

Hi @shortydutchie, you should be able to set n_jobs in the optimisation settings to 1. Does this fix your issue?

[shortydutchie]

Hi,

Thanks - I added n_jobs=1 to the "settings" section, it may make some difference in that it limits the number of processes (rather than use all cpus available which it did before), so that may be slightly better (I could only test a small job so not entirely sure). However I still get the warning.

Think I understand why - the descriptor calculations in qsartuna are farmed out in parallel through loky - this is fine. Loky uses fork+exec, spawning a fully separate job, whereas multiprocessing uses fork only to create a thread. The two don't go wholly together I suspect.

I think I will have to ditch my multiprocessing.Pool(....), mypool.map(...) wrapper approach and use loky throughout to resolve this. Probably have to do something along the lines of:

import loky
from loky import get_reusable_executor
executor = get_reusable_executor(max_workers=2, timeout=2)
myoutput=executor.map(my_function, [<list_with_args>])

where my_function is the qsartuna-optimize wrapper function.

Right now things seem to work despite the warnings, perhaps not in the most efficient fashion but as long as it runs overnight I'm fine with it...

Thanks!