multiproc⚓︎
Paralellisation via multiprocessing. Limit num. of CPUs used by numpy
to 1.
Pool(NPROC=None)
⚓︎
Initialize a multiprocessing Pool
.
- Uses
pathos/dill
for serialisation. - Provides unified interface for multiprocessing on/off (as a function of NPROC).
There is some overhead associated with the pool creation,
so you likely want to re-use a pool rather than repeatedly creating one.
Consider using functools.partial
to fix kwargs.
Note
In contrast to reading, in-place writing does not work with multiprocessing. This changes with "shared" arrays, but that has not been tested here. By contrast, multi*threading* shares the process memory, but was significantly slower in the tested (pertinent) cases.
Warning
multiprocessing
does not mix with matplotlib
, so ensure func
does not
reference xp.stats.LP_instance
. In fact, func
should not reference xp
at all, because it takes time to serialize.
See example use in mods.QG
and da_methods.ensemble.LETKF
.