Skip to content

multiproc⚓︎

Paralellisation via multiprocessing. Limit num. of CPUs used by numpy to 1.

Pool(NPROC=None) ⚓︎

Initialize a multiprocessing Pool.

  • Uses pathos/dill for serialisation.
  • Provides unified interface for multiprocessing on/off (as a function of NPROC).

There is some overhead associated with the pool creation, so you likely want to re-use a pool rather than repeatedly creating one. Consider using functools.partial to fix kwargs.

Note

In contrast to reading, in-place writing does not work with multiprocessing. This changes with "shared" arrays, but that has not been tested here. By contrast, multi*threading* shares the process memory, but was significantly slower in the tested (pertinent) cases.

Warning

multiprocessing does not mix with matplotlib, so ensure func does not reference xp.stats.LP_instance. In fact, func should not reference xp at all, because it takes time to serialize.

See example use in mods.QG and da_methods.ensemble.LETKF.