Skip to content

Commit 4825a0c

Browse files
authored
A legendary tale of why we should make pmap default to using CachingPool (#33892)
Once upon a time, there was a young julia user first getting started with parallelism. And she found it fearsomely slow. And so she did investigate, and she did illuminate upon her issue. Her closures, they were being reserialized again and again. And so this young woman, she openned an issue #16345. Lo and behold, a noble soul did come and resolve it, by making the glorious `CachingPool()` in #16808. 3 long years a later this julia user did bravely return to the world of parallism, with many battle worn scars. and once more she did face the demon that is `pmap` over closures. But to her folly, she felt no fear, for she believed the demon to be crippled and chained by the glorious `CachingPool`. Fearlessly, she threw his closure over 2GB of data into the maw of the demon `pmap`. But alas, alas indeed, she was wrong. The demon remained unbound, and it slew her, and slew her again. 100 times did it slay her for 101 items was the user iterating upon. For the glorious chains of the the `CachingPool()` remains unused, left aside in the users tool chest, forgotten.
1 parent 503d5b4 commit 4825a0c

File tree

2 files changed

+5
-3
lines changed

2 files changed

+5
-3
lines changed

NEWS.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,8 @@ New library features
3434
Standard library changes
3535
------------------------
3636

37+
* `pmap` now defaults to using a `CachingPool` ([#33892]).
38+
3739
#### Package Manager
3840

3941
#### LinearAlgebra

stdlib/Distributed/src/pmap.jl

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,8 @@ For multiple collection arguments, apply `f` elementwise.
4040
Note that `f` must be made available to all worker processes; see
4141
[Code Availability and Loading Packages](@ref code-availability) for details.
4242
43-
If a worker pool is not specified, all available workers, i.e., the default worker pool
44-
is used.
43+
If a worker pool is not specified all available workers will be used via a [`CachingPool`](@ref).
44+
4545
4646
By default, `pmap` distributes the computation over all specified workers. To use only the
4747
local process and distribute over tasks, specify `distributed=false`.
@@ -153,7 +153,7 @@ function pmap(f, p::AbstractWorkerPool, c; distributed=true, batch_size=1, on_er
153153
end
154154

155155
pmap(f, p::AbstractWorkerPool, c1, c...; kwargs...) = pmap(a->f(a...), p, zip(c1, c...); kwargs...)
156-
pmap(f, c; kwargs...) = pmap(f, default_worker_pool(), c; kwargs...)
156+
pmap(f, c; kwargs...) = pmap(f, CachingPool(workers()), c; kwargs...)
157157
pmap(f, c1, c...; kwargs...) = pmap(a->f(a...), zip(c1, c...); kwargs...)
158158

159159
function wrap_on_error(f, on_error; capture_data=false)

0 commit comments

Comments
 (0)