Profile and speed up

[cjacoby]

I did a bunch of experimenting about a month ago, and noticed that the Mux was significantly slowing down my sampling/training process.

Todo:

• Benchmark & analyze where the time is getting lost
• Fix it, if possible. At least understand it.

[bmcfee]

Is your profiling code available somewhere?

[cjacoby]

It was in a work thing; I'll try to refactor it into something sharable in the next couple days. (I'm on the road to Seattle all day today, won't have internet for real till tomorrow)

…

On Thu, Jan 5, 2017, 10:26 Brian McFee ***@***.***> wrote: Is your profiling code available somewhere? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#49 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4t8yHuvCmv81Wh2W84v-Fv3-MzvxlYks5rPTXLgaJpZM4LXA-s> .

[bmcfee]

Ping! We should sync up on this some time. It would be great to put out 1.0 in the near future.

[cjacoby]

I agree. I've been thinking about this. Want to hang about it soon? Free almost all of tomorrow and PT morning Saturday.

…

On Thu, Mar 2, 2017, 09:52 Brian McFee ***@***.***> wrote: Ping! We should sync up on this some time. It would be great to put out 1.0 in the near future. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#49 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA4t8y3otHT6H-XGp0UCKh_E6IezSV2Rks5rhwHJgaJpZM4LXA-s> .

[cjacoby]

Thinking about making another pescadores repo to do this. pescador-benchmarks or something. Thoughts?

[bmcfee]

I did a bunch of experimenting about a month ago, and noticed that the Mux was significantly slowing down my sampling/training process.

Now that we have pytest in place, I threw pytest-profiling at it and plotted the results in snakeviz. It looks like the biggest chunk is actually in random index sampling np.random.choice:

note that the per-time cost is miniscule, but we call it 28604 times, so it adds up.

Otherwise, it looks like time is dominated by the underlying streamers/generators, which is what we'd expect.

[bmcfee]

@ejhumphrey and I just had a chat, and it looks like the rng bottleneck could be easily accelerated by adding a buffering mechanism on top of rng. I'll bench it and report back.

[bmcfee]

The savings here are pretty minimal -- I think it may not be worth implementing.

EDIT: for context, the issue is that muxen often need to change the sampling distribution, eg, whenever a streamer dies or is reactivated. This triggers a cache invalidation for pre-fetched random samples, so we have to reset the pre-fetcher, which leads to wasted sampling. We do get a modest saving (order of tens of milliseconds) in the benchmark, but the added complexity of having an indirection layer may not be worth it.

Similarly, getting clever about when to invalidate the cache is probably a bad idea from a maintainability perspective.

[ejhumphrey]

😞

[bmcfee]

Random thought: @cjacoby are you sure it was mux and not buffer that was slowing down? AFAICT, the mux overhead is relatively minimal, notwithstanding calls out to rng, and depending on your seed pool size, a bunch of vector math. Buffering will typically invoke copies though, and that could for sure be a performance hit.