Comparison between Mesa and other ABM packages updated to Mesa 3

[Tortar]

Dear all,

I would like you to inform that I updated the ABM Frameworks Comparison at https://github.com/JuliaDynamics/ABMFrameworksComparison to Mesa 3 and your work on the third version shows because benchmark have improved quite a bit! I open this discussion to let you know about this and let you improve the Mesa versions further if you find some more performance opportunities I missed during the update

[EwoutH]

Thanks, both for tracking this and informing us about it!

These are some nice improvements indeed:

It's also cool that for larger simulations the gap shrinks (where you generally need performance the most).

If I may, I would challenge you to add a Game of Life comparison. I think we have a good chance to be the the absolute fastest ;).

[Tortar]

Tried for the sake of it :D

In Agents.jl the one here https://juliadynamics.github.io/AgentsExampleZoo.jl/dev/examples/game_of_life_2D_CA/ (actually after JuliaDynamics/Agents.jl#1160) takes 1.5 seconds while the code you linked takes 2.5 seconds on my computer. So close, but still a bit slower :D

[Tortar]

I think Game of Life is a bit "hackable" in the sense that it is not really showing a lot of features of ABMs in general and indeed it is in the subset of Cellular Automata, we already have Schelling for this kind of models, so I think it's okay not to have it, but if you have suggestions on other small ABMs which can show some more independent dimensions apart from those three implemented, feel free to share!

[quaquel]

Thanks a lot for this update. It's great to see the improvements relative to 2.14 (thanks @EwoutH).

It also highlights some places where further improvements would be nice. I had a quick look at the Python code. I think there are a few places where minor changes would result in small improvements (e.g., np.arctan2 vs math.arctan2 because often numpy is slower on floats than the math library). However, the big things seem to be (1) self.model.grid.select_random_empty_cell() which should be easy to fix because Grid has an empties property layer that is currently not used for select_random_empty_cell; and (2) getting the attribute(s) of all agent (i.e., [agent.direction for agent in self.agents]).

Another thing I am curious about is to drop the weakrefs stuff in AgentSet and see how big the overhead of this is. When introduced, it gave about a 20% overhead on our benchmarks. However, with all improvements made elsewhere, the relative overhead might be larger now. To be clear: I am not arguing at all for removing the weakrefs. There are good memory reasons for doing this. However, it is also worthwhile to track the performance impact of it.

[Tortar]

Hi @quaquel, nice write-up, fell free to add this improvements to the benchmarks, I think having a mantainer of Mesa doing that would be very useful for the robustness of the comparison!

[quaquel]

I am in the middle of teaching so will be tricky to find time in the short term, but I will try to at least run a few quick tests.

[quaquel]

for future reference

%%timeit
np.arctan2(x1, x2)

561 ns ± 2.69 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

%%timeit
math.atan2(x1, x2)

38.2 ns ± 0.0835 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

so, for scalars, math is an order of magnitude faster than numpy.