Memory leak!

[timspainNERSC]

The TOPAZ-ERA5 year-long thermodynamics only run has a memory usage of roughly 2 MB per day (18.5 MB–70-some MB over the course of a 31 day run). This implies something is leaking memory.

[timspainNERSC]

A 128x128 double array has a size of 128 kiB, or 8 arrays per MiB. This implies roughly 16 arrays per day of run time, or one array every 90 minutes, or every 9 timesteps. This implies that it is not an array that is created an leaked every timestep, but either a smaller amount of data every timestep or an array with lower frequency.

[dorchard]

valgrind is a tool that can be used for this (see leak-check argument in the Quick start guide).

[timspainNERSC]

And Valgrind doesn't work on modern MacOS :(

[dorchard]

Oh noes :( I will have another think...

[timspainNERSC]

I've been using std::cerr and Mach task_info.

Based on this stackexchange post

[timspainNERSC]

Most of the memory is leaked in library code.

Half of the apparent leaking occurred when doing whole-array mathematics on ModelArrays. The internet suggests that Eigen is a bit lax at cleaning up some temporary arrays, which the ModelArray maths certainly used. Changing the SlabOcean update to be a per-element calculation remove the leak there, at the cost of some performance.

The majority of the remaining leak occurs when reading the netCDF forcing files. Again the leak is occurring during library calls, so my code is not directly responsible. Refactoring would seem to be impossible at this point, but I can at least reduce the number of calls by only reading the forcing files when I know the values in ERA or TOPAZ change (one per hour and once per day respectively).

It has also been suggested by @draenog and @a-smith-github that the memory is not truly leaking, but is just released memory allocations that haven't yet been cleaned up by the OS.

[einola]

This is still an issue.

I think the problem is with the output. It seems like the model buffers the output and writes out everything at the end. If that's the case, then it makes perfect sense that the memory usage grows. My reasons for guessing this are:

• The diagnostic output file remains very small until the end of the run
• Memory usage grows much faster if the output is frequent. Outputting every timestep, I reached 4 GB after about a month-long run, but with daily outputs, I'm still only at 2 GB after 8 months (approximate numbers and timings).