Profiling the memory usage of a function

I'm writing these notes here before they get lost and I have to gather all this information again.

There are a few python libraries for memory profiling, however it seems memory_profiler has a wider functionality, allowing you to run your executable while you watch the memory usage, using decorators, profiling your code line-by-line, etc. It's a third party library, so it has to be manually installed. See more documentation on: Memory Profiler

I took the decorator approach such that we can see at which line the memory usage is blowing up. Unfortunately, we need to update the source code - by adding decorators - to make it work.

I wanted to profile JobSubmitterPoller, so we need to first import memory_profiler in the source code:

+from memory_profiler import profile

and then add the profile decorator to each function that we want to get a memory report back. By default, the report is printed out to sys.stdout, however, if we're running unit tests, we better define a file stream such that we can see that output (only failed unit tests write anything to sys.stdout). Thus, we need to apply more changes to JobSubmitterPoller.py, as following:

...
+    refreshFp = open('refreshCache_stats.log', 'w+')
+    @profile(stream=refreshFp)
     def refreshCache(self):
...

and to the algorithm method that we want to profile as well:

...
+    algorithmFp = open('algorithm_stats.log', 'w+')
     @timeFunction
+    @profile(stream=algorithmFp)
     def algorithm(self, parameters=None):
...

Now we just need to run any unit tests that will call any of these functions, and we'll get those memory log back in the component directory, something like: test/python/WMComponent_t/JobSubmitter_t/algorithm_stats.log

The heaviest JobSubmitter unit test is:

nosetests JobSubmitter_t.py:JobSubmitterTest.testMemoryProfile

NOTE: if your test crashes, those file descriptors will be left open until they are eventually closed :(