Out of memory when using Block.register_op_hook

[shuo-ouyang]

Hi guys, I am using Block.register_op_hook to capture hidden layers' output during inference. However, the GPU memory is gradually increasing and finally causes OOM when I collecting many layers. I found there is a raw pointer declared in mxnet::common::ExecuteMonOutputCallback and does not delete it. I suspect the reason is this unreleased pointer, but the program core dumped after I release this pointer manually.

https://github.com/apache/incubator-mxnet/blob/a6a7ab4e534022080e2ae90d11e96ca58d338dc7/src/common/utils.cc#L103

My questions:

1. What is the reason of OOM?
2. Can we avoid OOM?