Add more stats to replay buffers

[jamartinh]

Hi @ymd-h I was thinking on a useful feature I think could be more or less simple to implement and also very useful for RL.

Adding basic stats inside replay buffers.

E.g. for instance

• An internal counter that stores the accumulated numbed of buffer insertions, including bulk insertions, so that one could query how much samples has been inserted into the buffer besides the size which will freeze at max len.
• A counter on the number of samples queried trough sample method
• Others that at this time I do not know xD

Perhaps many measures that are used that are usually calculated outside can be incorporated without penalizing performance such as sample age, ratio of priority updates and so on.

As an idea maybe, but not sure, a creation_timestamp and modification_timestamp can be incorporated also to any sample in the buffer, not sure on this, just thinking.

These kind of stats will help to track then training stats such as throughput and the ratio between samples_insertions / samples_consumed for instance and also can be combined with gradient_steps for other useful ratios or measures used in RL algos to trace performance and adjust learning.

[ymd-h]

@jamartinh
Thank you for your proposal.
I am happy with your active feedbacks!

These features might be handy and helpful, however, it is low priority for me as long as it only calculates statistics.
These can be calculated outside if users want.

without penalizing performance

Any calculations have impact to some extent, so that it is better to choice worthwhile features to put inside.

---

FYI: DeepMind's Reverb can limit sampling speed by specifying replay ratio, which is more meaningful, but which requires more works.
Right now, I feel working cost is too much for unknown possible needs