Memory Leak cause by plots.update_data()

[JayJoeSeventh]

Hi, thanks for creating finplot. So far, I have a lot of fun using it.
Recently I encounter one issue, that is when I am doing live plotting. Quite a similar code I have with the 'issue 129' below
#129

The thing is, I found that computer RAM will go higher and higher after days.
When I check the size of 'plots' using pympler, it did go higher and higher.
I can't del (delete) the 'plots' object because if I do, the live plotting will stopped.
Any advise on how to solve this issue?

[highfestiva]

There is no code within finplot itself that's hogging RAM, but there might be a leak through bad use of API's, etc. Or pyqtgraph or PyQT is causing the leak. Best would be if you could create a minimal example that leaks a lot of memory fast through auto-generated random data, and perhaps I could use some tools to figure out what's going on.

[JayJoeSeventh]

Thanks for your reply. Below is part of my code (sorry the 'add code' function is not working correctly), where you can see from the print(asizeof(plots)), the size kept increasing gradually, I think that's the reason which causes my laptop RAM to increase over time.
Looking forward to hear from you. Thanks a lot!

import finplot as fplt
import pandas as pd
import time
from pympler.asizeof import asizeof
from datetime import datetime, timedelta, timezone 
import gc 

dateparse = lambda x: datetime.strptime(x, '%Y-%m-%d %H:%M:%S')


def startPlotting(symbol, limit, time):
	plots = []
	w = fplt.foreground = '#eef'
	b = fplt.background = fplt.odd_plot_background = '#242320'
	fplt.cross_hair_color = w + 'a'
	ax1, ax2 = fplt.create_plot(f'BTCUSDT 1m', rows=2, init_zoom_periods=limit-10, maximize=True) 
	fplt.candle_bull_color = fplt.candle_bull_body_color = "#26a69a"
	fplt.candle_bear_color = fplt.candle_bear_body_color = "#ef5350"
	fplt.windows[0].ci.layout.setRowStretchFactor(0, 1)
	fplt.windows[0].ci.layout.setRowStretchFactor(1, 1)

	def update():

		nbars = 800 
		global step 
		if step > (len(df)-nbars): 
			print('==========\nEnd of historical data\n==========')
			exit()
		df_1m_data = df.iloc[0+step:nbars+step, :].copy(deep=True)
		step = step + 1

		# EMAs
		df_1m_data['ema10'] = round(df_1m_data['close'].ewm(span=10, adjust=False).mean(), 1)
		df_1m_data['ema20'] = round(df_1m_data['close'].ewm(span=20, adjust=False).mean(), 1)
		df_1m_data['ema200'] = round(df_1m_data['close'].ewm(span=200, adjust=False).mean(), 1)

		if not plots:
			plots.append(fplt.candlestick_ochl(df_1m_data[['open', 'close', 'high', 'low']][-1*limit-1 : ], ax=ax1))
			# EMAs
			plots.append(fplt.plot(df_1m_data['ema10'][-1*limit-1 : ], ax=ax1, color='#14fc08'))
			plots.append(fplt.plot(df_1m_data['ema20'][-1*limit-1 : ], ax=ax1, color='#fc0808'))
			plots.append(fplt.plot(df_1m_data['ema200'][-1*limit-1 : ], ax=ax1, color='#fcaf08'))

		else:
			plots[0].update_data(df_1m_data[['open', 'close', 'high', 'low']][-1*limit-1 : ], gfx=False) 
			# EMAs
			plots[1].update_data(df_1m_data['ema10'][-1*limit-1 : ], gfx=False)
			plots[2].update_data(df_1m_data['ema20'][-1*limit-1 : ], gfx=False)
			plots[3].update_data(df_1m_data['ema200'][-1*limit-1 : ], gfx=False)

			plots[0].update_gfx()
			plots[1].update_gfx()
			plots[2].update_gfx()
			plots[3].update_gfx()

		del df_1m_data
		gc.collect() # To solve memory leak caused by df_1m_data

		print(asizeof(plots)) # Here you can see the memory leak caused by plots

	update()
	fplt.timer_callback(update, 0)
	fplt.show()


step = 0 
df = pd.read_csv(f'2022_w33_w40.csv', index_col='datetime', parse_dates=True, date_parser=dateparse)
startPlotting("BTCUSDT", 80, "1m")

2022_w33_w40.csv

[highfestiva]

mem-leak.zip

[highfestiva]

Hadn't thought about this scenario. So the default used to be to add incrementally more data, but I hadn't thought about the case where you actually want to remove prior data. In my last commit, the default is to extend the data with the new data supplied (instead of NaNs, which is causing the leak).

Hm. Probably better is simply to let the user a) supply all the data — including new, or b) supply all the new data — excluding old (which should no longer be available in the chart). Principle of least surprise, I suppose.

I'll let you know when I've fixed it.

[highfestiva]

Fix available in e4ab803, allowing updates to shrink data.

[JayJoeSeventh]

This do the trick! Thank you very much, sir. I owe you.