Realtime / live plotting help

[Camelket]

I wasnt sure if I should start an issue or a disussion for this kind of thing.
Trying to figure out how to get live plotting to work. I checked the examples, but my brokers API is different, so I need to get new data in a separate process (I also want to save the data to file) and one update might include more than a single tick. For the example ticks are generated.
I am stuck, any pointers are much appreciated.

Issues:

1. when the plot is updating with new ticks only bull part of the candles get drawn
   (anything above open of the candle) and anything below open is invisible, but
   the volume gets updated correctly (changes color).
2. The viewbox isnt updating correctly (cutting of top or bottom of a
   candle sometimes) regardless of direction.

Tried:
1) made sure the generated ticks are as expected
2) checking if the chart2.df is correctly updated with the new ticks
3) only using candlesticks in a single ax plot
4) less and more frequent updates

finplot 1.6
python 3.9.2

This is the minimal example code and can be run as is to see the issues:

import multiprocessing
import finplot as fplt 
import logging
import pandas as pd
from random import uniform, randint
from time import sleep

def generate_tickdata(start="2021-04-07 09:30:00.0000", amount=100, freq="5S"):
    prev_price = uniform(0, 5)
    for x in pd.date_range(start, periods=amount, freq=freq, tz="utc"):
        # wait between new tick
        sleep(0.1)
        if (prev_price - 1.25) > 0:
            price = uniform(prev_price - 0.45, prev_price + 0.2)
        else:
            price = uniform(0, prev_price + 1)
        vol = randint(0, 10000)
        prev_price = price
        yield {"time": [x], "price": [price], "size": [vol]}

def populate_tickdata_queue(qout):
    for x in generate_tickdata():
        ndf = pd.DataFrame(x)
        ndf.set_index("time", inplace=True)
        qout.put(ndf)


class Chart2():

    def __init__(self, qchart, rows):
        self.qchart = qchart
        # not creating the axs here as plotitems arent picklable
        self.axs = None
        self.rows = rows
        # store the plotitems here as name:item
        self.plots = {}
        self.df = pd.DataFrame(columns=["time", "open", "high", "low", "close", "volume"]).set_index("time")
        # store the name:item mapping and other information regarding the plotitem
        # like color, columns that are needed to update the item, ect.
        self.indicators = {}

    def run(self):
        self.process = multiprocessing.Process(target=self.start_chart)
        self.process.start()

    def start_chart(self):
        # wait until there is a tick in the queue 
        while self.qchart.empty():
            sleep(0.1)
        # get the a single tick
        ndf = self.qchart.get()
        try:
            # update self.df with the new tick(s)
            self.update_df(ndf)
        except ValueError:
            pass
        else:
            if not self.plots:
                self.axs = list(fplt.create_plot("test_chart", rows=self.rows, maximize=False))
                # create plotitems for candlesticks and volume
                self.init_chartelement("candles", self.df, ["open", "high", "low", "close"], self.axs[0])
                self.init_chartelement("volume", self.df, ["open", "close", "volume"], self.axs[1])
                fplt.timer_callback(self.update_chart, 0.1)
                fplt.show()

    def update_chart(self):
        if self.qchart.empty():
            return
        else:
            while not self.qchart.empty():
                ndf = self.qchart.get()
                try:
                    self.update_df(ndf)
                except ValueError as e:
                    logging.debug(e)
                else:
                    self.update_all_elements(self.df)
            return

    def refresh_view(self):
        # running fplt 1.6 so native refresh() isnt available yet
        # copy pasted it and added fplt prefix as needed
        for win in fplt.windows:
            vbs = [ax.vb for ax in win.axs]
            for vb in vbs:
                vb.pre_process_data()
            if fplt.viewrestore:
                if fplt._loadwindata(win):
                    continue
        fplt._set_max_zoom(vbs)
        for vb in vbs:
            datasrc = vb.datasrc_or_standalone
            if datasrc and (vb.linkedView(0) is None or vb.linkedView(0).datasrc is None or vb.master_viewbox is not None):
                vb.update_y_zoom(datasrc.init_x0, datasrc.init_x1)
        fplt._repaint_candles()
        for win in fplt.windows:
            fplt._mouse_moved(win, None)

    def init_chartelement(self, indicator, data, columns, ax, color="b", style=None):
        '''initalize a plotitem'''
        if indicator == "candles":
            for x in ["open", "close", "high", "low"]:
                if x not in data.columns:
                    raise AttributeError("missing/misspelled columns for candlestick data")
            chart_ele = fplt.candlestick_ochl(data[columns], ax=ax)
        elif indicator == "volume":
            chart_ele = fplt.volume_ocv(data[columns], ax=ax)
        else:
            chart_ele = fplt.plot(data[columns], ax=ax, color=color, style=style)
        self.plots[indicator] = chart_ele
        self.indicators[indicator] = {"columns": columns, "ax": ax, "color": color, "style": style}


    def update_all_elements(self, new_data=None):
        # update all the plotitems with the new data (updated self.df in this case)
        if new_data is None:
            new_data = self.df
        for indi in self.indicators:
            self.plots[indi].update_data(new_data[self.indicators[indi]["columns"]], gfx=False)
        self.refresh_view()

    def update_df(self, ndf):
        '''ndf is a dataframe with one or more new ticks'''
        if ndf is None:
            raise ValueError
        tempdf = pd.DataFrame(columns=["time", "open", "high", "low", "close", "volume"]).set_index("time")
        # resampling into 1 minute rows
        #  (real application sends more than a single tick in an update)
        resa = ndf.resample("1T")
        tempdf[["open", "high", "low", "close"]] = resa["price"].ohlc().dropna()
        tempdf["volume"] = resa["size"].sum()
        if len(ndf) > 0:
            if len(self.df) == 0:
                self.df = tempdf
                return
            ldrawn = self.df.iloc[-1]  # ldrawn = last tick drawn
            ftick = tempdf.iloc[0]  # ftick = first new tick
            if ldrawn.name == ftick.name:
                if ldrawn.high < ftick.high:
                    self.df.loc[ldrawn.name, "high"] = ftick.high
                if ldrawn.low > ftick.low:
                    self.df.loc[ldrawn.name, "low"] = ftick.low
                self.df.loc[ldrawn.name, "close"] = ftick.close
                self.df.loc[ldrawn.name, "volume"] = ftick["volume"] + self.df.loc[ldrawn.name, "volume"]
                tempdf = tempdf.drop(tempdf.iloc[0].name)
            # if it was a new bar, append it to the main dataframe
            if len(tempdf) > 0:
                self.df = self.df.append(tempdf)
        return


if __name__ == "__main__":

    mgr = multiprocessing.Manager()
    # queue for the ticks to feed the chart
    qchart = mgr.Queue()
    tickdataproc = multiprocessing.Process(target=populate_tickdata_queue, args=(qchart,))
    chart = Chart2(qchart, 2)
    chart.run()
    tickdataproc.start()
    tickdataproc.join()
    chart.process.join()
`

[highfestiva]

@Camelket I appreciate your work, but unfortunately the part about updating the chart in a separate process seems to be the problem. I do not know the inner workings of PyQt, but I think it is meant to be used via the event loop in a polled matter. That is why all examples contain timer callbacks that happen after finplot.show() is called. Good luck!

[Camelket]

Good to hear. Always open for any critique that helps improve my code :)

[Camelket]

Do you have a "should be added/ reworked/ looked over next"-list that I could take a look at?

[highfestiva]

Not really, it seems easy to produce bugs in realtime plots, so perhaps that needs some rework, but not sure how. Also, the main goal of the lib is to support backtesting, which has nothing to do with realtime stuff, so make sure to never sacrifice simplicity of use. You can look at some of the closed issues to get a hang of what's causing concern. Also seems like Mac users are having problems, you wouldn't happen to have one? Mine is ancient, and probably won't run any more. :-/

[Camelket]

Never had a mac saddly. I will definitly try to keep the backtesting approach in mind, what kind of backtesting is your endgoal though ? how do you integrate your library into backtesting? (do you mainly use it as a visual guide to study patterns/indicators and manually derive data from that ?)

[highfestiva]

Yeah, and when the idea has been implemented, I use it to plot order positions, taken orders, wallet development, liquidations, and so forth. I.e. static graphs.