help with noob code?

[Web3dGuy]

So I've hacked together something that I thought should work, I could use some help finding where I went wrong.
My goal:
download historic price
download current price
keep historic price and append new prices without redownloading historic
have current live updating candle

currently the solution is to write a temporary csv file with history, then add new lines, read csv into a dataframe, then plot all.
however I feel like I'm feeding data in the wrong way . one problem is, moving averages and boll band should take the close time and close price for input but when I feed anything other than open time into them I get errors. the other thought I had is that I'm handling the indexing the wrong way, but when I try to index using the time key, more errors.

The program runs , drops a csv, the candles update , but are messed up and the opens and closes don't line up.
Then I notice every time the update happens the plot likes to zoom in ? probably due to input problems listed above.

play with my code, tell me all the things I did wrong.

Thanks :)

import pandas as pd
import finplot as fplt
import requests
import json


def get_bars(symbol, interval, limit):
    url = 'https://api.binance.com/api/v1/klines?symbol=' + symbol + '&interval=' + interval + '&limit=' + limit
    data = json.loads(requests.get(url).text)
    cur_data = pd.DataFrame(data)
    cur_data.columns = ['time', 'Open', 'High', 'Low', 'Close', 'Volume', 'Close_time',
                        'qav', 'num_trades', 'taker_base_vol', 'taker_quote_vol', 'ignore']
    cur_data['time'] = pd.to_datetime(cur_data['time'], unit='ms', origin='unix')
    cur_data['Open'] = pd.to_numeric(cur_data["Open"])
    cur_data["Close"] = pd.to_numeric(cur_data["Close"])
    cur_data['High'] = pd.to_numeric(cur_data["High"])
    cur_data['Low'] = pd.to_numeric(cur_data["Low"])
    cur_data['Volume'] = pd.to_numeric(cur_data["Volume"])
    cur_data['Close_time'] = pd.to_datetime(cur_data['Close_time'], unit='ms', origin='unix')
    cur_data['qav'] = pd.to_numeric(cur_data["qav"])
    cur_data['num_trades'] = pd.to_numeric(cur_data["num_trades"])
    cur_data['taker_base_vol'] = pd.to_numeric(cur_data["taker_base_vol"])
    cur_data['taker_quote_vol'] = pd.to_numeric(cur_data["taker_quote_vol"])
    cur_data['ignore'] = pd.to_numeric(cur_data["ignore"])
    cur_data.to_csv('bars.csv', mode='a', header=False, index=False)
    return cur_data


def get_bars_historic(symbol, interval, limit):
    url = 'https://api.binance.com/api/v1/klines?symbol=' + symbol + '&interval=' + interval + '&limit=' + limit
    data = json.loads(requests.get(url).text)
    histdata = pd.DataFrame(data)
    histdata.columns = ['time', 'Open', 'High', 'Low', 'Close', 'Volume', 'Close_time', 'qav', 'num_trades',
                        'taker_base_vol', 'taker_quote_vol', 'ignore']
    histdata['time'] = pd.to_datetime(histdata['time'], unit='ms', origin='unix')
    histdata['Open'] = pd.to_numeric(histdata["Open"])
    histdata["Close"] = pd.to_numeric(histdata["Close"])
    histdata['High'] = pd.to_numeric(histdata["High"])
    histdata['Low'] = pd.to_numeric(histdata["Low"])
    histdata['Volume'] = pd.to_numeric(histdata["Volume"])
    histdata['Close_time'] = pd.to_datetime(histdata['Close_time'], unit='ms', origin='unix')
    histdata['qav'] = pd.to_numeric(histdata["qav"])
    histdata['num_trades'] = pd.to_numeric(histdata["num_trades"])
    histdata['taker_base_vol'] = pd.to_numeric(histdata["taker_base_vol"])
    histdata['taker_quote_vol'] = pd.to_numeric(histdata["taker_quote_vol"])
    histdata['ignore'] = pd.to_numeric(histdata["ignore"])
    histdata.to_csv('bars.csv', header=True, index=False)
    return histdata


def calc_bollinger_bands(df):
    r = df['Close'].rolling(20)
    df['bbh'] = r.mean() + 2*r.std()
    df['bbl'] = r.mean() - 2*r.std()


def update():
    get_cur_data = get_bars('ETHUSDT', '1m', '1')
    current_data = get_cur_data[['time', 'Open', 'Close', 'High', 'Low', 'Volume', 'Close_time']]
    df_total = pd.read_csv('bars.csv').drop_duplicates('time').reset_index(drop=True)
    df_total['time'] = pd.to_datetime(df_total['time'])
    df_total['Close_time'] = pd.to_datetime(df_total['Close_time'])
    print('current data #########################################')
    print(current_data.to_string())
    print('######################################################')
    print('Historic data ########################################')
    print(historic_data.tail().to_string())
    print('######################################################')
    print('Total data ########################################')
    print(df_total.tail().to_string())
    print('######################################################')
    candlesticks = df_total['time Open Close High Low'.split()]
    calc_bollinger_bands(df_total)
    #volumes = DataStruct['time Open Close Volume'.split()
    bollband_hi = df_total['time bbh'.split()]
    bollband_lo = df_total['time bbl'.split()]
    if not plots:
        global ax, ax2
        fplt.plot(df_total['time'], df_total['Close'].rolling(9).mean(), ax=ax, color='#ff0000', legend='ma-9')
        fplt.plot(df_total['time'], df_total['Close'].rolling(24).mean(), ax=ax, color='#00ff00', legend='ma-24')
        fplt.plot(df_total['time'], df_total['Close'].rolling(100).mean(), ax=ax, color='#0000ff', legend='ma-100')
        fplt.plot(df_total['time'], df_total['Close'].rolling(200).mean(), ax=ax, color='#005555', legend='ma-200')
        plots.append(fplt.candlestick_ochl(candlesticks))
        plots.append(fplt.plot(bollband_hi, color='#4e4ef1'))
        plots.append(fplt.plot(bollband_lo, color='#4e4ef1'))
        #plots.append(fplt.volume_ocv(volumes, ax=ax.overlay()))
        macd = df_total.Close.ewm(span=12).mean() - df_total.Close.ewm(span=26).mean()
        signal = macd.ewm(span=9).mean()
        df_total['macd_diff'] = macd - signal
        fplt.volume_ocv(df_total[['time', 'Open', 'Close', 'macd_diff']], ax=ax2, colorfunc=fplt.strength_colorfilter)
        fplt.plot(macd, ax=ax2, legend='MACD')
        fplt.plot(signal, ax=ax2, legend='Signal')
    else:
        # every time after we just update the data sources on each plot
        plots[0].update_data(df_total)
        plots[1].update_data(bollband_hi)
        plots[2].update_data(bollband_lo)
        #plots[3].update_data(volumes)


fplt.background = '#222222'
fplt.foreground = '#ffffff'
fplt.odd_plot_background = '#222222'
fplt.cross_hair_color = '#00ff00'
fplt.candle_bull_color = '#00ff00'
fplt.candle_bull_body_color = '#007700'
fplt.candle_bear_color = '#ff0000'
fplt.volume_bull_color = fplt.candle_bull_color
fplt.volume_bull_body_color = fplt.candle_bull_body_color
fplt.volume_bear_color = fplt.candle_bear_color
plots = []
# clear the bars.csv
f = open("bars.csv", "w")
f.truncate()
f.close()
ax, ax2 = fplt.create_plot('Realtime ETHUSDT 1m  (Binance)', init_zoom_periods=100, maximize=True, rows=2)
ax.set_visible(crosshair=True, xaxis=True, yaxis=True, xgrid=True, ygrid=True)
get_hist_data = get_bars_historic('ETHUSDT', '1m', '200')
historic_data = get_hist_data[['time', 'Open', 'Close', 'High', 'Low', 'Volume', 'Close_time']]
fplt.timer_callback(update, 5.0)
fplt.show()

[Trembles]

You can get rid of that wall of numeric conversions in your functions with a one liner,

data = [pd.to_numeric(i) for i in data]

And it seems that when you call get_bars() in the update function you aren't doing anything with the returned data? Then you have to open the csv again to get the data you just return with the same function.

And in that save to csv you are saving the same data multiple times?

[Web3dGuy]

A bit of fooling around and I realized the I should be using plots.append(fplt.plot()) instead of fplt.plot() alone. The volume overlay seems to get in the way so I put it on a separate axis. I'm not sure why but if I remove the volume plot all together and adjust the plots updates index values, the zoom problem comes back. one more question , is there an easy way to switch on and off axis visibility? Then i can just turn off volume or MACD via boolean.

working code:

import pandas as pd
import finplot as fplt
import requests
import json


def get_bars(symbol, interval, limit, mode='a', header=False):
    url = 'https://api.binance.com/api/v1/klines?symbol=' + symbol + '&interval=' + interval + '&limit=' + limit
    data = json.loads(requests.get(url).text)
    cur_data = pd.DataFrame(data)
    cur_data.columns = ['Open_time', 'Open', 'High', 'Low', 'Close', 'Volume', 'Close_time',
                        'qav', 'num_trades', 'taker_base_vol', 'taker_quote_vol', 'ignore']
    cur_data.to_csv('bars.csv', mode=mode, header=header, index=False)
    return cur_data


def get_bars_historic(symbol, interval, limit):
    return get_bars(symbol, interval, limit, mode='w', header=True)


def calc_bollinger_bands(df):
    r = df['Close'].rolling(20)
    df['bbh'] = r.mean() + 2*r.std()
    df['bbl'] = r.mean() - 2*r.std()


def update():
    get_bars('ETHUSDT', '1m', '1')
    df_total = pd.read_csv('bars.csv').drop_duplicates('Close_time', keep='last').reset_index(drop=True)
    candlesticks = df_total['Open_time Open Close High Low'.split()]
    calc_bollinger_bands(df_total)
    volumes = df_total['Open_time Open Close Volume'.split()]
    bollband_hi = df_total['Open_time bbh'.split()]
    bollband_lo = df_total['Open_time bbl'.split()]
    ma9 = df_total['Close'].rolling(9).mean()
    ma24 = df_total['Close'].rolling(24).mean()
    ma100 = df_total['Close'].rolling(100).mean()
    ma200 = df_total['Close'].rolling(200).mean()
    if not plots:
        global ax, ax2, ax3

        plots.append(fplt.candlestick_ochl(candlesticks))
        plots.append(fplt.plot(bollband_hi, color='#4e4ef1'))
        plots.append(fplt.plot(bollband_lo, color='#4e4ef1'))
        plots.append(fplt.volume_ocv(volumes, ax=ax2))
        plots.append(fplt.plot(ma9,ax=ax, color='#ff0000'))
        plots.append(fplt.plot(ma24,ax=ax, color='#00ff00'))
        plots.append(fplt.plot(ma100,ax=ax, color='#0000ff'))
        plots.append(fplt.plot(ma200,ax=ax, color='#005555'))

        macd = df_total.Close.ewm(span=12).mean() - df_total.Close.ewm(span=26).mean()
        signal = macd.ewm(span=9).mean()
        df_total['macd_diff'] = macd - signal
        fplt.volume_ocv(df_total[['Open_time', 'Open', 'Close', 'macd_diff']], ax=ax3, colorfunc=fplt.strength_colorfilter)
        fplt.plot(macd, ax=ax3, legend='MACD')
        fplt.plot(signal, ax=ax3, legend='Signal')
    else:
        # every time after we just update the data sources on each plot
        plots[0].update_data(candlesticks)
        plots[1].update_data(bollband_hi)
        plots[2].update_data(bollband_lo)
        plots[3].update_data(volumes)
        plots[4].update_data(ma9)
        plots[5].update_data(ma24)
        plots[6].update_data(ma100)
        plots[7].update_data(ma200)
        for item in plots:
            item.update_gfx()

fplt.background = '#222222'
fplt.foreground = '#ffffff'
fplt.odd_plot_background = '#222222'
fplt.cross_hair_color = '#00ff00'
fplt.candle_bull_color = '#00ff00'
fplt.candle_bull_body_color = '#007700'
fplt.candle_bear_color = '#ff0000'
fplt.volume_bull_color = fplt.candle_bull_color
fplt.volume_bull_body_color = fplt.candle_bull_body_color
fplt.volume_bear_color = fplt.candle_bear_color
fplt.right_margin_candles = 0

plots = []
# clear the bars.csv
f = open("bars.csv", "w")
f.truncate()
f.close()

ax, ax2, ax3 = fplt.create_plot('Realtime ETHUSDT 1m  (Binance)', init_zoom_periods=100, rows=3)
ax.set_visible(crosshair=True, xaxis=True, yaxis=True, xgrid=True, ygrid=True)
ax2, ax3.set_visible(crosshair=True, xaxis=False, yaxis=True, xgrid=False, ygrid=True)
get_bars_historic('ETHUSDT', '1m', '200')
fplt.timer_callback(update, 5.0)

fplt.show()

[highfestiva]

Seems that ax2.hide() is what you're looking for.

[highfestiva]

Thanks for the donation man! Question: I’m thinking of building a bot builder where the user design the algorithm and an optimizer optimizes the parameters and delivers a script for a fee. Do you think there might be a market for such a product if it’s easy to pick an algo that can 10x small amounts in a year?

[Web3dGuy]

Yeah, that seems like a pretty good idea, I'd probably put it to use. Normally my strategy is based off of simple stuff like the 100 and 200 moving average along with RSI. For example if the hourly is about to touch off the 200 MA I'll check to see the hourly and 4hr RSI are below at least 40 before assuming a bounce or decent entry. I'll also check for candle compression, and what wave we are on in the trend, if at wave 4 ill start taking some profit and usually be out after wave 5, so some kind of wave counting would be cool. I think it would be cool to have a system where you could for example draw a parallel channel or triangle on a trend and if there is an exit attempt opposite the position by a threshold, consider the RSI and MACD levels on a 2 timeframes before making a decision, and/or send a notification SMS/call for action. Detection of support and resistance areas based on a time area would be cool too, Number of tops bottoms could be counted and considered before doing something stupid. Not sure about letting a bot take control but I am definitely interested in not missing an opportunity, so really smart signals is where I'm at. Regarding the coffee's donation ;) I figured it's well deserved and well.. I'll probably have a few more questions when I get some time to actually look deeper and understand your code, I am also aware that time is not cheap.

[highfestiva]

I use a bot which is very trivial, but was able to 13x in a month, which I thought was really good. But it lost it equally quickly. I think in the long run I prefer it doing something stupid from time to time, rather than me making stupid, emotional, irrational decisions all the time. :) I guess most people trading are much better at that, so I'm probably not catering to the bot peoples. What profits are you able to make on what instruments?

[Web3dGuy]

I've done pretty good with ETH & ADA this year, up all in all, not by crazy amounts. Definitely losing some trades. I don't have the balls to hold for 10x returns. Sometimes the devil gets to me and I play with leverage and lose lol. Did 300% a couple weeks ago on ETH with 20x, woo big win then two days later lose all but 20% by missing the chart for a bit. I play for accumulation these days. Really regret selling my 5 coins a handful of years ago, I mean I made good profit but got 100% out of crypto at $3.5k thinking it needed some downtime before going again.... then it went straight for 10k , facepalm moment. Always hold, this shits going places!

…

On Wed., Mar. 3, 2021, 1:08 a.m. Jonas Byström, ***@***.***> wrote: I use a bot which is very trivial, but was able to 13x in a month, which I thought was really good. But it lost it equally quickly. I think in the long run I prefer it doing something stupid from time to time, rather than me making stupid, emotional, irrational decisions all the time. :) I guess most people trading are much better at that, so I'm probably not catering to the bot peoples. What profits are you able to make on what instruments? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#132 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTE57Y4IB62W6NAM66CEXDTBXVAZANCNFSM4YJOQHNQ> .

[Web3dGuy]

This 200ma bounce on btc is a big deal.

[highfestiva]

Look into futures, so you're able to short and make money when the price drops. My bot was able to make a 60% gain by shorting BTC@57400 with 7x leverage (but only after a ton of time spent backtesting the algorithm on BTC).