Get top 100 crypto ohlcv (1h, 4h, 1d) as CSV

[GitMutlu]

Maybe you can integrade this script into yours to scan the top 100 crypto and make a watchlist.txt to automate

Get top 100 crypto ohlcv (1h, 4h, 1d) as CSV

import os
import pandas as pd
from binance.client import Client
from datetime import datetime, timedelta

API_KEY = ""
API_SECRET = ""

client = Client(API_KEY, API_SECRET)

symbols = ["BTCUSDT", "ETHUSDT", "ADAUSDT", "XRPUSDT", "SOLUSDT", "AVAXUSDT", "INJUSDT", "BNBUSDT"]

symbols = [
"1INCHUSDT",
"AAVEUSDT",
"ADAUSDT",
"ALGOUSDT",
"APEUSDT",
"APTUSDT",
"ARUSDT",
"ATOMUSDT",
"AVAXUSDT",
"AXSUSDT",
"BALUSDT",
"BCHUSDT",
"BLURUSDT",
"BNBUSDT",
"BTCUSDT",
"BTGUSDT",
"BTTUSDT",
"CAKEUSDT",
"CELRUSDT",
"CHZUSDT",
"CHRUSDT",
"CRVUSDT",
"CVXUSDT",
"DAIUSDT",
"DASHUSDT",
"DCRUSDT",
"DOGEUSDT",
"DOTUSDT",
"ENJUSDT",
"EOSUSDT",
"ETCUSDT",
"ETHUSDT",
"FETUSDT",
"FILUSDT",
"FLOWUSDT",
"FTMUSDT",
"FXSUSDT",
"GALAUSDT",
"GMXUSDT",
"GRTUSDT",
"HBARUSDT",
"HNTUSDT",
"HOTUSDT",
"ICPUSDT",
"IMXUSDT",
"INJUSDT",
"KAVAUSDT",
"KLAYUSDT",
"KSMUSDT",
"LDOUSDT",
"LINKUSDT",
"LRCUSDT",
"LTCUSDT",
"LUNCUSDT",
"MAGICUSDT",
"MANAUSDT",
"MATICUSDT",
"MKRUSDT",
"MINAUSDT",
"NEARUSDT",
"NEXOUSDT",
"OSMOUSDT",
"QNTUSDT",
"REQUSDT",
"RPLUSDT",
"ROSEUSDT",
"RVNUSDT",
"RUNEUSDT",
"SANDUSDT",
"SHIBUSDT",
"SNXUSDT",
"SOLUSDT",
"SSVUSDT",
"STXUSDT",
"SUSHIUSDT",
"TFUELUSDT",
"TONUSDT",
"TRXUSDT",
"TWTUSDT",
"UNIUSDT",
"VETUSDT",
"WAVESUSDT",
"WBTCUSDT",
"XECUSDT",
"XLMUSDT",
"XMRUSDT",
"XRPUSDT",
"ZECUSDT",
"ZILUSDT"
]

timeframes = {
"1h": Client.KLINE_INTERVAL_1HOUR,
"4h": Client.KLINE_INTERVAL_4HOUR,
"1d": Client.KLINE_INTERVAL_1DAY
}

def fetch_ohlcv(symbol, interval, start_str="1 Jan, 2017"):

klines = client.get_historical_klines(symbol, interval, start_str)

df = pd.DataFrame(klines, columns=[
    "Date", "Open", "High", "Low", "Close", "Volume",
    "close_time", "quote_asset_volume", "trades",
    "taker_base", "taker_quote", "ignore"
])

df["Date"] = pd.to_datetime(df["Date"], unit='ms')

df = df[["Date", "Open", "High", "Low", "Close", "Volume"]].astype(float, errors="ignore")

df.sort_values("Date", inplace=True)
df.reset_index(drop=True, inplace=True)

return df

output_folder = "stock-pattern\src\ohlcv"

os.makedirs(output_folder, exist_ok=True)

for symbol in symbols:
base_symbol = symbol.replace("USDT", "").lower()
for tf, interval in timeframes.items():
filename = os.path.join(output_folder, f"{base_symbol}_{tf}.csv")

    print(f"=== Processing: {symbol} - {tf} ===")
    
    if not os.path.exists(filename):
        print(f"File {filename} does not exist. Get full data...")
        df_full = fetch_ohlcv(symbol, interval, start_str="1 Jan, 2017")
        
        if not df_full.empty:
            df_full = df_full.iloc[:-1]  
        
        df_full.to_csv(filename, index=False, sep=',')
        print(f"New file saved as: {filename}")
        
    else:
        print(f"File {filename} exists. Find new candles...")
        df_existing = pd.read_csv(filename, parse_dates=["Date"])
        df_existing.sort_values("Date", inplace=True)
        df_existing.reset_index(drop=True, inplace=True)

        if df_existing.empty:
            print("Existing dataset is empty, please retrieve full data...")
            df_full = fetch_ohlcv(symbol, interval, start_str="1 Jan, 2017")
            
            if not df_full.empty:
                df_full = df_full.iloc[:-1] 
            df_full.to_csv(filename, index=False, sep=',')
            continue

        last_date_in_file = df_existing["Date"].iloc[-1]
        print(f"Last known date {filename}: {last_date_in_file}")

        start_time = last_date_in_file + timedelta(seconds=1)
        start_str = start_time.strftime("%d %b, %Y %H:%M:%S")

        df_new = fetch_ohlcv(symbol, interval, start_str=start_str)

        if df_new.empty:
            print("No new data. Remove the last row (open candle) from the existing data.")
            if not df_existing.empty:
                df_existing = df_existing.iloc[:-1]
            
            df_existing.to_csv(filename, index=False, sep=',')
        else:
            print(f"Number of newly retrieved rows: {len(df_new)}")

            df_combined = pd.concat([df_existing, df_new], ignore_index=True)
            df_combined.drop_duplicates(subset=["Date"], keep="first", inplace=True)
            df_combined.sort_values("Date", inplace=True)
            df_combined.reset_index(drop=True, inplace=True)

            if not df_combined.empty:
                df_combined = df_combined.iloc[:-1]

            df_combined.to_csv(filename, index=False, sep=',')
            print(f"File updated: {filename}")

print("Proces finished")

[BennyThadikaran]

Hey thanks for the script. I've been meaning to accumate stock data for crypto for testing purposes. From what i understand you're accumulating klines from 2017 onwards.

I'll give it a closer look on the weekend and run it.

[GitMutlu]

This script fetches historical OHLCV (Open, High, Low, Close, Volume) data for multiple cryptocurrency trading pairs from Binance, starting from January 1, 2017. It retrieves data for three different timeframes: 1-hour, 4-hour, and 1-day.

How it works:

1. Fetch Data:

• It uses Binance's API to download historical klines (candlestick data).

2. Data Processing:
   • The script creates a pandas DataFrame, keeping only relevant columns (Date, Open, High, Low, Close, Volume).
   • Dates are converted into a readable format.
   • The data is sorted chronologically.
3. Storage:
   • The data is saved in CSV format under stock-pattern/src/ohlcv/, with separate files for each trading pair and timeframe.
   • If the file does not exist, it downloads the full dataset from Binance.
   • If the file exists, it checks for new data and appends it while avoiding duplicate entries.
   • The last row (which may contain an open/incomplete candle) is removed before saving.

The script ensures that the data is always up-to-date by checking the last recorded timestamp and fetching new candles if available.