Missing data for bitcoin

[LeGabriel]

Hi, it is a wonderful database,

But I believe that there is missing data for the bitcoin.

For these timestamps and period of missing data.

2014-11-22 00:03:00 - 40.039583 days
2015-05-01 05:59:00 - 30.834028 days
2016-04-01 02:00:00 - 61.000000 days
2016-08-02 20:02:00 - 7.910417 days

Is it normal ? is it just me ?

Thank you a lot for you database, it helps me a lot :)

[saleh-old]

I have my own gathering data, and I am missing data for these days too. It's on Market's side:

https://api-pub.bitfinex.com/v2/candles/trade:1m:tBTCUSD/hist?start=1416614400000&end=1416657540000&sort=1&limit=1440

returns empty:

[]

[LeGabriel]

Thank you !
And is it absent on all market's sides ?

[saleh-old]

What do you mean by "all sides"?

[LeGabriel]

sorry for my english it's not my mother tongue.
I was meaning, can we find the information anywhere else ?

[saleh-old]

No. We need to push their support team to look into it.

[im4kv]

@sullyfischer any updates on this issue?

[saleh-old]

I’m going to create a mix of Bitfinex and Coinbase that doesn’t have missing candle. I’ll offer it for free on my site when it’s ready: Http://jesse-ai.com Subscribe to it to be notified when it’s ready.

…

On Mar 4, 2020, 7:45 PM +0330, Iman Khosravi ***@***.***>, wrote: @sullyfischer any updates on this issue? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[QuentinFAIDIDE]

Here you can see the proportion of holes in this dataset when aggregating the candles in bigger 15minutes ones. imgur link
The red bars are daily proportions of missing candles. When they reach the top, that means that 100% of the days candles are empty.
Clearly, if you take the data after July 2017 (1.510e12 timestamps), holes are small and it's acceptable, they never exceeds 25%.
However, you should be careful not to rely too much on pre-07/17 data, especially on BTCUSD, LTCUSD, or ETHUSD. I don't know if it's liquidity related of something else, but there are lots of days where there are no candles at all.

[saleh-old]

@QuentinFAIDIDE wow you went through a lot of trouble explaining it. thank you

[QuentinFAIDIDE]

@saleh-mir You're welcome, I needed this graph to select the most complete parts anyway, I just figured out it would be useful to you guys as I remembered there was an issue opened for the missing candles.

[saleh-old]

In my framework I'm doing my best to fill missing candles with candles from other exchanges so that backtests won't be broken because of them. I'm trying to end of with a clean data for a few years. Check out the framework if you haven't. You'll probably like it: http://github.com/jesse-ai/jesse

[gister9000]

@saleh-old
Does jessie framework have that feature now?

[saleh-mir]

@gister9000 yes. Now if you import candles for Bitfinex using it, it will use another driver (Coinbase) for those missing period and you don't have to deal with them