Mongo does not contain contract data for all contract prices stored in arctic

[TobiasAntiGravity]

Hi

Hope you all are doing well!

I am working on compilihng a new database. I have run the system in paper trading mode. Looking into the db I find the following; there are 3031 symbols in contract prices in arctic. In futures_contracts in mongo, however - there are only 836 documents.

example of ALUMINIUM instrument futures_contract documents;

['ALUMINIUM.202204', 'ALUMINIUM.202202', 'ALUMINIUM.202206', 'ALUMINIUM.202201', 'ALUMINIUM.202203', 'ALUMINIUM.202207', 'ALUMINIUM.202205', 'ALUMINIUM.202208', 'ALUMINIUM.202209']

example of ALUMINIUM instrument contract prices symbols from arctic:

['ALUMINIUM.20210400', 'ALUMINIUM.20210500', 'ALUMINIUM.20210600', 'ALUMINIUM.20210700', 'ALUMINIUM.20210800', 'ALUMINIUM.20210900', 'ALUMINIUM.20211000', 'ALUMINIUM.20211100', 'ALUMINIUM.20211200', 'ALUMINIUM.20220100', 'ALUMINIUM.20220200', 'ALUMINIUM.20220300', 'ALUMINIUM.20220400', 'ALUMINIUM.20220500', 'ALUMINIUM.20220600', 'ALUMINIUM.20220700', 'ALUMINIUM.20220800', 'ALUMINIUM.20220900', 'ALUMINIUM.20221000', 'ALUMINIUM.20221100', 'ALUMINIUM.20221200', 'ALUMINIUM.20230100', 'ALUMINIUM.20230200', 'ALUMINIUM.20230300', 'ALUMINIUM.20230400', 'ALUMINIUM.20230500', 'ALUMINIUM.20230600', 'ALUMINIUM.20230700', 'ALUMINIUM.20230800', 'ALUMINIUM.20230900', 'ALUMINIUM.20231000', 'ALUMINIUM.20231100', 'ALUMINIUM.20231200', 'ALUMINIUM.20240100', 'ALUMINIUM.20240200', 'ALUMINIUM.20240300', 'ALUMINIUM.20240400', 'ALUMINIUM.20240500', 'ALUMINIUM.20240600', 'ALUMINIUM.20240700', 'ALUMINIUM.20240800', 'ALUMINIUM.20240900', 'ALUMINIUM.20241000', 'ALUMINIUM.20241100', 'ALUMINIUM.20241200', 'ALUMINIUM.20250100', 'ALUMINIUM.20250200', 'ALUMINIUM.20250300', 'ALUMINIUM.20250400', 'ALUMINIUM.20250500', 'ALUMINIUM.20250600', 'ALUMINIUM.20250700', 'ALUMINIUM.20250800', 'ALUMINIUM.20250900', 'ALUMINIUM.20251000', 'ALUMINIUM.20251100', 'ALUMINIUM.20251200', 'ALUMINIUM.20260100', 'ALUMINIUM.20260200', 'ALUMINIUM.20260300', 'ALUMINIUM.20260400', 'ALUMINIUM.20260500', 'ALUMINIUM.20260600', 'ALUMINIUM.20260700', 'ALUMINIUM.20260800', 'ALUMINIUM.20260900', 'ALUMINIUM.20261000', 'ALUMINIUM.20261100', 'ALUMINIUM.20261200', 'ALUMINIUM.20270100', 'ALUMINIUM.20270200', 'ALUMINIUM.20270300']

So the question I have is quite simply; Is this to be expected - or have I managed destroy the database integrity somehow?

An aside;
I am migrating the data to a new production db now. My plan is to create the mongo futures_contracts documents that are missing (given that the contract price is non empty). Does this sound like ok idea, or is it bad for some reason I am not seeing?

[robcarver17]

Tha's to be expected. futures_contracts will only contain contracts that you are expected to get live prices for, or have in the past. contract_prices contains all your backfilled futures data.

"My plan is to create the mongo futures_contracts documents that are missing (given that the contract price is non empty). Does this sound like ok idea, or is it bad for some reason I am not seeing?"

You can but it will be a complete waste of time and will have zero effect.

[robcarver17]

Not even that. futures_contracts is only used to see what contracts are required to be currently sampled (and it stores the expiry dates as well FWIW. If you add say 202112, then it will immediately get flagged as 'not sampling anymore' as it's expired.

[TobiasAntiGravity]

The reason I ended up here is; I am fixing rolls (some instruments should have been rolled in the past) - but I could not create a new multiple prices series, based on an amended roll calendar, because contract prices were missing (even though they existed in arctic, they did not exist in mongo).

At least that was my understanding of the problem - maybe my hypothesis about what was wrong, was wrong

[robcarver17]

You had contract_prices missing, maybe so, but that has nothing to do with futures_contracts

[robcarver17]

futures_contracts does not contain any price data

[TobiasAntiGravity]

ah - I presumed that the code was looking at the mongo overview of which prices were present, and then going to the actual arctic prices series. I was thinking about it like a relational database, but this, of course, is not. Lesson learned.

I should have noticed "only used to see what contracts are required to be currently sampled" my text parser sucks.

Thank you for the input Mr. Carver.