multi process

[GregKon]

Question:
In case I use yfinance-cache from 2 process (not threads)
Is it safe from corrupt database point of view (or any)?

E.g:
Example 1:
First process update AAPL stock asset
Second in the same time also try update AAPL stock asset

Example 2:
First process update AAPL stock asset
Second in the same time read only AAPL stock asset

/Greg

[ValueRaider]

No, but how can that situation happen? download() drops duplicate symbols.

[GregKon]

I am not sure what you mean. I will try to explain my case

I have application what yfinance_cache.
Application may be run with multiply instances I want extensively use yfinance_cache.
Every instance of application may try to update database of single ticker: e.g.: AAPL

1. I use to have situation that something go wrong and I have to remove all directory, py-yfinance-cache
2. for now I use FileLock() globally whenever access yfinance_cache is used.

However it is not efficient solution.
Potentially I was thinking if I may:
A) Use separate FileLock on every ticker( e.g. for AAPL)

Question: But what about metadata in for exchange? e,g, "exchange-NYQ"

B) Use separate FileLock on exchange? possible? make sense?

C) some internal FileLock() inside yfinance_cache (as my patch) just on modyfi/read database.
But his require much more work by me, I do not decide to do this. Before I do I would like to ask, maybe there is 1 or 2 plaes in source code what I may simply add this FileLock() and use it more efficiently.

[ValueRaider]

Oh multiple instances, yeah that will corrupt cache.

This file is where all ticker file operations happen https://github.com/ValueRaider/yfinance-cache/blob/main/yfinance_cache%2Fyfc_cache_manager.py

YFC already has internal exchange lock
https://github.com/ValueRaider/yfinance-cache/blob/main/yfinance_cache%2Fyfc_time.py#L40
... but is designed to work with team of processes spawned by download() function.
https://github.com/ValueRaider/yfinance-cache/blob/main/yfinance_cache%2Fyfc_multi.py#L56

[GregKon]

I will have a look than, if single file for file operations, than it shall be ok.
and running test is just run: run-tests.sh
there is also make-test.sh

Oh multiple instances, yeah that will corrupt cache.

This file is where all ticker file operations happen https://github.com/ValueRaider/yfinance-cache/blob/main/yfinance_cache%2Fyfc_cache_manager.py

YFC already has internal exchange lock https://github.com/ValueRaider/yfinance-cache/blob/main/yfinance_cache%2Fyfc_time.py#L40 ... but is designed to work with team of processes spawned by download() function. https://github.com/ValueRaider/yfinance-cache/blob/main/yfinance_cache%2Fyfc_multi.py#L56

[ValueRaider]

I've realised this is difficult/impossible because of my dividend logic - it assumes only one update per ticker at a time.

There might be an easier solution: your multiple instances get data from a new local cache e.g. dict of 1d price data, and your server runs YFC periodically to update your cache. Then the cache consistency and parallel access are separate.

[GregKon]

so far I use global lock before every call to API

            with yfinance_cache_filelock():
                import yfinance_cache as yfc

Performance is low.
Propably I will stay with this solution and wait if demand on performance will increase.

1. it is option what you mentioned
2. I will also propably check if option is disable dividends and as much as possible from init.py.