Extended initial startup time

[LennyN95]

Running idc after installation takes around 14 seconds to start the first time following a fresh install:

idc  2.14s user 2.18s system 30% cpu 13.934 total

Since the installation includes idc-index-data, which already takes a considerable amount of time to download, I would expect all required data to be loaded and the cli to respond immediately.

• What causes the long initial loading time?
• If this is not a bug, I suggest adding a prompt to inform the user that the first execution may take some time.

[fedorov]

@LennyN95 I am sorry, I missed github notification, and am seeing this just now...

It must be the time needed to load the main index and prior versions index into pandas dataframes https://github.com/ImagingDataCommons/idc-index/blob/main/idc_index/index.py#L67-L83.

The reason we do this is because I thought querying in-memory is faster than from files (but we have not done any benchmarking to confirm this), and also to provide users with the ability to use pandas dataframe operations and not just SQL. In your opinion, is this startup delay too much of a price to pay for this?

[LennyN95]

@fedorov isn't the snippet you shared run every time? Because the issue I describe is observed only after a fresh install and only when running the script the first time.

In your opinion, is this startup delay too much of a price to pay for this?
I think it is fair but since it's not expected and only occurs once, I'd print something to inform the user.

Maybe moving from pandas to polars (pandas alternative, written in rust) could give us some speed-up too.
Right now I'm running our of capacity but when there's some spare time in the future, I'd be happy to support this.

[fedorov]

Because the issue I describe is observed only after a fresh install and only when running the script the first time.

I am not sure then - I think we would need to do some profiling to understand the issue first!