List and load all the datasets in Polaris

[vincenzo-palmacci]

Hello everyone! So I would have a rather naive question, that I sadly can't figure alone.

How do I list all the datasets in Polaris so that I can load them all in an iterative way?

[cwognum]

Hey @vincenzo-palmacci , thanks for asking!

To answer your question, you can use list_datasets() . This will return the artifact IDs, which you can then use to load each of the datasets. Please note that you may want to use the limit and offset parameter to iteratively fetch all datasets.

Something like 👇 (not tested):

import polaris as po
from polaris.hub.client import PolarisHubClient

with PolarisHubClient() as client: 
    datasets = client.list_datasets()

for dataset_id in datasets: 
    dataset = po.load_dataset(dataset_id)

[vincenzo-palmacci]

Thanks for your reply! Does this work also for benchmarks? Like I can use a list_benchmarks() function?

[vincenzo-palmacci]

Apparently the method list_datasets() does not exist

[cwognum]

My bad, it should be client.list_datasets() not po.list_datasets(). I updated my original comment!

And indeed, we have similar functionality for benchmarks! However, since we have more than 100 benchmarks you will need to set the limit and offset parameters!

[cwognum]

I would be curious to learn a little more about your use case!

Would you be willing to share what you're working on because of which you want to iterate over all datasets?

[vincenzo-palmacci]

Just training some ML models :) Many thanks for your help!

[cwognum]

One thing worth noting is that the collection of datasets hosted on Polaris is quite diverse already (and I expect it to get even more diverse quick!).