Large custom dataset for link prediction

[GianlucaDeStefano]

Hi all,

I have a set of custom datasets that I use for my Link Prediction experiments.
Until now they all have been based on the InMemoryDataset class, but now that I am trying to scale I have to switch to the lower Dataset interface as they don't fit in memory.

The issue is that I am not sure how to implement this without breaking the remaining part of my codebase.
At the moment, a single huge HeteroData object represents the entire dataset. This is very convenient, for example, to obtain the edge_label and edge_label_index of the edges to predict:

data['node', 'edge_1', 'node'].edge_label
data['node', 'edge_1', 'node'].edge_label_index

Switching to a not in memory dataset would break this as since I wouldn't have a single Data object anymore these tensors would be incomplete and since I use them as parameters for the LinkLoader this is a problem.

Is there maybe an example showing how to use the Dataset interface for link prediction purposes?

For my use case what should the def get(self, idx) return?
In this example
it returns an entire graph. Is this always the case to the returned object should change depending on the task?

[rusty1s]

Thanks for the discussion. torch_geometric.data.Dataset should be used for larger datasets (not necessarily larger graphs), meaning that you may have 1 billion small graphs that do not fit into memory. Currently, our solution to enable large-scale graph learning on a single giant graph that does not fit into memory is provided behind the FeatureStore and GraphStore interfaces (see here), and we are actively working on providing implementations for that. Possible solutions, for example, include, storing node features in a database, and accessing them on-the-fly during sampling.

[Xurorx]

Thank you @rusty1s for your active contribution, I am also expecting to see an example using "FeatureStore" and "GraphStore" to enable large-scale graph learning.

[mattivi]

@rusty1s wonderful work on FeatureStore and GraphStore interfaces for giant graphs!

Do you have a skeleton example for implementation and integration with one key-value stores backends and graph db? Will you release examples and further updates in the next releases? Thanks a lot.

[rusty1s]

We are working with several graph database providers to finally bring some examples. For now, we just have an in-memory example: https://github.com/pyg-team/pytorch_geometric/blob/master/torch_geometric/testing/feature_store.py#L10

[YiKangOY]

Hi, I have about millions of small graphs and I am using torch_geometric.data.Dataset. But I have to create a .pt file for each graph, which is very slow when preprocessing the dataset.
May I ask if there is any efficient way to store lots of small graphs with the torch_geometric.data.Dataset?
Thanks!

[rusty1s]

Yes, we just released OnDiskDataset to support this use-case efficiently :)