Creating In-Memory dataset in GCP bucket

[Sanjaykrishnamurthy]

I am reading custom dataset in PyG dataloader from GCP bucket like this. However the proprocessed files get saved in local instead of in remote GCP bucket. How to override the processed_file_names function to have it save in GCP bucket. Here is the snippet of my code.

`
from google.cloud import storage
fs = gcsfs.GCSFileSystem()

# Read item properties, train and test files from Google BUcket.
storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)

class GraphDataset(pyg_data.InMemoryDataset):
    def __init__(self, root, file_name, transform=None, pre_transform=None):
        self.root = root
        self.file_name = file_name
        super().__init__(root, transform, pre_transform)
        with fs.open(os.path.join('gs://storage/', self.processed_paths[0]), 'rb') as f:
            self.data, self.slices = torch.load(f)

    @property
    def raw_file_names(self):
        return [f'{self.file_name}']

    @property
    def processed_file_names(self):
        return [f'{self.file_name}_plane.pt']

    def download(self):
        pass

    def process(self):
        raw_data_file = f'{self.root}/{self.raw_file_names[0]}'
        blob = bucket.blob(raw_data_file)
        sessions = pickle.loads(blob.download_as_string())

        data_list = []
        for session, y in zip(sessions[0], sessions[1]):
            data_list.append(pyg_data.Data(x=x, edge_index=edge_index, y=y))

           data, slices = self.collate(data_list)
                with fs.open(os.path.join('gs://storage/', self.processed_paths[0]), 'wb') as f:
                                  torch.save((data, slices), f)

train_dataset = GraphDataset('user/'+config['user']+'/dataset/'+config['folder_name']+'/model_train_pickle/raw', 'srgnn_train_sess')

`

[rusty1s]

Can you clarify the issue? Please note that self.processed_paths[0] will refer to the path {root}/processed/*.pt. Is this desired in your case?

[Sanjaykrishnamurthy]

So in my GCP bucket, I have my data stored in raw folder. Now I want the processed files to be stored in processed folder in GCP bucket. But when I try to do it, it creates a processed folder in my local. How do I make it store in the the GCP bucket location. I want the self.processed_paths[0] to point to the GCP bucket location. Is it possible?

[rusty1s]

Oh, I see. This is probably because of this line, right? So, one way to resolve this would be to probably create your own/overwrite the _process function in your custom dataset. Alternatively, we can also fix this on our end directly in _process.

[Sanjaykrishnamurthy]

Ok. thanks for the clarrification.
So is this a correct way to override the _process function?

from torch_geometric.data import dataset
def my_custom_process(): makedirs(at_gcp)
dataset._process = my_custom_process

[rusty1s]

You can just do

class MyDataset(InMemoryDataset):
    def _process():
        makedirs(...)
        self.process()

[Sanjaykrishnamurthy]

Thanks 😄