difference between add_self_loops: bool = True and root_weight: bool = True

[Romain-Nicolle]

Hello everyone, I have a question regarding the documentation of some convolutional layers. I am quite new to pytorch_geometric so sorry if my question is obvious.

Does the 'root_weight' parameter from some convolutional layers (e.g. RGCNConv) is similar to the 'add_self_loops' parameter from some other convolutional layers (e.g. GCNConv) ?
Or are there some minor differences justifying the use of different words ?
It seems to me that 'add_self_loops' is restricted to homogeneous graphs whereas 'root_weight' is for Heterogeneous Graphs maybe ? It seems to me that there isn't a convolutional layer with both arguments so that would make sense.

I want to perform node embeddings for each node on a heterogeneous graph WITHOUT using the node features of the node I am trying to embed, but only using the node features of all the neighbors across all edge_types. So for now, I was only using layers with add_self_loops = False. Can I use other layers with root_weight = False instead ? If yes, then I have more convolutions to try.
And obviously, I need to be sure that there is no self_loops in the edge_index.

That's what I understood from a comment in the issue #561 (#561), but I just want to be sure to not make any mistakes.

Thanks in advance

:-)

[rusty1s]

There is a slight difference between add_self_loops and root_weight. add_self_loops just means that we add self-loops to the graph on the fly, while root_weight refers to a simple skip-connection that learns a distinct weight matrix. As such, with add_self_loops=True, the model cannot really discriminate central node features from features of its neighbors.

[Romain-Nicolle]

Thank you @rusty1s for your fast and precise answer. I understand better the difference between the two and I can see the different applications when it's set to True.
So, just to confirm, when it's set to False, in both cases, the nodes embeddings will only be computed using the features from the neighbors ? The features of the central node will NOT be used at all, am I right ?

In my specific case, I try to predict the central node class using only the features from the neighbors, so I want to be sure that I don't make any mistake while setting root_weight = False .

[rusty1s]

Yes, add_self_loops=False and root_weight=False have identical computation since they both will not include central nodes during message passing.

[Romain-Nicolle]

thank you so much for your answer, that's perfect :-)
I will close the discussion since I have my answer.

[Romain-Nicolle]

Hi @rusty1s. Thank you again for your fast and precise answer last time. I re-open this topic because I have additional questions on the matter.

I spent some time on the github discussions and I found that this problem was similar to what I was trying to do = I want to predict the class of a node using ONLY the features from its neighbors and not using its own features because I have a lot of features missing from my test set. So I use add_self_loops = False or root_weight = False depending on the Graph Convolutional Layer that I am using.

And it works perfectly fine when I use just one Graph Convolutional Layer. But I am note sure that this works when I have at least 2 Graph Convolutional Layers.
Let's admit that I have 10 nodes (see graph below):

• central node A which is connected to nodes B, C and D by bidirected edges
• node B is connected to nodes A, E and F by bidirected edges
• node C is connected to nodes A, G and H by bidirected edges
• node D is connected to nodes A, I and J by bidirected edges

I will denote by h^L($\nu$) the embedding of node $\nu$ at layer L.

At layer 0, the node embeddings h⁰($\nu$) are simply the initial node features, so easy.
At layer 1, the node embeddings h¹($\nu$) depends on the h⁰($i$) of the neighbours for $i\in$ N($\nu$), so:

• h¹($A$) depends on h⁰($B$), h⁰($C$) and h⁰($D$)
• h¹($B$) depends on h⁰($A$), h⁰($E$) and h⁰($F$)
• h¹($C$) depends on h⁰($A$), h⁰($H$) and h⁰($G$)
• and so on ... so at this point it only depends on the features of the neighbors.

At layer 2, the node embeddings h²($\nu$) depends on the h¹($i$) of the neighbours for $i\in$ N($\nu$), so:

• h²($A$) depends on h¹($B$), h¹($C$) and h¹($D$) which in turns depend (among others) on h⁰($A$).
• and so on ...

So the central node will still "see" its own initial features ? At least partially or transformed ? Am I right ? Because each node is its own neighbor at least 2-hops away.
So that means that the weights learnt during the training will reflect that ? That the final node embedding is still dependent on the node's initial features ?

Could I correct this if I put only directed edges going to the nodes in the test set ? That way the initial features (that are missing) will not go back to the test set node during message passing ?
But I fear that if I use only directed edges going to my nodes in the test set (and not going from them), whereas I have mostly directed edges for my training nodes, then my model will poorly generalize.
The graph convolutions that are inductive are still a bit confusing for me. i don't fully understand how the weights matrices are updated to take into account new nodes that were not part of the initial graph.
And I want to be sure that I am not making any mistakes in my architecture by not treating differently training and testing nodes.

Imagine that I train my model on the initial graph containing only the black bidirectional edges. Then, I slightly update my network with the blue node that is on the receiving end of two unidirectional blue edges coming from D and E for example (see graph below). I know that my model will be able to compute a node embedding and do a prediction, but will it be relevant ?

Sorry for the long post and I hope I'm clear.

Thanks in advance :-)

[rusty1s]

Your observation is correct. Even without add_self_loops you can propagate your own node features back to the seed node given you have more than 1 layer. The easiest solution to this problem would be probably to utilize NeighborLoader with the disjoint option, and to simply mask out seed node features:

loader = NeighborLoader(data, num_neighbors, ..., disjoint=True)

for batch in loader:
     batch.x[:batch.batch_size].zero_()
     model(batch.x, batch.edge_index)

This way, you have disjoint subgraphs for every node you want to predict on, and simply zero out all the seed node features.

[Romain-Nicolle]

Ok, thank you @rusty1s for your quick and precise answer once again. I've just read parts of the documentation from the NeighborLoader object and indeed it seems incredible. I should read more about temporal graphs to better understand it, I will check the examples provided.

So, just to be extra sure, if I train a model, for example a GATConv(..., add_self_loops=False, ...) or a SAGEConv(..., root_weight=False, ...) with the

loader = NeighborLoader(data, num_neighbors, subgraph_type='directional', ...., disjoint=True)

for batch in loader:
    batch.x[:batch.batch_size].zero_()
    model(batch.x, batch.edge_index)

then, no matter how many Graph Convolutional layers I use, the final node embeddings will not depend on the initial node embeddings i.e. the initial node features ? That each subgraph generated for each node will only contain the edges pointing to this node, so this node will only be a 'destination' in the edge_index and never a 'source' ? And the weights learn will reflect that ? Obviously I will still mask out the features of each node when I want to predict (be it train or test set).

I guess that masking out the seed node features with a call to .zero_() might make the add_self_loops=False or the root_weight=False unnecessary. It makes senses that you cannot learn weights to multiply a 'message' for a message passing framework when the message is zero.

I am once again really thankful for your quick and precise answer and for the pyg package. I am also absolutely amazed that each time I have a question that I find really specific, there is already a function to implement that, or another topic about that etc.

[rusty1s]

The seed nodes will still potentially be source nodes. However, it does not propagate any features. Another way to do it is to actually discard the appearance of the seed nodes. This is a good idea as well. This would be as simple as

for batch in loader:
    edge_index = batch.edge_index
    edge_index = edge_index[:, edge_index[0] >= batch.batch_size]

Which method to use depends a bit on your application. Zero out node features would mean that a GNN is still aware that a seed node exists (e.g., to learn a propert COUNT feature), while the latter method would completely discard it as source node from the computation graph.

[Romain-Nicolle]

Thanks you so much once again for all your explanations and your quick answer @rusty1s !!
I think I still need the seed nodes to be source nodes without them propagating any features.
I will go back to testing to find the appropriate setting.

Thank you again for everything.
I will close this topic and I hope I will not have any troubles for a long time.

best regards

[Romain-Nicolle]

Hi again @rusty1s, thank you again for your answers last time but I have another question.

I was able to experiment with the NeighborLoader(..., disjoint = True, ...) by modifying the code provided there.

And it works perfectly fine for the validation and testing sets when my model has been previously trained.

But then I tried to use this approach as well during training time.
I wanted to see if I could train my model that way: for each node in each mini-batch, I want the prediction to be based only on the neighbours' features. And not using at all its own iniitial features to compute its final embedding/prediction.
(But still using its features to send a message to its neighbours in the other subgraphs when it isn't the central node itself, because disjoint = True in the NeighborLoader. At least that was my understanding/hope of what was happening.)

But when I do that, the model doesn't train at all. The loss and the validation don't change at each epoch.
Please find below the slightly updated code for the examples that I used:

train_input_nodes = ('paper', data['paper'].train_mask)
val_input_nodes = ('paper', data['paper'].val_mask)
kwargs = {'batch_size': 1024, 'num_workers': 6, 'persistent_workers': True} # num_workers = 6

# comment that line because I will only use these loaders and not hgt_loader
# if not args.use_hgt_loader:
train_loader = NeighborLoader(data, num_neighbors=[10] * 2, shuffle=True, disjoint=True, input_nodes=train_input_nodes, **kwargs)
val_loader = NeighborLoader(data, num_neighbors=[10] * 2, disjoint=True, input_nodes=val_input_nodes, **kwargs)

@torch.no_grad()
def init_params():
    # Initialize lazy parameters via forwarding a single batch to the model:
    batch = next(iter(train_loader))
    batch = batch.to(device, 'edge_index')
    model(batch.x_dict, batch.edge_index_dict)

def train():
    model.train()

    total_examples = total_loss = 0
    for batch in tqdm(train_loader): 
        optimizer.zero_grad()
        batch = batch.to(device, 'edge_index')
        batch_size = batch['paper'].batch_size
		## put the features of the batch to 0
        batch['paper'].x[:batch['paper'].batch_size] = batch['paper'].x[:batch['paper'].batch_size].zero_()
		
        out = model(batch.x_dict, batch.edge_index_dict)['paper'][:batch_size]
        loss = F.cross_entropy(out, batch['paper'].y[:batch_size])
        loss.backward()
        optimizer.step()

        total_examples += batch_size
        total_loss += float(loss) * batch_size

    return total_loss / total_examples
	
@torch.no_grad()
def test(loader):
    model.eval()

    total_examples = total_correct = 0
    for batch in tqdm(loader):
        batch = batch.to(device, 'edge_index')
        batch_size = batch['paper'].batch_size
		## put the features of the batch to 0
        batch['paper'].x[:batch['paper'].batch_size] = batch['paper'].x[:batch['paper'].batch_size].zero_()
		
        out = model(batch.x_dict, batch.edge_index_dict)['paper'][:batch_size]
        pred = out.argmax(dim=-1)

        total_examples += batch_size
        total_correct += int((pred == batch['paper'].y[:batch_size]).sum())

    return total_correct / total_examples

And when I try to train the model:

init_params()  # Initialize parameters.
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

for epoch in range(1, 21):
    loss = train()
    val_acc = test(val_loader)
    print(f'Epoch: {epoch:02d}, Loss: {loss:.4f}, Val: {val_acc:.4f}')

The output looks like something like this:

Epoch: 01, Loss: 5.9103, Val: 0.0008
100%|██████████| 615/615 [00:39<00:00, 15.52it/s]
100%|██████████| 64/64 [00:03<00:00, 18.39it/s]
Epoch: 02, Loss: 5.9103, Val: 0.0007
100%|██████████| 615/615 [00:39<00:00, 15.52it/s]
100%|██████████| 64/64 [00:03<00:00, 18.28it/s]
Epoch: 03, Loss: 5.9103, Val: 0.0008
100%|██████████| 615/615 [00:39<00:00, 15.38it/s]
100%|██████████| 64/64 [00:03<00:00, 17.72it/s]
Epoch: 04, Loss: 5.9103, Val: 0.0008
100%|██████████| 615/615 [00:38<00:00, 15.98it/s]
100%|██████████| 64/64 [00:03<00:00, 17.06it/s]

I do not understand why it does that. I am afraid I might have misunderstood what the disjoint=True did. I also tried to set the subgraph_type argument of the NeighborLoader to 'directional' or 'bidirectional but it didn't change anything.

I thought that maybe I had to set the argument to 'induced' but I got this error message:

ValueError: 'disjoint' sampling not supported for neighbor sampling via 'torch-sparse'. Please install 'pyg-lib' for improved and optimized sampling routines.

Which is a bit strange because the package pyg-lib (0.3.1+pt21cu118) is properly installed. Looking at the source code for the NeighborSampler which is used by the NeighborLoader, it seems to me that the 'induced' option for Heterogeneous Graph is not yet implemented. At line 357:

        if isinstance(seed, dict):  # Heterogeneous sampling:
            # TODO Support induced subgraph sampling in `pyg-lib`.

It's a small detail but I guess the wrong error is raised ?
Do you want me to open a 'bug issue' maybe to adress this specific problem ? At first I got confused about my installation of pyg-lib and I suspect that others might as well.

So what do you think of it ? Would the 'induced' option help me solve my problem ?
Or even with that, it would not be possible to properly train a (>2 GNN layers) model where, for each node, the prediction is made using only the neighbors' features and not the node-on-the-training-set-to-be-predicted-on ?

Sorry for the long post and thanks in advance.

best regards,

[rusty1s]

Are you referring to this line?

        batch['paper'].x[:batch['paper'].batch_size] = batch['paper'].x[:batch['paper'].batch_size].zero_()

I cannot spot an issue with that.

Regarding the error message on disjoint+induced, I just fixed this on master. Long story short, disjoint+induced is not yet supported. I think in your case, just using directional mode is perfectly fine though.

[Romain-Nicolle]

Ok thanks for your quick answer which is extremely reassuring.
I went back to my code and I found the mistake: I had two instances of the model, one called model_1 and the other called model_2. I was training the model_1 while using the optimizer defined for the model_2, or the other way around.

I feel extremely silly but I'm glad that I was able to find it thanks to your answer. You are a life-saver once again.
Thanks a lot :-)

I will close that issue.