Does the backward hook need to be customised?

[Maenansyn]

According to the tutorial, there is only need to define the init and the forward method.

I created a backbone wrapping two ResNets to process two modals in parallel. That looks like the following (somehow the code-tag is not working correctly at least in the preview - first for-loop has 2 lines, second has one) :

Code

def __init__(self, RGBCfg = ResNet_cfg, DepthCfg = ResNet_cfg): super(customisedModel, self).__init__() self.RGBNet = build_backbone(RGBCfg) # ResNet(50) self.DepthNet = build_backbone(DepthCfg) # ResNet(50) if torch.cuda.is_available(): self.device = torch.device('cuda') else: self.device = torch.device('cpu')

@auto_fp16()
def forward(self, x):
r"""
Args:
x = tensor containing a batch of pairs of colour and a depth image (instances, channels (6), height, width)
"""
start = 0
middle = int(x.size()[1]/2)
end = int(x.size()[1])
rgb = torch.zeros((x.size()[0], int(x.size()[1]/2), x.size()[2], x.size()[3]), device=self.device)
depth = torch.zeros((x.size()[0], int(x.size()[1]/2), x.size()[2], x.size()[3]), device=self.device)
for i in range(x.size()[0]):
rgb[i] = x[i][start:middle]
depth[i] = x[i][middle:end]
feature_maps_colour = self.RGBNet(rgb)
feature_maps_depth = self.DepthNet(depth)
feature_maps_fusioned = []
for i in range(len(feature_maps_colour)):
feature_maps_fusioned.append(torch.maximum(feature_maps_colour[i], feature_maps_depth[i])) # or different other fusion methods
feature_maps_fusioned = tuple(feature_maps_fusioned)
return feature_maps_fusioned

Using Faster R-CNN with normal ResNet and the modals separately creates decent results, the feature maps of the parallel ResNets (before fusioned) behave equal to the separate ones and the fusioned version does look a bit different to the other ones due to the fusion, but still looks workable and familiar towards the other ones. Still, when training, I get no detections, loss decreases quickly and accuracy is stagnating around 99.9 and 100%.

samples_per_gpu: 2
learning rate: 0.0025 accordingly (batch-size of 8 -> 0.01)
momentum and decay unchanged (0.9 and 0.0001), optimizer: SGD

loss-config unchanged:
loss_cls=dict(
type='CrossEntropyLoss', use_sigmoid=True, loss_weight=1.0),
loss_bbox=dict(type='L1Loss', loss_weight=1.0))

So, do I need to customise the backward-hook as well or does it work through the customised model correctly automatically because the forward-methods gives "guidance" and is iterated just backwards?