The problem about the prop. stratagy in ViTDet.

[TsingWei]

According to Table 2a and Table 3 in the paper of ViTDet, the bottleneck seems to be an alternative to global attention. Especially in Table 3, the way of bottleneck use less memory and less time than the way of global. However, in the code of ViTDet, the bottleneck seems to be an option instead of an alternative, meaning that the global attn will always be performed.

detectron2/detectron2/modeling/backbone/vit.py

Lines 211 to 230 in 2b98c27

	def forward(self, x):
	shortcut = x
	x = self.norm1(x)
	# Window partition
	if self.window_size > 0:
	H, W = x.shape[1], x.shape[2]
	x, pad_hw = window_partition(x, self.window_size)
	x = self.attn(x)
	# Reverse window partition
	if self.window_size > 0:
	x = window_unpartition(x, self.window_size, pad_hw, (H, W))
	x = shortcut + self.drop_path(x)
	x = x + self.drop_path(self.mlp(self.norm2(x)))
	if self.use_residual_block:
	x = self.residual(x.permute(0, 3, 1, 2)).permute(0, 2, 3, 1)
	return x

Table 3: