Skip to content

Commit e7ad6c2

Browse files
LuBaolujoergroedel
authored andcommitted
iommu/vt-d: Fix incorrect cache invalidation for mm notification
Commit 6bbd42e ("mmu_notifiers: call invalidate_range() when invalidating TLBs") moved the secondary TLB invalidations into the TLB invalidation functions to ensure that all secondary TLB invalidations happen at the same time as the CPU invalidation and added a flush-all type of secondary TLB invalidation for the batched mode, where a range of [0, -1UL) is used to indicates that the range extends to the end of the address space. However, using an end address of -1UL caused an overflow in the Intel IOMMU driver, where the end address was rounded up to the next page. As a result, both the IOTLB and device ATC were not invalidated correctly. Add a flush all helper function and call it when the invalidation range is from 0 to -1UL, ensuring that the entire caches are invalidated correctly. Fixes: 6bbd42e ("mmu_notifiers: call invalidate_range() when invalidating TLBs") Cc: stable@vger.kernel.org Cc: Huang Ying <ying.huang@intel.com> Cc: Alistair Popple <apopple@nvidia.com> Tested-by: Luo Yuzhang <yuzhang.luo@intel.com> # QAT Tested-by: Tony Zhu <tony.zhu@intel.com> # DSA Reviewed-by: Jason Gunthorpe <jgg@nvidia.com> Reviewed-by: Alistair Popple <apopple@nvidia.com> Signed-off-by: Lu Baolu <baolu.lu@linux.intel.com> Link: https://lore.kernel.org/r/20231117090933.75267-1-baolu.lu@linux.intel.com Signed-off-by: Joerg Roedel <jroedel@suse.de>
1 parent 85b80fd commit e7ad6c2

File tree

1 file changed

+26
-0
lines changed

1 file changed

+26
-0
lines changed

drivers/iommu/intel/svm.c

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,13 +216,39 @@ static void intel_flush_svm_range(struct intel_svm *svm, unsigned long address,
216216
rcu_read_unlock();
217217
}
218218

219+
static void intel_flush_svm_all(struct intel_svm *svm)
220+
{
221+
struct device_domain_info *info;
222+
struct intel_svm_dev *sdev;
223+
224+
rcu_read_lock();
225+
list_for_each_entry_rcu(sdev, &svm->devs, list) {
226+
info = dev_iommu_priv_get(sdev->dev);
227+
228+
qi_flush_piotlb(sdev->iommu, sdev->did, svm->pasid, 0, -1UL, 0);
229+
if (info->ats_enabled) {
230+
qi_flush_dev_iotlb_pasid(sdev->iommu, sdev->sid, info->pfsid,
231+
svm->pasid, sdev->qdep,
232+
0, 64 - VTD_PAGE_SHIFT);
233+
quirk_extra_dev_tlb_flush(info, 0, 64 - VTD_PAGE_SHIFT,
234+
svm->pasid, sdev->qdep);
235+
}
236+
}
237+
rcu_read_unlock();
238+
}
239+
219240
/* Pages have been freed at this point */
220241
static void intel_arch_invalidate_secondary_tlbs(struct mmu_notifier *mn,
221242
struct mm_struct *mm,
222243
unsigned long start, unsigned long end)
223244
{
224245
struct intel_svm *svm = container_of(mn, struct intel_svm, notifier);
225246

247+
if (start == 0 && end == -1UL) {
248+
intel_flush_svm_all(svm);
249+
return;
250+
}
251+
226252
intel_flush_svm_range(svm, start,
227253
(end - start + PAGE_SIZE - 1) >> VTD_PAGE_SHIFT, 0);
228254
}

0 commit comments

Comments
 (0)