[bug] `prefetch_related` stops working in Django 3.0

[catriuspham]

It used to work in Django 2.x, but now using prefetch_related on a SortedM2M fields make it an empty one.

[AlanCoding]

I added a new test here #208 and I'm not getting any hits. Maybe because you're using Django 2.x, we start testing with Django 3.0.