Transition not showing the children if transition is not active

[Joostdeboer]

Using version:
v1.7.13 (lower version needed to make the Popover component disappear properly)

Reproduction:
https://codesandbox.io/p/sandbox/lucid-margulis-zhcytl
To see the issue, click on e.g. "Test1" or "Test4".

Issue:
I'm running into the following issue. I have a header menu I wished to include some items in that appear as a dropdown/popover when clicked. I added a transition to the caret to make it flip over, but instead found something I'm not quite sure of how to fix.
What looks to happen is that, whatever is inside the Transition itself, seems to unmount when the transition is not active. I've tried using the show prop, or appear or unmount, but none of these provided the desired effects.

Perhaps I'm not approaching this correctly. If not, I'd like some advice as to how to properly implement this transition. But it seems like it is bugged. How is the Transition supposed to work here? Are the elements inside the transition supposed to never show its children if not active?