Only can join through a VPN.

[Dev-Boaz]

I host my own server at home and have public IPs, but for the past three months, I can only connect through a VPN.

[BytelyPlay]

You should use the private IP if it's on the the same LAN or a VLAN that is on the same router.
If you use the public IP and the router doesn't support harpin (I'm not 100% sure what it's called) on the same LAN it'll just not work.
The public IP should only be used if not on the same LAN/router which would explain why a VPN would work.

[Dev-Boaz]

Ah, I didn't have this problem at first, maybe. I had to fix one setting in my pfSense.