Rollup merge of #99966 - RalfJung:try-dont-panic, r=lcnr

avoid assertion failures in try_to_scalar_int

Given that this is called `try_to_scalar_int`, we probably shouldn't `assert_int` here. Similarly `try_to_bits` also doesn't `assert!` that the size is correct.

Also add some `track_caller` for debugging, while we are at it.

r? ```@oli-obk```

Author: matthiaskrgr