CP-48676: Reuse pool sessions on slave logins

[snwoods]

This was first introduced in #5986 but was later reverted as the mutex it introduced caused a deadlock with the DB mutex. This PR has now been reworked to use an Atomic variable, removing the need for the mutex.

This PR allows pool sessions to be reused (if the flag is turned on, which it is by default). This greatly speeds up communications between hosts which is important for repetitive calls e.g. one customer was running get_servertime frequently on all hosts in a pool which was causing problems due to the length of each call.

When we get this reusable session, we can optionally validate the session before using it but this increases the duration of the call, so this is off by default. Excepting a toolstack restart, the reusable session should always be valid as it is excluded from deletion in destroy_db_session

[psafont]

Some minor style issues. The general approach seems quite sensible, even if a race can cause xapi to use more than one session (to avoid a loop when getting an invalid resusable session)

[snwoods]

The Ring3: BVT + BST suiteruns I have run were all green, both with (211574) and without (211571) additional debug logging. The debug suiterun shows that the pool sessions are being reused as expected.

[snwoods]

During testing (BVT + BST etc.) the resuable_pool_session never became invalid because we prevent the resuable_pool_session from being destroyed in destroy_db_session. Something I'm unsure about though is, is there any possible situation where the session does become invalid (excluding a toolstack restart where a new session will be made anyway)? If that were to happen and Xapi_globs.validate_reusable_pool_session = false, is_valid_session would still return true (as session != null) and whatever the session is used for would fail. I'm unsure if we would then get out of this state.

I've marked all the comments as resolved where appropriate. This is still an outstanding question for me though.

[edwintorok]

This is still being tested, apparently the member is not reusing the session enough times (we were trying to determine whether it is able to cope with a coordinator restart, and it seems it can, but then it is not trying to reuse the session either).

Putting back into draft until we figure out whether once it is reusing the session properly if it still works.

[edwintorok]

OK looked at this in more detail, and the pool session was mostly actually created on the coordinator, so this is fine (it avoids lots of DB calls to create sessions, and reuses them).
The SM sessions doesn't yet take advantage of this, it'll need a separate PR to reuse SM sessions and avoid logouts on SM.

[snwoods]

Yes as Edwin said, we were initially confused about which way the improvement should be going, hence the comment about the coordinator restarting. In actual fact, this PR is to avoid remaking sessions on the coordinator, so the coordinator or member restarting is not an issue (in the former case, the coordinator will get a new reusable session and in the latter, the reusable session will persist). As you can see in the image below, the difference from reusing the session is 1.1ms without this change and 0.05ms with the change.

[snwoods]

I've added a commit which defaults reuse_pool_sessions to false, so this PR is a noop and safe to merge. We can then run some nightlies with the commit reverted and then revert the commit on master when we are confident.