IH-615: move metrics collection out of RRDD

[last-genius]

Moves out two metrics collectors out of RRDD into separate plugins:

• CPU-related metrics
• Network-related metrics

Three more collectors are still in xcp_rrdd.ml, this is because memory-, HA-, and SR-cache-related metrics collection is tightly integrated with RRDD itself, exposing calls through RRDD bindings for XAPI to use. To convert these into plugins (and drop the collection-related code in xcp_rrdd.ml, and stop linking to Xenctrl), these calls need to be done through standard message switch mechanism instead. This is for a future PR.

Also:

• netdev metrics generation is tightly integrated with networkd, serializing these into a file, which the plugin reads from. not terribly efficient but i don't see a way to disentangle it from networkd, so it's kept this way.

Needs to be merged together with a corresponding spec repo PR, I will open it after XenRT testing completes (currently running BVT/BST) + also need to test manually

[last-genius]

Encountered crashes with the previous version of the code, determined that the cause was removing deprecated xcp-rrdd calls, so reverted these and had to also leave HA-related metrics collection as is for now, it needs to be converted to use the message switch separately. This is currently finishing up XenRT testing (#205763) after some manual testing and looking good.

UPD: BST+BVT passed

[robhoes]

netdev metrics generation is tightly integrated with networkd, serializing these into a file, which the plugin reads from. not terribly efficient but i don't see a way to disentangle it from networkd, so it's kept this way.

Essentially, networkd is an rrdd plugin using protocol "v0". I think that much of the code in network_monitor_thread.ml can be merged into the new rrdd plugin. Part of the data, however, goes to xapi (see monitor_dbcalls.ml) and goes into its DB. And that particular data isn't numbers, so does not fit will in the RRD protocol.

[psafont]

after some manual testing

What did this involve? Opening xencenter and checking that all the metrics were still reported? (and check that they changed when adding traffic / cpu load)

[last-genius]

What did this involve? Opening xencenter and checking that all the metrics were still reported? (and check that they changed when adding traffic / cpu load)

I initially tested with xe host-data-source-query, but yes, XenCenter also shows performance stats for hosts and VMs (in particular, the modified CPU and network-related stats):

[last-genius]

Rebased on top of master, there were some dune/makefile changes that modified things around rrdd plugins.

[robhoes]

We have discussed moving the part of network_monitor_thread.ml (xcp-networkd) that ends up doing into the RRDs (the rx and tx counters) to the new RRD plugin daemon. We will leave the other stats (carrier, duplex, etc), which are read by xapi and put in its DB, in xcp-networkd.

I am approving this PR now so that we can merge it, but we should do the above relatively soon as well.