CP-49078: Preprocess fields into a Hashtbl within get_record

[contificate]

Flame graphs indicate that, under load created by parallel "xe vm-list" commands, the DB action get_record is hit often. This function constructs an API-level record by marshalling an association list that maps field names to unmarshalled string values. To do this, it serially queries all the field names using List.assoc. This has rather large cost in doing lexicographical string comparisons (caml_compare on string keys).

To avoid this, regardless of record size, we preprocess the association lists __regular_fields and __set_refs into a (string, string) Hashtbl.t and query that to construct each record field.

---

This benefit of this change is most notable for large records (such as VM and pool). The cost of the previously generated code, which does a bunch of serial List.assoc calls, incurs the quadratic cost of list traversal (compounded by the costly lexicographical comparison of strings during the search).

To produce measurements, I sampled xapi under a load of 500 consecutive xe vm-list invocations (using the same sampling rate with perf) on a host with a single VM.

Without the change, the get_record done internally by xe vm-list makes up for ~33.59% of the samples (33.59% = 1,592,782,255 samples). With the change, get_record accounts for ~7.56 of the samples (of which there are substantially fewer collected: 7.56% = 264,948,239). So the number of samples for get_record has dropped from 1,592,782,255 to 264,948,239 (assuming perf's sampling is reliable).

You can see the visual difference in the flame graphs:

Before:

After:

Of course, this is benefit as measured in aggregate (perf sampling), so quite a fast and loose comparison. In practice, the xe vm-list stress test goes from 7.4s to 6.2s (as get_record makes up only a small part of the work done for a single xe vm-list).

[contificate]

I've noticed we can do the same thing inside API as well, so I've done that:

Before (the areas each commit is targeting):

After:

The time change for the serial vm-list stress is now 4.8s (from 7.4s) after both commits.

[contificate]

BVT+BST (208107) all green except 2 known issues (I think).

[contificate]

Added a third commit that writes Schema to retrofit hash tables there as well.

Before:

After:

This removes 17% of the samples collected. From 40% for Db_cache_impl.fun_ to 23%.

In future, I'd like to rewrite parts of the database, including the schema.

[lindig]

How are tables shrining or removed?

[contificate]

How are tables shrinking or removed?

The database internals are unchanged. It's still Maps of Maps.

The 3 relevant commits do the following:

• In generated DB actions, in get_record_internal, instead of using List.assoc to find __regular_fields and __set_refs, I construct hash tables from those association lists and query those.
• In generated API module, the RPC conversion functions don't use List.assoc either. Like above, they consult a hash table created on the fly.

In these cases, the hash tables are ephemeral. In the third commit:

• The Schema module now stores tables and columns in their own tables.

This probably persists for the entirety of xapi since the Schema is constructed from the datamodel and consulted often.

The benefit is purely removing the huge cost of comparing strings.

The 3 relevant commits remove 3s off the total time taken by my serial xe vm-list test (which does 500 xe vm-list invocations consecutively). From 7.4s to 4.4s, so it's low hanging fruit but unlikely to be notable for single invocations but lessens the previous blowup at scale.

---

In each case, I experimented with precomputing a perfect hash table instead. I don't think it's worth the effort, because:

• The hash function should effectively be compiled in so that the compiler runtime and program runtime agree. It's hard to beat OCaml's own builtin hash function which will hash strings at the granularity of words, not individual characters.
• You still need to populate mappings in the first 2 commit cases. So, really, you'd just have a static resolution scheme. It's hard to balance compile time against the cost of the hash functions (I was doing the "Compress, Hash, and Displace" scheme). Hashtbl provides a flexible alternative that's hard to compete with.

Hashtbl won out against all other implementations: balanced trees (Map), compressed prefix trees, my own perfect hashing scheme, etc. due to pure constant factors.

---

The other side effect is that it kind of "normalises" the shape of flame graphs we collect. Instead of their width being dominated by List.assoc and friends, now the lookups are roughly the same size as the things around them - removing this noise makes it more obvious where our attention should go next.