Riak 3.4 - Updated Defaults

[martinsumner]

As part of Riak 3.4 - are there default setting that should be changed.

Some potential changes ...

leveled_reload_recalc

This is by default disabled, having been introduced in 2.9.2 as a non-default setting (it is disabled by default), but enablement is the recommended mechanism.

The reason for it being non-default was that it was not backwards compatible. That is to say if you enable it, and then have to regress to an earlier version, then it is is necessary to ensure that it is enabled (and configured) in the eralier version in order to achieve rollback.

Given the length of time since it was enabled, and given that all non-functional testing is now done with this setting - enabled should now be the default.

There is a potential argument that the configuration should be removed altogether, that recalc mode be enforced.

buckets.default.allow_mult

The default for typed buckets is that allow_mult = true. this configuration sets the default for non-typed buckets and by default this is configured to allow_mult = false. This is an obvious source of confusion, and is related to a the transition from Riak 1.4 to Riak 2.0. The default should now be allow_mult = true, so that all buckets - typed and non-typed have the same default.

buckets.default.merge_strategy

The default for typed buckets is to use dotted-version-vectors (strategy 2). This controls the default configuration for non-typed buckets, and is by default set to strategy of 1 - the legacy Riak 1.4 vector clock.

The default should be set to 2 to resolve the difference in default between typed and non-typed buckets.

handoff_deletes

Not handing off deletes causes issues with a background AAE activity required following recovery from failure - basho/riak_kv#1850. Enabling handoff_deletes was added in 3.0.14 but it was disabled by default as a change in behaviour in a minor version.

There was also a Chesterton's Fence issue, in that it was unclear why such deletes did not originally handoff, and a code comment that implied it was not an accident - so there was concern about unknown unknown impacts of the change.

Given the ongoing in-production use of the configuration, it should now be considered to be safe to be on by default.

leveled.compression_method

The original configuration supported lz4/native - whereby native uses zlib compression (which is a BIF rather than a NIF in Riak). The default is currently native, but across volume tests and different types of data, zstd has proven to be consistently more performant without impact on the compression achieved.

This was introduced in Riak 3.2.1. Enabling in configuration zstd would prevent rollback to a version < 3.2.1. There is also the additional complication that in OTP 28, zstd compression will become a BIF - https://www.erlang.org/doc/apps/stdlib/zstd.html.

Given that all performance testing is based on zstd, this should be the default.

[martinsumner]

Another potential change is with regards to:

• buckets.default.pr (currently set to 0);
• buckets.default.notfound_ok (currently set to true).

This combinations of defaults, when combined with other details means that when two nodes fail there is a likelihood that some GETs may get a false not_found response. Repeated GETs for the same object should eventually work (unless read_repair_primaryonly = enabled).

Is this outcome, as a result of default settings, unexpected given the core commitments of Riak to availability of data? So should at least one of these settings be different by default, e.g.

• buckets,default.pr = 1, or;
• buckets.default.notfound_ok = false.

It would require release notes to have a caveat, that where the previous defaults are expected they will now need to be specifically configured as per-bucket or per-type properties before transition. Is this satisfactory/manageable as part of a major update?