Roadmap - Draft Schedule and Scope for OpenRiak 3.2, 3.4 & 3.6

[martinsumner]

The Riak roadmap highlights plans for the current and next two major Riak releases:

• OpenRiak 3.2
• OpenRiak 3.4
• OpenRiak 3.6

OpenRiak 3.2

OpenRiak 3.2 - Scope

OpenRiak KV 3.2.3 has now been released. There's not expected to be any feature development on the 3.2 path going forward, however there is an outstanding snag-list of minor changes that may lead to a 3.2.4 release:

• Memory Backend - Inefficiency riak_kv#45
• https://github.com/OpenRiak/riak_kv/issues/52
• https://github.com/OpenRiak/riak_kv/issues/54
• https://github.com/OpenRiak/riak/issues/22
• https://github.com/OpenRiak/riak_kv/issues/58
• Fold objects by_sqn continues to fold after the Journal snapshot SQN martinsumner/leveled#465

OpenRiak 3.2 - Schedule

OpenRiak 3.2 will be maintained as the recommended (by the OpenRiak community) production release of Riak until the end of 2025.

OpenRiak 3.4

OpenRiak 3.4 - Scope

The 3.4 release is expected to be optimised for OTP26, but will also support OTP24 in the initial release to assist with transition.

There are planned to be some significant new features:

• Enhancements to allow existing secondary indexes to be queried in a more flexible and developer-friendly way.
• A change to make conditional PUTs (for Compare-And-Swap type updates) more predictable in the context of race conditions.
• More flexible configuration of logging to make it easier to divert or suppress certain types of logs (e.g. backend logs, or logs with recurring metric updates).
• Enhancements to improve observation and control of Tictac AAE.
• Publication, and perhaps integration, of some existing third party tools to help with coordination between Riak and other database services (e.g. OpenSearch replication and reconciliation, readable S3 backups).

A big part of the scope of the broader work related to 3.4 release will be an update and refresh to the documentation. For the 3.4 release it is hoped that the documentation can be simplified by focusing only on the features that are expected to be used in Riak going forward. As part of this, a number of features will go dark within Riak 3.4: the features will still be present, and tested as part of the release, however they will not feature as a primary part of the 3.4 documentation (but guidance will still be found in previous versions in the documentation).

The features that are currently expected to enter dark mode in Riak 3.4:

• Riak Ensemble
• write-once path
• eleveldb backend
• memory backend
• multi-backend
• map/reduce queries
• Riak data-types
• legacy secondary index API (replaced by new query API)
• legacy anti-entropy (replaced by tictacaae)
• legacy replication methods (replaced by nextgenrepl)
• Object-specific PUT/GET options (for those options that can instead by set as a bucket property)
• Last write wins (allow_mult will continue to be configurable)
• Delete modes (only keep to be recommended)

For these "dark-mode" features it will be expected that for the OpenRiak 3.6 release, one of the following will occur: relaunch; pareto-replacement or retirement (default).

Relaunch

A relaunch may occur if there exists feedback that a feature is critical to a use-case that cannot be adequately resolved through alternative means. The relaunch will require a reassessment of the existing tests to ensure that the community is in a position to support the feature for the long-term.

As part of the reassessment of the feature, the intention will be to ensure that any relaunched feature is fully-compatible with all other features to be maintained in OpenRiak going forward. An important long-term goal is to reduce the number of caveats associated with feature combinations. It is also important that any relaunched feature does not contradict the non-functional goals of high-availability and predictable performance in Riak - either by addressing issues of scale or applying appropriate constraints.

For example if Riak data types are to be relaunched, then they may be enhanced to allow for the setting of secondary indexes on CRDT maps, and addressing issues of scale; so that CRDT objects can be expected to behave as with other objects.

Pareto Replacement

Generally replacement will be preferred to relaunch, where as part of that replacement 80% of the value can be retained at 20% of the overhead.

For example, if Map/Reduce is used primarily for multi-get or filtered fetches of JSON paths in objects: a feature may be added to perform those specific tasks only in a more reliable and developer-friendly way. For Riak data-types, if most of the use is for small sets or counters associated with existing keys: limited Riak data-types may be included for use in object metadata only.

Retirement (default)

If a feature is retired, it will be removed from OpenRiak 3.6. It will continue to be supported in OpenRiak 3.4. The changes necessary to uplift OpenRiak 3.4 for OTP 27 or 28 are not expected to be significant - so the potential longevity of the 3.4 release, in terms of compatibility with supported operating systems and hardware, may be to the end of the decade.

The focus of the maintenance team is expected move to OpenRiak 3.6 towards the end of 2026, so getting help with issues in OpenRiak 3.4 beyond 2027 may depend on the Riak community and the paid-for support providers within it.

OpenRiak 3.4 - Schedule

OpenRiak 3.4 is currently under active development, but is available for pre-production testing, along with an updated functional test system. The following roadmap of events is expected:

• March 2025 - release of initial non-functional test results, and peer-review complete of major functional changes.
• May 2025 - tagging of initial release-candidate for formal testing.
• October 2025 - tagging of production-ready release, with renewed documentation and exemplar clients.
• 2026 - potential uplift of OpenRiak 3.4 to be made compatible with later OTP releases.

OpenRiak 3.6

The OpenRiak 3.6 release is expected to be optimised for OTP28, but will also support OTP26 in the initial release to assist with transition.

OpenRiak 3.6 - Scope

The scope of the OpenRiak 3.6 release will be determined by the feedback on the impact of features entering 'dark-mode' in OpenRiak 3.4. Currently it is expected that development will be focused on supporting the transition away from retired features and the resolution of internal technical debt. OpenRiak 3.6 will better utilise underlying BEAM capabilities where such capabilities can replace bespoke development. OpenRiak 3.6 will remove all legacy code relying on suppressed deprecation warnings - in particular the use of Erlang FSMs. It is also expected that large C/C++ dependencies (other than compression libraries) will be removed, improving build times and build portability.

OpenRiak 3.6 - Schedule

Work on providing an initial openriak-3.6 branch and resolving compatibility issues related to the OTP 28 update is expected to be complete by March 2026. A roadmap for release candidate and other release dates will be made available then.

[tburghart]

A Note on Protocol Compatibility

There are ongoing discussions around TLS everywhere, and such capability is likely to coincide with the 3.6 timeframe. It's also possible that the preferred implementation of consistently-configured TLS may dictate a change in how connection handshaking and authentication are performed that might, in turn, warrant a version bump to 4.0 as existing clients would no longer be wire-compatible.

If that turns out to be the case, I see a 4.0 release somewhat in parallel with 3.6.

[martinsumner]

OTP 28 and Riak 4.0

This is a summary of the work which is expected to be required to get ready for OTP 28 and ensure we have a valid release candidate for Riak 4.0:

• There is a PCRE library update in OTP 28. Riak makes use of the PCRE library. Some experimentation has been done with switching to re2, but the benefits of using re2 mainly came when pre-compiling regular expressions - and via a NIF we could not pre-compile on one node and use on others. However, doing the same thing in PCRE wasn't safe, but maybe is in OTP 28.1. Compatibility needs to be checked, and there may be a need for a capability check, especially if pre-compiling and distribution is to be used.
• Riak, via leveled, makes use of zstd compression and zstd compression is added as part of the standard library in OTP 28. There are some naming compatibility issues to resolve, but also to prove we can transition up/down between standard library and NIF-based versions. Check required on relative performance (the NIF use in leveled has a hack to improve performance around selection of dirty non-dirty NIFs). Now it is part of the standard library, is compression used elsewhere, where there should be benefit from the change?
• For Riak 3.4 a back-ported version of the OTP JSON library is used - from Riak 4.0/OTP28 we should use the standard JSON library.
• There is an open issue on leveled about refactoring serialisation and deserialisation of binary blocks in leveled ledgers in-part to resolve a binary_to_term performance regression post OTP 26.
• There is widespread use of deprecated string functions in Riak. Some of this has already been looked at as part of refactoring the HTTP API. This refactoring pushes the project towards considering a more major refactoring here; in particular with regards to connection management on the PB API, and perhaps the use of webmachine in the HTTP API.
• All riak repos have a pre-compiled rebar3 binary within the repo, this at least has to be made OTP 28 compatible, but the preferred approach would be to remove the binary altogether. Loading a pre-compiled binary was done in part to address issues caused by different developers collaborating, but using different rebar3 versions locally; so in changing this it is necessary to take measures to avoid these re-occurring as an overhead.
  • As part of updating the rebar3 library, there is an outstanding issue of local changes made to the Riak version (to provide alternate messages on use of deprecated functions).
• Checks of dialyzer and eqc test passes are required in OTP 28 for all repos - though this job may be reduced by removing some repos as part of Riak's own deprecation strategy.
• There are specific issue sin OTP 28 wih the use of maybe and else keywords, though these have already been addressed in most cases via community contributions.
• There are still a number of gen_fsm behaviour modules (with suppressed deprecation warnings) within Riak. The three challenging ones to address are riak_kv_get_fsm, riak_kv_put_fsm and riak_core_vnode. The can has been kicked down the road too often already with these legacy behaviours, and so these should be updated as part of the OTP 28 upgrade.

[pjaclark]

I'm not keen on:

• removing the memory backend as it's useful for insanely fast ephemeral data like sessions.
• removing multi-backend as it is essential for OpenRiak CS and anyone with different storage criteria for different data - i.e. fast SSDs for often-updated/read data and slowed disks for uncommonly updated/read data.
• removing CRDTs (OpenRiak data-types) is also a bad idea in my opinion. Client-side solutions are nowhere near as good.
• removing write-once path as it is ideal for scenarios where data immutability is guaranteed, such as logging, event storage, or archival systems, especially as it offers 2x performance over the normal system

We've just found an issue with TicTac AAE with a multi-backend that uses leveled that we're looking into, so might it be worth us (TI Tokyo) taking ownership of the multi-backend (and multi-backend-prefix) feature? And also the memory backend as we also use that.

I think Rob Carbone has expressed interest in taking over CRDTs.

Is there an alternative to the write-once path? Happy to use another good solution (and document it so other people can move to the better solution).

[martinsumner]

I'm generally skeptical about a lot of the claims about how fast things can be made through configuration in Riak, and whether whatever speed improvements are found are worth the confusion caused by the (often unexplained) side-effects.

Take write-once for example. What do we mean by the write-once path being "twice as fast"? In the real-world isn't what matters write-throughput in the cluster not client latency to an individual PUT request? How does write-once compare with LWW with regards to write throughput? What happens with any latency improvement when throughput increases towards capacity - especially compared to modern Riak with vnode-polling for coordinator selection.

If we use write-once, then replication doesn't work ... and take a look at this note in the actual Riak docs:

Write-once buckets do not support Riak commit hooks. Because Riak objects are inserted into the realtime queue using a postcommit hook, realtime replication is unavailable for write-once buckets. Fullsync replication will, however, replicate the data.

What do we think will happen to throughput in the cluster when we try and run full-sync to replicate all these objects that weren't sent via real-time?

Do we have a clear list of all the features that don't work as expected with write once path. node_confirms, sync_on_write won't work as expected I'm sure. Do we know for sure if AAE (of either form) works as expected in all scenarios? There certainly aren't any tests that confirm that it does - there are only two tests of the write once path (one checking a subset of PUT properties, one checking behaviour in handoff).

write once also leads to big risks of mailbox_overload responses ... is this clear to any user, and what do we advise the application should do in this case?

The in-memory backend is in-theory faster, but is it "insanely" faster in the session object scenario - given that in persisted backends most recently written objects are in effect in-memory (in the page cache). There's then the question of how fast
it is once you're in the garbage collection scenario, given that requires a full table scan every time it is triggered (and there's nothing to prevent the whole preflist of vnodes entering the scenario at the same time). Also what happens if the previous GET has seen a TTL expiry (which again prompts a full table scan to remove index entries). Also the backend does not have a queue of its own, it relies on the vnode queue - and in most high-capacity scenarios it is the vnode queue (shared across all backends) which governs latency.

For the last release there was a lot of effort put into showing stability of Riak under failure scenarios with heavy load. Do these same characteristics hold true if we're using the in-memory backend or the write-once path? Are we being honest about those characteristics if we support configurations where the characteristics are not true.

Things like tiered storage, and handling session objects are valid use cases - but that doesn't mean that the way of handling them in the past is the right solution for Riak in the future. For example, tiered storage could be handled at the backend
(as it already is in eleveldb). The vnode already has an (off-by-default) metadata cache, adding a LRU cache for reads (with eligibility being a bucket property) might be a better answer to improving read latency for session objects (if this is what we require most).

There is a lot of work required to properly support these configurations going forward if we want to maintain them in Riak 4.0. I think there would be rightly an expectation in any user community that any configuration we support is tested thoroughly both functionally and non-functionally with every release to ensure there is no regression. Right now there is little testing of non-standard paths and backend setups beyond the singular use of the bitcask or leveled backend. There's a tax to pay on functional development to keep all configurations in-line. For example it is perfectly possible for nextgenrepl to support the write-once path, it just wasn't worth to those paying for the feature to add the extra cost of doing it. How do we address this on an ongoing basis as features are extended?

[tburghart]

We do currently use multi-backend and LWW. We may be able to get away from LWW by using only allow_mult, but the inertia of a very large dataset likely makes it impractical for us to retire multi-backed support in the foreseeable future.

I'm inclined to keep the memory backend around because a) it's small, simple, and with some cleanup should be a very low maintenance and testing burden; and b) it provides a compelling solution to a class of use cases where it's desirable to avoid writing data to persistent storage.

Multi- and memory- backends complement a tiered storage strategy. I don't like the multi-backend, and wish we weren't using it, but I do think it's a necessary component for numerous use cases.

[martinsumner]

Let us say we have a set of valid user needs, that are currently either wholly or in-part met through the use of multi-backend:

• CPU efficient mutations to rapidly changing temporary objects (e.g. for web session objects);
• Minimal latency reads of some objects (e.g. for web session objects);
• Support for temporary objects, with variable TTLs by bucket;
• Support for the economically-efficient distribution of data across storage tiers.

If we could meet these user needs in Riak 4.0 without the need for multi-backend, would that then allow for the removal of multi-backend in 4.0 (either the total removal, or its transition to being a dark undocumented feature to be configured only by self-supporting expert users)?

The only other user need that I have seen that was met by multi-backend was the ability to rapidly drop buckets (i.e. you put a bucket in a specific backend, so that you can drop it by rm -rf rather than hnving to delete all the objects), but I'm not sure how common that need is.

[martinsumner]

I think a possible way forward for 3.6/4.0 is to have just two supported backends:

• multi_backend where each backend is bitcask;
• single leveled backend.

These are backend settings which are well understood and tested in production environments.

By reducing the number of options, we can then make it much easier to test and document how to handle different scenarios, and what the implications are of the backend choice. Scenarios to be covered might be:

• use tiered storage;
• backup/recovery;
• enable compression;
• enabling anti-entropy;
• storing temporary objects with guaranteed latency.

For the last case (storing temporary obejcts), the answer for the multi-backend/bitcask scenario might be to add a memory backend, and there might be a need to develop an alternative way for the leveled case. However, there is some significant work to be done with the memory backend - I don't think right now it is fit for purpose.