On Mir architecture

[sergefdrv]

This discussion is supposed to summarize main advantages and concerns
regarding the architecture of the Mir framework. Particular concerns
and architectural choices are to be discussed in detail separately.

[sergefdrv]

Advantages

• Modularity. Mir's architecture separates concerns between
  several pluggable modules, such as core protocol logic,
  communication, write-ahead log etc. There is a good barrier between
  modules for they should only interact by sending each other events
  through a common event handling mechanism. However, the event
  handling mechanism may be less convenient for achieving
  finer-grained modularity because it requires a significant amount
  of boilerplate code.

• Atomicity and sequentiality of code. Event handling happens
  strictly sequentially and atomically in each module. Moreover,
  event handling is required to be fully deterministic. This is
  supposed to simplify reasoning about logical correctness and
  absence of concurrency bugs, e.g. deadlocks, livelocks etc.
  Nevertheless, due to inherently concurrent nature of distributed
  systems, different modules have to handle incoming events
  independently and concurrently. Thus, hidden concurrency bugs
  cannot be easily avoided, but will likely be lifted from the usual
  level of synchronization primitives to the level of event-based
  interaction and inter-dependencies between individual concurrent
  modules.

• Traceability of execution. Mir's event-based architecture
  enables a natural way of capturing traces of cross-module
  interaction, e.g. for diagnostic purposes. Individual module's
  implementation doesn't need to deal with tracing explicitly.
  However, it may be less convenient for achieving finer-grained
  traceability because of requiring additional intra-module events,
  i.e. incurring a significant amount of boilerplate code.

• Reproducibility of execution. Mir's architecture requiring
  deterministic event handling makes it possible to support running
  reproducible test scenarios. In principle, any architecture
  enabling controlled injection of otherwise non-deterministic
  events, such as those related to the local real-time clock or
  communication between nodes, would work as well.

[sergefdrv]

Concerns

• Code structure. Mir's event-based architecture seems to induce
  code fragmentation since handling of individual events is supposed
  to be simple, fast, and non-blocking, i.e. expensive or complex
  operations should be split or delegated to separate modules.
  Admittedly, one can still maintain decent level of code readability
  by placing logically connected event handlers next to each other in
  source code. However, splitting otherwise logically continuous
  operations into multiple event handlers makes them non-atomic. This
  can become a source of concurrency bugs and complicate logical
  reasoning about correctness of the implementation.

• Boilerplate code. Programming in Mir framework seems to require
  a significant amount of boilerplate code due to its event-based
  architecture. Each new operation requires to define a new event
  sub-/type, as well as additional code to conveniently construct and
  properly dispatch the new event sub-/type.

• Flow control. Mir's architecture doesn't seem to offer any
  convenient mechanism to enable flow control within the system.
  Conversely, it seems to demand non-blocking operation for most
  event types as well as implicitly rely on unbounded buffers to hold
  events waiting for being processed. One can argue that there is
  logically an upper bound on the amount of internally-produced
  events waiting in the event buffers. However, when it comes to
  external events, such as reception of messages from potentially
  malicious peer nodes, or potentially blocking or time-consuming
  operations, such as appending to a persistent write-ahead log or
  processing of ordered requests by the application, there must be
  some kind of flow-control mechanism. The event-based architecture
  of Mir seems to require implementing some non-trivial inter-module
  flow-control protocols in such cases.