Too much "over-abstraction" in early version.

[waterflier]

We are still in the early stages, and the number of lines of code I envision should be much fewer than what we have now. From an abstract perspective, one important consideration is to focus on what parts of the system are likely to change. We only need to make appropriate abstractions for things that might change, rather than making all "concepts" extensible at once. Doing so would greatly increase the complexity of the system and obscure its core issues. This reminds me of some terrifying meetings in the past, dominated by Java frameworks: reading the code made it almost impossible to intuitively understand any flow, because every function was just an interface definition. After half an hour of reading, I still didn't understand "how the system iterates or safely completes a backup task with exception handling."

From the perspective of our goals, the core design of the backup_suite is to solve the following key issues:

• BackupTarget is extensible (this is the main direction of extension).
• The extensibility of BackupSource is limited; implementing one LocalDirSource should be almost sufficient.
• task_mgr / engine hardly needs to be extended. Most of the other components in the system are passive components, and task_mgr should act as a driver, managing various details and processes (it doesn't always have to be elegant). This will clarify the boundaries of other components, and we can refactor this gradually.
• Meta (the name isn't great, it's too generic; I understand it as BackupTaskDb) doesn't need to be considered for extensibility. As long as correctness is ensured, implementing one based on MetaDB is enough.
• Integration with buckyos. I understand that many of the extensibility designs in Meta are for future integration with buckyos, but I don't think it's necessary to think about that at this early stage. We should focus on building a solid and standalone backup product first. A simple and reliable implementation can be easily reused in another system.

Keeping things simple is not easy. It requires a deep understanding of the system's critical complexities.

[streetycat]

Our abstraction primarily focuses on Source and Target. The Traits related to Engine, Task, and CheckPoint are mainly to clearly outline their interfaces and functional boundaries during the design phase. Storage is mainly considered for potentially supporting different databases in the future.

[streetycat]

Additionally, the database records various transmission-related information. I hope that plugin developers can easily extend functionalities based on these features. They should focus only on the target storage providers they want to support, without worrying about the storage and recovery of transmission states. In the Engine, there are several steps at each stage:

1. Preprocess to obtain a state (implemented by the plugin)
2. Save the state (implemented by the Engine)
3. Execute a process using this state (implemented by the plugin)
4. Move to the next state

This approach can guide plugin developers to consciously break down the process into several steps, thereby reducing the risks associated with process re-entry.

A classic scenario is blockchain transactions:

1. Generate tx
2. Save tx
3. Deliver tx

However, this also introduces a lot of complexity. Once this part is introduced into the Engine, it becomes very abstract. Several tables in the database are used to save the state of the Target plugin:

1. task_source_state: Saves the original state and mirror state of the Source when initiating a backup, for state recovery upon backup completion.
2. checkpoint_transfer_map: Saves the Map of each data block to the Target transfer address.
3. checkpoint_key_value: The Target might have some items to store.

[waterflier]

I understand the problem the current design aims to solve. My primary concern is that the design is not well-balanced at this stage:

• At this stage of the code, the priority should be on achieving a simple (simple is definitely reliable) and high-performance solution for completing a typical backup task.
• In terms of extensibility, it should be enough for us to focus on extending the backup target. I believe there won't be much demand for extending the backup source in the short term (if we can support directories backup source well, that would be great).
• We don't need to make the entire framework extensible everywhere when the overall structure is not yet fully determined. In practice, providing some simple components and later combining them for new scenarios is much simpler and more intuitive than solving problems within a framework using various post/pre/hooks.

[streetycat]

Our abstraction primarily focuses on Source and Target. The Traits related to Engine, Task, and CheckPoint are mainly to clearly outline their interfaces and functional boundaries during the design phase. Storage is mainly considered for potentially supporting different databases in the future.

Sorry, I have lost something, In my design, Source, Engine, and Target are three independent processes that interact with each other via HTTP protocol. Consequently, the implementation of Engine, Task, and CheckPoint resides within the Engine itself, with no abstraction layers. However, for Source, Target, and the integrated product, an HTTP client implementation is required to remotely call the corresponding interfaces in the Engine.

The need for abstraction arises from the implementation requirements rather than from an extensibility design perspective.

@waterflier