To solve the OOM problem of Optimizing and MOR when there are too many eq-delete records

[wangtaohz]

Arctic KeyedTable will have too many eq-delete records in some situations:

• spark insert many records into the ChangeStore with write.upsert.enabled=true to avoid data duplication.
• Optimizing is blocked for a long time while streaming ingestion is working.
• ...

If there are too many eq-delete records, the Optimizing and MOR are easy to OOM, since all the Primary-Key values of these eq-delete records have to be loaded into a map in memory.

In a typical case with two string columns as Primary-Key (the length of each string column is about 20 bytes), I find that about 4,200,000 records occupy 1.8G memory, which seems to be reasonable ?

[zhoujinsong]

Too much delta data is indeed very easy to cause out-of-memory problems.
In my opinion, there are two ways to alleviate this problem：

• Split optimizing tasks based on their memory usage to protect the optimizer from OOM problems.
• Introduce disk-based storage engine like RocksDB to expand our storage capacity

[zhoujinsong]

#925 had introduced disk-based storage to expand storage capacity when loading deleted datas.