Self-optimizing classification and goals

[zhoujinsong]

I want to start a discussion about the category of self-optimizing and make it the rule when designing the optimizing planning for all kinds of table formats.

In my option, I would like to divide the optimizing tasks into the following three categories:

Type	Definition	Goals	Task size	Scheduling frequency
Minor optimizing	Optimizing on fragment files( smaller than 16MB by default)	Reduce small files and transfer a write-friendly file to a read-friendly file	Small (like 16 MB)	Very often( like every 10 minutes )
Major optimizing	Optimizing on segment files ( bigger than 16MB and smaller than 128MB by default)	Merge files to target size and merge redundant files	Medium (like 128 MB)	More frequent ( like every 1 hour)
Full optimizing	Optimizing on all files under table/partition	Rewrite all files in order	Dependent on table/partition size	Less frequent (like 1 day)

Let me know if you have any ideas about that.

[baiyangtx]

In feature, if the batch and olap jobs read/write the base store is the reacommanded way to use mixed-iceberg, the change to base optimizing will be match more important.
So I recommand that:

1. All change to base optimizing are minor optimizing, no matter the file size is.
2. The schedule delay of minor optimizing shoud be guaranteed, currently the optimizing group could do one table and one type optimizing at once, I suggest that the minor optimizing schdule shoud be high priority and will no blocker by the other optimizing type.

[baiyangtx]

I think it has TWO types of optimizing jobs.

1. The purpose of optimizing job is to improve the query performance. All three catagories you listed are beloned to this type.
2. The purpose of optimizing job is to push forward of watermark. This type of optimizing job is more important for it has business meaning.

[shidayang]

@zhoujinsong I agree with you, we need a Major optimizing to make file from 16m to 128m. Many 16m file has two problems:

1. Affect query efficiency
2. Waste of storage resources

[wangtaohz]

For the Mixed format table's Minor Optimizing, when moving files from change store to base store, rewriting these data files from change store(they are likely to be fragment files < 16M) also brings the following advantage:

• making the data file only associated with one pos-delete file (otherwise it depends on the count of nodes)

[wangtaohz]

In practice, if not rewriting files from change store, we could see a data file in base store associated with 1~4 pos-delete files when there are 4 nodes.