segment gc when ts_cnt > 1

[vagetablechicken]

OpenMLDB/src/storage/segment.cc

Lines 437 to 458 in 21184d5

	void Segment::ExecuteGc(const std::map<uint32_t, TTLSt>& ttl_st_map, StatisticsInfo* statistics_info) {
	if (ttl_st_map.empty()) {
	return;
	}
	if (ts_cnt_ <= 1) {
	ExecuteGc(ttl_st_map.begin()->second, statistics_info);
	return;
	}
	bool need_gc = false;
	for (const auto& kv : ttl_st_map) {
	if (ts_idx_map_.find(kv.first) == ts_idx_map_.end()) {
	return;
	}
	if (kv.second.NeedGc()) {
	need_gc = true;
	}
	}
	if (!need_gc) {
	return;
	}
	GcAllType(ttl_st_map, statistics_info);
	}

ref https://utqcxc5xn1.feishu.cn/docx/FTbtdV25eoZDkjxODpCc44qhnlc , if we have a table with indexes in same keys but different ts, e.g.

CREATE TABLE talkingdata(
    ip int,app int,device int,os int,channel int,click_time timestamp,attributed_time timestamp,is_attributed int,
    index(key=(ip), ts=click_time, ttl=1s, ttl_type=absolute),
    index(key=(ip), ts=attributed_time),
    index(key=(app,os), ts=click_time)
);

index0 and index1 will in the same segment and ts_cnt_==2, so segment gc will trigger GcAllType, it'll use the wrong expire time.

when ts_cnt_<=1, ExecuteGc will calc expire time:

OpenMLDB/src/storage/segment.cc

Lines 398 to 408 in 21184d5

	void Segment::ExecuteGc(const TTLSt& ttl_st, StatisticsInfo* statistics_info) {
	uint64_t cur_time = ::baidu::common::timer::get_micros() / 1000;
	switch (ttl_st.ttl_type) {
	case ::openmldb::storage::TTLType::kAbsoluteTime: {
	if (ttl_st.abs_ttl == 0) {
	return;
	}
	uint64_t expire_time = cur_time - ttl_offset_ - ttl_st.abs_ttl;
	Gc4TTL(expire_time, statistics_info);
	break;
	}

But GcAllType won't, it'll use a small time (ttl value, not the expire time, e.g. ttl=1m, time value will be 1970-01-01) to do gc. Normally, no row will be gc cuz row ts > small time, so the data never expire, you can check by show table status.

[Riccardo231]

Hello, I'd like to join the project by working on this. Could you assign me the issue? Thank you

[vagetablechicken]

Hello, I'd like to join the project by working on this. Could you assign me the issue? Thank you

@ricor07 thanks for being willing to contribute~

@Shouren @aceforeverd is this issue unassigned? Could we let @ricor07 take a look at this and work on a fix?

BTW, let me know if I can help.