Skip to content

[opt](nereids) support prune partitions by specified tablet ids #50424

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

zhaorongsheng
Copy link
Contributor

@zhaorongsheng zhaorongsheng commented Apr 25, 2025

What problem does this PR solve?

Issue Number: close #50255

support prune partitions by specified tablet ids

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhaorongsheng zhaorongsheng changed the title [fix](scan) Support Tablet/Sample Partition Prunning [fix](scan) Fix Invalid Partition Number About Tablet/Sample Query Apr 25, 2025
@zhaorongsheng zhaorongsheng force-pushed the bugfix-partition-prunner-about-querying-tablet branch from 63365f6 to 692c422 Compare April 30, 2025 10:38
@zhaorongsheng zhaorongsheng reopened this Apr 30, 2025
@zhaorongsheng
Copy link
Contributor Author

run buildall

@zhaorongsheng
Copy link
Contributor Author

run buildall

@zhaorongsheng zhaorongsheng changed the title [fix](scan) Fix Invalid Partition Number About Tablet/Sample Query [fix](scan) Fix Invalid Partition Number About Specified Tablets Query May 6, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 33747 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c1500c5a3725caf3f10f28dfbb011cf1a527ec99, data reload: false

------ Round 1 ----------------------------------
q1	25997	4971	4964	4964
q2	2065	270	198	198
q3	10401	1253	676	676
q4	10225	980	513	513
q5	7533	2338	2308	2308
q6	177	162	132	132
q7	926	748	612	612
q8	9311	1280	1035	1035
q9	6948	5025	5105	5025
q10	6970	2348	1926	1926
q11	486	293	276	276
q12	358	353	225	225
q13	18584	3664	3176	3176
q14	222	225	213	213
q15	521	482	481	481
q16	437	431	364	364
q17	599	847	354	354
q18	7421	7134	7143	7134
q19	2022	984	547	547
q20	334	323	214	214
q21	3762	3408	2397	2397
q22	1052	1008	977	977
Total cold run time: 116351 ms
Total hot run time: 33747 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5098	5023	4978	4978
q2	242	321	226	226
q3	2128	2656	2295	2295
q4	1357	1758	1341	1341
q5	4460	4415	4435	4415
q6	211	169	125	125
q7	1969	1956	1740	1740
q8	2539	2530	2677	2530
q9	7205	7255	7078	7078
q10	2986	3152	2752	2752
q11	564	530	485	485
q12	676	751	604	604
q13	3467	3822	3349	3349
q14	298	302	277	277
q15	523	473	466	466
q16	437	475	434	434
q17	1138	1476	1397	1397
q18	7790	7556	7541	7541
q19	797	818	847	818
q20	1953	1963	1820	1820
q21	5013	4681	4657	4657
q22	1107	1042	1050	1042
Total cold run time: 51958 ms
Total hot run time: 50370 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191738 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c1500c5a3725caf3f10f28dfbb011cf1a527ec99, data reload: false

query1	1385	1095	1100	1095
query2	6409	1813	1825	1813
query3	11014	4501	4551	4501
query4	54785	24538	23341	23341
query5	4934	559	456	456
query6	334	199	201	199
query7	4887	504	291	291
query8	320	282	253	253
query9	5547	2571	2541	2541
query10	449	310	269	269
query11	15274	14999	14697	14697
query12	160	107	101	101
query13	1022	503	399	399
query14	10021	7066	6375	6375
query15	192	210	174	174
query16	7090	666	482	482
query17	1091	762	590	590
query18	1542	436	337	337
query19	205	202	175	175
query20	132	133	131	131
query21	214	133	111	111
query22	4264	4381	4241	4241
query23	34002	33207	33640	33207
query24	6081	2459	2450	2450
query25	467	469	399	399
query26	714	294	152	152
query27	1791	475	339	339
query28	2652	2123	2111	2111
query29	571	551	429	429
query30	271	227	191	191
query31	858	866	796	796
query32	70	72	64	64
query33	457	369	303	303
query34	788	849	526	526
query35	827	872	751	751
query36	912	1007	909	909
query37	110	97	78	78
query38	4155	4234	4152	4152
query39	1475	1446	1435	1435
query40	218	122	107	107
query41	57	52	57	52
query42	123	113	114	113
query43	487	505	480	480
query44	1310	811	810	810
query45	183	175	173	173
query46	848	1028	637	637
query47	1843	1861	1808	1808
query48	392	420	307	307
query49	689	517	444	444
query50	653	701	400	400
query51	4313	4188	4171	4171
query52	118	113	101	101
query53	235	258	191	191
query54	591	591	511	511
query55	84	81	86	81
query56	337	300	279	279
query57	1178	1206	1158	1158
query58	274	256	260	256
query59	2732	2802	2645	2645
query60	340	320	310	310
query61	130	128	136	128
query62	736	727	672	672
query63	226	185	189	185
query64	1797	1087	689	689
query65	4287	4262	4211	4211
query66	734	401	315	315
query67	15954	15422	15339	15339
query68	6836	829	500	500
query69	532	289	263	263
query70	1220	1112	1106	1106
query71	501	348	283	283
query72	5805	4833	4818	4818
query73	1442	671	352	352
query74	9120	9258	8644	8644
query75	4027	3181	2675	2675
query76	4254	1200	745	745
query77	705	388	280	280
query78	10103	10031	9272	9272
query79	3836	809	551	551
query80	649	522	455	455
query81	490	253	227	227
query82	492	124	99	99
query83	308	256	235	235
query84	302	107	90	90
query85	808	358	313	313
query86	373	286	295	286
query87	4404	4341	4247	4247
query88	3379	2194	2196	2194
query89	405	318	288	288
query90	1821	212	216	212
query91	148	150	111	111
query92	77	61	58	58
query93	2699	956	566	566
query94	656	408	301	301
query95	372	293	291	291
query96	496	570	269	269
query97	3164	3196	3116	3116
query98	241	211	204	204
query99	1421	1392	1250	1250
Total cold run time: 299427 ms
Total hot run time: 191738 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.6 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c1500c5a3725caf3f10f28dfbb011cf1a527ec99, data reload: false

query1	0.03	0.04	0.03
query2	0.13	0.10	0.12
query3	0.26	0.19	0.20
query4	1.59	0.19	0.20
query5	0.59	0.60	0.59
query6	1.17	0.73	0.73
query7	0.02	0.02	0.01
query8	0.03	0.04	0.03
query9	0.56	0.54	0.52
query10	0.57	0.57	0.57
query11	0.16	0.12	0.11
query12	0.15	0.10	0.11
query13	0.61	0.60	0.59
query14	0.79	0.80	0.80
query15	0.86	0.84	0.85
query16	0.39	0.38	0.38
query17	1.03	0.99	1.03
query18	0.21	0.19	0.20
query19	1.93	1.76	1.81
query20	0.02	0.01	0.01
query21	15.40	0.91	0.56
query22	0.76	1.20	0.77
query23	14.81	1.38	0.62
query24	9.53	0.89	0.30
query25	0.29	0.25	0.07
query26	0.72	0.15	0.14
query27	0.05	0.05	0.05
query28	8.42	0.76	0.41
query29	12.62	3.90	3.29
query30	0.26	0.09	0.08
query31	2.81	0.61	0.38
query32	3.24	0.53	0.47
query33	3.02	3.07	3.03
query34	15.82	5.11	4.45
query35	4.52	4.53	4.45
query36	0.69	0.50	0.48
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.55 s
Total hot run time: 28.6 s

@zhaorongsheng
Copy link
Contributor Author

run buildall

@zhaorongsheng
Copy link
Contributor Author

run buildall

@zhaorongsheng
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33886 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fe4027884d51b8ee81aa7aed0b080d06fe328861, data reload: false

------ Round 1 ----------------------------------
q1	25767	5279	5033	5033
q2	2062	307	193	193
q3	10353	1217	687	687
q4	10230	1017	525	525
q5	7500	2343	2341	2341
q6	182	172	136	136
q7	911	763	632	632
q8	9300	1270	1103	1103
q9	6715	5046	5066	5046
q10	6859	2332	1893	1893
q11	492	292	267	267
q12	348	347	213	213
q13	17782	3698	3059	3059
q14	236	239	203	203
q15	521	492	498	492
q16	418	424	376	376
q17	588	854	343	343
q18	7399	7216	7064	7064
q19	1726	947	586	586
q20	336	335	223	223
q21	3921	2665	2476	2476
q22	1038	1031	995	995
Total cold run time: 114684 ms
Total hot run time: 33886 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5116	5121	5140	5121
q2	240	322	229	229
q3	2151	2637	2263	2263
q4	1345	1797	1360	1360
q5	4429	4355	4392	4355
q6	225	174	134	134
q7	1979	1910	1831	1831
q8	2625	2571	2501	2501
q9	7134	7059	6848	6848
q10	3030	3187	2770	2770
q11	558	516	495	495
q12	657	759	572	572
q13	3532	3918	3305	3305
q14	281	298	286	286
q15	512	477	505	477
q16	447	498	447	447
q17	1154	1574	1366	1366
q18	7858	7447	7350	7350
q19	825	928	1067	928
q20	2000	2043	1952	1952
q21	5024	4742	4644	4644
q22	1104	1059	1023	1023
Total cold run time: 52226 ms
Total hot run time: 50257 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192873 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fe4027884d51b8ee81aa7aed0b080d06fe328861, data reload: false

query1	1400	1106	1062	1062
query2	6408	1802	1811	1802
query3	11018	4473	4451	4451
query4	53254	25430	23602	23602
query5	5092	582	461	461
query6	322	194	197	194
query7	4939	511	281	281
query8	316	255	238	238
query9	5556	2586	2588	2586
query10	443	326	266	266
query11	15083	14957	14896	14896
query12	165	112	108	108
query13	1062	524	399	399
query14	10144	6431	6403	6403
query15	208	213	181	181
query16	7120	706	512	512
query17	1106	754	634	634
query18	1534	412	322	322
query19	211	206	170	170
query20	138	132	133	132
query21	210	133	113	113
query22	4422	4264	4287	4264
query23	34229	33498	33404	33404
query24	6730	2435	2464	2435
query25	454	458	401	401
query26	719	275	158	158
query27	2484	517	346	346
query28	2861	2111	2123	2111
query29	591	585	436	436
query30	271	217	196	196
query31	876	872	805	805
query32	74	65	64	64
query33	471	354	316	316
query34	786	858	548	548
query35	791	831	783	783
query36	958	992	879	879
query37	115	120	79	79
query38	4247	4437	4128	4128
query39	1517	1469	1421	1421
query40	229	123	108	108
query41	59	52	53	52
query42	121	108	105	105
query43	509	512	498	498
query44	1377	827	818	818
query45	189	189	174	174
query46	895	1033	671	671
query47	1830	1837	1800	1800
query48	397	428	311	311
query49	680	518	432	432
query50	696	729	417	417
query51	4187	4216	4241	4216
query52	111	112	106	106
query53	243	267	191	191
query54	591	589	511	511
query55	88	80	80	80
query56	311	308	300	300
query57	1127	1194	1139	1139
query58	266	263	266	263
query59	2692	2776	2741	2741
query60	338	325	309	309
query61	130	134	130	130
query62	761	754	729	729
query63	236	195	190	190
query64	1788	1024	683	683
query65	4468	4261	4250	4250
query66	711	408	306	306
query67	15719	15368	15278	15278
query68	7138	902	520	520
query69	536	313	265	265
query70	1227	1135	1119	1119
query71	503	318	296	296
query72	5955	4786	4903	4786
query73	1437	672	355	355
query74	8871	8926	8632	8632
query75	4013	3224	2677	2677
query76	4189	1198	801	801
query77	688	374	293	293
query78	9860	10090	9270	9270
query79	2315	808	579	579
query80	613	529	476	476
query81	482	249	221	221
query82	407	125	94	94
query83	329	263	234	234
query84	300	102	84	84
query85	817	351	314	314
query86	329	305	300	300
query87	4378	4407	4263	4263
query88	2940	2206	2170	2170
query89	409	320	289	289
query90	1951	213	222	213
query91	142	138	109	109
query92	75	61	58	58
query93	1157	943	584	584
query94	681	413	309	309
query95	360	295	289	289
query96	493	564	270	270
query97	3247	3300	3140	3140
query98	227	212	210	210
query99	1439	1391	1258	1258
Total cold run time: 296582 ms
Total hot run time: 192873 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.65 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fe4027884d51b8ee81aa7aed0b080d06fe328861, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.11	0.11
query3	0.26	0.19	0.19
query4	1.59	0.19	0.19
query5	0.57	0.58	0.58
query6	1.17	0.72	0.73
query7	0.03	0.02	0.01
query8	0.04	0.04	0.04
query9	0.57	0.53	0.51
query10	0.56	0.57	0.56
query11	0.17	0.11	0.11
query12	0.15	0.12	0.11
query13	0.61	0.60	0.59
query14	0.78	0.78	0.80
query15	0.87	0.86	0.87
query16	0.37	0.38	0.40
query17	1.02	1.04	1.05
query18	0.21	0.20	0.20
query19	1.87	1.86	1.85
query20	0.02	0.01	0.01
query21	15.41	0.90	0.55
query22	0.77	1.18	0.61
query23	15.00	1.38	0.64
query24	6.86	1.19	1.36
query25	0.53	0.19	0.07
query26	0.53	0.16	0.13
query27	0.05	0.05	0.05
query28	10.34	0.83	0.43
query29	12.54	3.88	3.31
query30	0.24	0.09	0.07
query31	2.84	0.58	0.39
query32	3.22	0.53	0.46
query33	3.05	3.10	3.05
query34	15.73	5.07	4.50
query35	4.48	4.54	4.51
query36	0.66	0.49	0.50
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.76 s
Total hot run time: 29.65 s

@zhaorongsheng
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34298 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4e3c608dd2b26d44d53795b5a9b01cb141524933, data reload: false

------ Round 1 ----------------------------------
q1	25964	5107	4981	4981
q2	2056	278	171	171
q3	10405	1261	681	681
q4	10221	1033	530	530
q5	7523	2290	2392	2290
q6	180	165	138	138
q7	910	760	614	614
q8	9306	1310	1116	1116
q9	6846	5032	4992	4992
q10	6877	2294	1868	1868
q11	486	275	274	274
q12	357	350	211	211
q13	17766	3646	3057	3057
q14	238	224	209	209
q15	529	486	500	486
q16	415	424	371	371
q17	598	856	358	358
q18	7500	7154	7132	7132
q19	1552	933	550	550
q20	322	347	213	213
q21	4028	3317	3098	3098
q22	1033	1007	958	958
Total cold run time: 115112 ms
Total hot run time: 34298 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5061	5017	5054	5017
q2	239	328	230	230
q3	2111	2641	2291	2291
q4	1317	1757	1396	1396
q5	4386	4319	4431	4319
q6	221	169	129	129
q7	2000	1948	1774	1774
q8	2604	2645	2586	2586
q9	7235	7233	7098	7098
q10	2977	3137	2732	2732
q11	581	513	508	508
q12	684	769	638	638
q13	3450	3843	3369	3369
q14	284	314	280	280
q15	517	473	472	472
q16	436	495	458	458
q17	1166	1557	1392	1392
q18	7795	7638	7409	7409
q19	808	843	889	843
q20	2017	1953	1816	1816
q21	5156	4826	4650	4650
q22	1060	1066	1020	1020
Total cold run time: 52105 ms
Total hot run time: 50427 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192182 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4e3c608dd2b26d44d53795b5a9b01cb141524933, data reload: false

query1	1411	1103	1072	1072
query2	6398	1729	1715	1715
query3	11066	4452	4510	4452
query4	53226	25511	23440	23440
query5	4834	577	445	445
query6	366	219	194	194
query7	4904	504	282	282
query8	320	270	228	228
query9	5665	2563	2584	2563
query10	444	314	277	277
query11	15068	14948	14787	14787
query12	166	118	104	104
query13	1061	508	395	395
query14	10197	6522	6431	6431
query15	215	192	187	187
query16	7112	693	518	518
query17	1118	785	575	575
query18	1590	401	316	316
query19	191	195	178	178
query20	141	119	121	119
query21	199	125	103	103
query22	4465	4546	4187	4187
query23	34070	33323	33375	33323
query24	6571	2417	2396	2396
query25	455	473	402	402
query26	697	275	150	150
query27	2385	516	341	341
query28	3066	2135	2120	2120
query29	597	553	424	424
query30	271	213	190	190
query31	883	898	808	808
query32	76	63	60	60
query33	458	357	305	305
query34	912	861	528	528
query35	803	816	729	729
query36	947	984	914	914
query37	123	115	80	80
query38	4189	4274	4176	4176
query39	1492	1449	1441	1441
query40	225	115	106	106
query41	56	51	52	51
query42	115	102	112	102
query43	502	497	464	464
query44	1365	814	814	814
query45	177	176	169	169
query46	841	1010	633	633
query47	1841	1884	1820	1820
query48	382	429	303	303
query49	698	507	420	420
query50	655	700	406	406
query51	4199	4231	4259	4231
query52	111	107	103	103
query53	232	258	191	191
query54	603	594	522	522
query55	81	81	88	81
query56	298	292	300	292
query57	1168	1222	1107	1107
query58	273	269	263	263
query59	2778	2798	2673	2673
query60	330	320	303	303
query61	131	128	130	128
query62	745	732	687	687
query63	219	189	190	189
query64	1802	1050	710	710
query65	4456	4233	4182	4182
query66	713	411	297	297
query67	15586	15507	15369	15369
query68	6874	879	515	515
query69	528	305	267	267
query70	1224	1133	1118	1118
query71	520	309	297	297
query72	5641	4902	4937	4902
query73	1458	686	342	342
query74	9024	9013	8671	8671
query75	4012	3183	2696	2696
query76	4214	1185	758	758
query77	770	369	273	273
query78	9969	10073	9234	9234
query79	2387	810	573	573
query80	631	521	436	436
query81	484	268	220	220
query82	437	125	93	93
query83	294	248	232	232
query84	289	98	85	85
query85	772	364	308	308
query86	360	287	282	282
query87	4423	4309	4308	4308
query88	3502	2244	2292	2244
query89	395	319	275	275
query90	1928	202	205	202
query91	139	147	121	121
query92	74	56	57	56
query93	1776	944	584	584
query94	664	413	302	302
query95	365	306	285	285
query96	487	571	285	285
query97	3155	3252	3155	3155
query98	228	211	204	204
query99	1433	1395	1288	1288
Total cold run time: 297191 ms
Total hot run time: 192182 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.01 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4e3c608dd2b26d44d53795b5a9b01cb141524933, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.10	0.11
query3	0.25	0.20	0.20
query4	1.64	0.18	0.11
query5	0.58	0.56	0.55
query6	1.21	0.71	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.52	0.50
query10	0.57	0.57	0.56
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.60
query14	0.79	0.79	0.80
query15	0.85	0.84	0.87
query16	0.41	0.39	0.39
query17	1.04	1.04	1.02
query18	0.20	0.19	0.20
query19	1.90	1.84	1.79
query20	0.01	0.02	0.01
query21	15.40	0.96	0.54
query22	0.75	1.26	0.65
query23	14.85	1.37	0.63
query24	7.69	0.74	0.72
query25	0.51	0.14	0.15
query26	0.62	0.16	0.12
query27	0.04	0.04	0.04
query28	9.62	0.86	0.46
query29	12.62	3.92	3.34
query30	0.25	0.09	0.06
query31	2.82	0.61	0.39
query32	3.23	0.54	0.46
query33	2.94	3.02	3.09
query34	15.78	5.09	4.49
query35	4.49	4.53	4.51
query36	0.67	0.48	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.02	0.02	0.02
query40	0.17	0.14	0.12
query41	0.09	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 103.93 s
Total hot run time: 29.01 s

@zhaorongsheng
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33888 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2a871a407b31c7583f828d25a8462c07f3b07d61, data reload: false

------ Round 1 ----------------------------------
q1	26076	5117	5023	5023
q2	2056	278	179	179
q3	10438	1246	678	678
q4	10247	1004	519	519
q5	8339	2397	2331	2331
q6	186	175	133	133
q7	908	747	612	612
q8	9317	1295	1089	1089
q9	6879	5169	5113	5113
q10	6809	2340	1878	1878
q11	480	283	261	261
q12	339	341	202	202
q13	17765	3655	3077	3077
q14	229	220	221	220
q15	525	478	478	478
q16	417	428	365	365
q17	604	920	389	389
q18	7587	7129	7206	7129
q19	1217	954	548	548
q20	334	330	242	242
q21	4341	2649	2443	2443
q22	1009	990	979	979
Total cold run time: 116102 ms
Total hot run time: 33888 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5056	5057	5055	5055
q2	241	341	226	226
q3	2113	2601	2291	2291
q4	1341	1772	1317	1317
q5	4580	4421	4403	4403
q6	211	167	126	126
q7	1970	1891	1697	1697
q8	2545	2565	2495	2495
q9	7199	7059	7140	7059
q10	2969	3169	2743	2743
q11	570	510	491	491
q12	654	743	574	574
q13	3491	3950	3212	3212
q14	270	292	261	261
q15	511	488	478	478
q16	438	493	446	446
q17	1129	1557	1368	1368
q18	7561	7410	7446	7410
q19	801	841	896	841
q20	1955	2041	1855	1855
q21	5076	4670	4513	4513
q22	1025	1021	998	998
Total cold run time: 51706 ms
Total hot run time: 49859 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184335 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2a871a407b31c7583f828d25a8462c07f3b07d61, data reload: false

query1	1003	472	473	472
query2	6559	1840	1863	1840
query3	6747	224	224	224
query4	26348	23493	22967	22967
query5	4347	644	454	454
query6	316	209	173	173
query7	4619	474	272	272
query8	287	238	225	225
query9	8611	2524	2513	2513
query10	470	323	260	260
query11	15562	15190	14719	14719
query12	161	111	118	111
query13	1646	531	380	380
query14	9146	6107	5962	5962
query15	193	190	162	162
query16	7165	643	506	506
query17	1209	748	579	579
query18	1997	401	316	316
query19	196	190	157	157
query20	125	112	119	112
query21	218	124	107	107
query22	4037	4187	4068	4068
query23	33846	33017	32664	32664
query24	8464	2314	2361	2314
query25	523	440	375	375
query26	1221	260	204	204
query27	2711	495	330	330
query28	4404	2069	2048	2048
query29	749	535	418	418
query30	278	215	185	185
query31	932	838	745	745
query32	78	60	62	60
query33	558	372	301	301
query34	783	847	523	523
query35	771	811	722	722
query36	958	992	874	874
query37	112	97	75	75
query38	4010	4166	4051	4051
query39	1432	1388	1392	1388
query40	209	116	107	107
query41	58	53	51	51
query42	121	104	103	103
query43	506	492	457	457
query44	1259	801	787	787
query45	175	174	164	164
query46	826	1008	652	652
query47	1769	1791	1735	1735
query48	372	419	319	319
query49	755	494	422	422
query50	658	683	387	387
query51	4139	4107	4069	4069
query52	106	102	95	95
query53	216	256	184	184
query54	573	556	498	498
query55	79	82	82	82
query56	285	316	298	298
query57	1130	1129	1108	1108
query58	256	240	242	240
query59	2651	2680	2514	2514
query60	311	314	304	304
query61	127	131	122	122
query62	820	722	652	652
query63	224	179	182	179
query64	4366	1014	657	657
query65	4268	4247	4255	4247
query66	1141	403	308	308
query67	15717	15408	15243	15243
query68	7530	870	504	504
query69	475	294	257	257
query70	1206	1086	1118	1086
query71	404	305	298	298
query72	5561	4625	4607	4607
query73	619	561	334	334
query74	8860	9279	8884	8884
query75	3133	3204	2686	2686
query76	3181	1169	733	733
query77	469	377	274	274
query78	9796	10197	9321	9321
query79	1446	846	559	559
query80	615	558	465	465
query81	491	261	224	224
query82	181	129	100	100
query83	259	262	239	239
query84	259	110	91	91
query85	829	480	299	299
query86	372	299	275	275
query87	4364	4451	4372	4372
query88	2830	2212	2243	2212
query89	381	305	278	278
query90	1922	212	210	210
query91	141	134	111	111
query92	75	59	58	58
query93	1399	923	576	576
query94	674	414	293	293
query95	371	289	284	284
query96	479	560	273	273
query97	3055	3176	3111	3111
query98	229	214	207	207
query99	1336	1389	1274	1274
Total cold run time: 268851 ms
Total hot run time: 184335 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.37 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2a871a407b31c7583f828d25a8462c07f3b07d61, data reload: false

query1	0.03	0.03	0.04
query2	0.12	0.11	0.11
query3	0.25	0.20	0.19
query4	1.59	0.20	0.20
query5	0.60	0.60	0.59
query6	1.21	0.73	0.71
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.56	0.52	0.51
query10	0.56	0.57	0.57
query11	0.16	0.10	0.11
query12	0.15	0.12	0.12
query13	0.62	0.61	0.61
query14	0.77	0.83	0.81
query15	0.88	0.84	0.85
query16	0.38	0.40	0.38
query17	1.05	1.03	1.04
query18	0.21	0.20	0.20
query19	1.88	1.76	1.79
query20	0.01	0.02	0.01
query21	15.39	0.91	0.54
query22	0.75	1.18	0.72
query23	14.89	1.38	0.64
query24	7.48	1.30	0.88
query25	0.47	0.25	0.14
query26	0.61	0.16	0.13
query27	0.04	0.05	0.05
query28	9.39	0.86	0.43
query29	12.53	3.88	3.26
query30	0.25	0.09	0.06
query31	2.84	0.59	0.38
query32	3.23	0.55	0.47
query33	2.98	3.10	3.11
query34	15.66	5.08	4.50
query35	4.48	4.51	4.46
query36	0.66	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.04	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.25 s
Total hot run time: 29.37 s

@zhaorongsheng
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33637 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2a871a407b31c7583f828d25a8462c07f3b07d61, data reload: false

------ Round 1 ----------------------------------
q1	26040	5063	4969	4969
q2	2079	285	177	177
q3	10402	1214	690	690
q4	10221	1000	554	554
q5	7563	2301	2368	2301
q6	178	169	135	135
q7	933	751	610	610
q8	9307	1283	1087	1087
q9	6834	5048	5088	5048
q10	6873	2265	1871	1871
q11	472	287	271	271
q12	346	352	204	204
q13	17783	3661	3054	3054
q14	224	221	220	220
q15	534	491	490	490
q16	421	428	369	369
q17	600	837	367	367
q18	7657	7025	7069	7025
q19	1307	930	578	578
q20	346	328	225	225
q21	4462	3382	2440	2440
q22	1044	1019	952	952
Total cold run time: 115626 ms
Total hot run time: 33637 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5074	5058	5054	5054
q2	234	321	232	232
q3	2162	2652	2316	2316
q4	1356	1773	1414	1414
q5	4388	4397	4402	4397
q6	216	169	129	129
q7	2040	1919	1776	1776
q8	2567	2621	2550	2550
q9	7176	7206	7121	7121
q10	2999	3196	2757	2757
q11	580	529	487	487
q12	666	755	637	637
q13	3444	3860	3253	3253
q14	278	327	306	306
q15	531	477	463	463
q16	437	483	439	439
q17	1155	1539	1386	1386
q18	7835	7585	7401	7401
q19	796	832	912	832
q20	2012	1996	1834	1834
q21	5025	4764	4757	4757
q22	1058	1062	1004	1004
Total cold run time: 52029 ms
Total hot run time: 50545 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191569 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2a871a407b31c7583f828d25a8462c07f3b07d61, data reload: false

query1	1391	1087	1051	1051
query2	6465	1750	1774	1750
query3	11028	4476	4558	4476
query4	54853	25625	22959	22959
query5	5163	468	449	449
query6	360	187	179	179
query7	4928	506	280	280
query8	325	260	242	242
query9	5939	2532	2550	2532
query10	451	323	262	262
query11	15070	15050	14950	14950
query12	177	110	102	102
query13	1057	488	374	374
query14	10096	6289	6218	6218
query15	195	202	170	170
query16	7055	655	516	516
query17	1090	746	595	595
query18	1545	427	322	322
query19	208	218	180	180
query20	133	125	127	125
query21	209	129	113	113
query22	4353	4442	4273	4273
query23	34218	33236	33301	33236
query24	6568	2431	2425	2425
query25	453	454	405	405
query26	686	277	153	153
query27	2300	500	335	335
query28	3311	2093	2065	2065
query29	589	548	421	421
query30	275	210	192	192
query31	832	875	797	797
query32	70	70	61	61
query33	459	363	313	313
query34	781	879	525	525
query35	789	829	781	781
query36	972	985	897	897
query37	109	102	82	82
query38	4115	4194	4174	4174
query39	1490	1439	1453	1439
query40	217	117	109	109
query41	55	57	53	53
query42	124	108	112	108
query43	501	513	472	472
query44	1303	817	813	813
query45	185	178	170	170
query46	830	1034	639	639
query47	1843	1884	1800	1800
query48	396	415	336	336
query49	703	500	425	425
query50	673	692	401	401
query51	4237	4215	4282	4215
query52	110	116	110	110
query53	242	262	189	189
query54	580	572	512	512
query55	81	81	80	80
query56	312	298	289	289
query57	1171	1201	1151	1151
query58	266	247	257	247
query59	2664	2721	2589	2589
query60	315	331	300	300
query61	133	134	126	126
query62	726	723	715	715
query63	235	190	203	190
query64	1594	1012	686	686
query65	4452	4362	4237	4237
query66	722	399	300	300
query67	15988	15599	15295	15295
query68	5501	877	508	508
query69	506	310	271	271
query70	1220	1084	1105	1084
query71	436	312	290	290
query72	5783	4964	5017	4964
query73	1289	711	349	349
query74	9137	9159	8674	8674
query75	3442	3170	2655	2655
query76	3657	1188	755	755
query77	546	370	282	282
query78	10277	10174	9305	9305
query79	1726	862	558	558
query80	636	517	431	431
query81	475	255	223	223
query82	339	128	96	96
query83	248	246	261	246
query84	295	96	87	87
query85	744	347	302	302
query86	336	311	302	302
query87	4384	4425	4338	4338
query88	2788	2210	2198	2198
query89	403	315	281	281
query90	1857	209	205	205
query91	152	137	107	107
query92	65	59	56	56
query93	2578	952	566	566
query94	657	397	305	305
query95	366	287	275	275
query96	486	567	270	270
query97	3150	3215	3113	3113
query98	221	213	197	197
query99	1368	1382	1300	1300
Total cold run time: 295834 ms
Total hot run time: 191569 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.8 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2a871a407b31c7583f828d25a8462c07f3b07d61, data reload: false

query1	0.04	0.03	0.03
query2	0.13	0.10	0.11
query3	0.26	0.19	0.19
query4	1.59	0.20	0.11
query5	0.56	0.55	0.56
query6	1.19	0.71	0.72
query7	0.03	0.02	0.01
query8	0.04	0.04	0.04
query9	0.58	0.52	0.50
query10	0.57	0.57	0.56
query11	0.16	0.10	0.11
query12	0.15	0.11	0.11
query13	0.61	0.59	0.60
query14	0.78	0.79	0.79
query15	0.89	0.84	0.87
query16	0.38	0.38	0.38
query17	1.00	1.02	1.06
query18	0.21	0.21	0.21
query19	1.94	1.79	1.80
query20	0.01	0.01	0.02
query21	15.39	0.88	0.54
query22	0.78	1.29	0.73
query23	14.74	1.41	0.61
query24	7.24	1.15	0.65
query25	0.53	0.20	0.08
query26	0.58	0.16	0.15
query27	0.06	0.05	0.05
query28	9.37	0.91	0.43
query29	12.59	3.89	3.26
query30	0.26	0.09	0.06
query31	2.82	0.57	0.38
query32	3.22	0.56	0.45
query33	3.10	3.06	3.03
query34	15.84	5.06	4.47
query35	4.56	4.53	4.49
query36	0.65	0.50	0.47
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 103.31 s
Total hot run time: 28.8 s

924060929
924060929 previously approved these changes May 8, 2025
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 8, 2025
Copy link
Contributor

github-actions bot commented May 8, 2025

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented May 8, 2025

PR approved by anyone and no changes requested.

@924060929 924060929 changed the title [fix](scan) Fix Invalid Partition Number About Specified Tablets Query [opt](nereids) support prune partitions by specified tablet ids May 8, 2025
Comment on lines +187 to +189
if (!selectedPartitions.isEmpty() && !selectedPartitions.contains(partition.getId())) {
continue;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove local variable selectedPartitions and

Suggested change
if (!selectedPartitions.isEmpty() && !selectedPartitions.contains(partition.getId())) {
continue;
}
if (prunedPartitionsByFilters != null && !prunedPartitionsByFilters.contains(partition.getId()))) {
continue;
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The type of prunedPartitionsByFilters is List. The function contains() may cost too much if the list's length is high.

@morrySnow morrySnow requested a review from gavinchou May 8, 2025 12:40
…e/PruneOlapScanPartition.java

Co-authored-by: morrySnow <morrysnow@126.com>
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label May 9, 2025
Copy link
Contributor

github-actions bot commented May 9, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 9, 2025
@924060929
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33431 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 68423e239aa934a9a0f78df13109ac6fa5db149b, data reload: false

------ Round 1 ----------------------------------
q1	25716	5026	4937	4937
q2	2058	264	193	193
q3	10404	1196	670	670
q4	10225	975	505	505
q5	7539	2198	2337	2198
q6	172	163	131	131
q7	877	753	612	612
q8	9315	1270	1120	1120
q9	6858	5007	5098	5007
q10	6812	2280	1858	1858
q11	468	272	253	253
q12	357	343	208	208
q13	17761	3642	3093	3093
q14	222	231	205	205
q15	518	495	469	469
q16	409	416	370	370
q17	589	840	358	358
q18	7434	7207	7138	7138
q19	1432	931	542	542
q20	328	322	222	222
q21	3875	3684	2405	2405
q22	1046	1005	937	937
Total cold run time: 114415 ms
Total hot run time: 33431 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5084	4989	4989	4989
q2	247	320	224	224
q3	2120	2621	2271	2271
q4	1343	1758	1379	1379
q5	4429	4410	4429	4410
q6	207	173	125	125
q7	1976	1948	1725	1725
q8	2535	2606	2453	2453
q9	7148	7144	6900	6900
q10	2982	3175	2739	2739
q11	564	504	487	487
q12	647	778	609	609
q13	3441	3865	3244	3244
q14	284	288	276	276
q15	528	474	476	474
q16	446	482	468	468
q17	1120	1534	1348	1348
q18	7689	7502	7378	7378
q19	787	808	947	808
q20	1974	2068	1924	1924
q21	5084	4749	4649	4649
q22	1109	1041	1003	1003
Total cold run time: 51744 ms
Total hot run time: 49883 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191643 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 68423e239aa934a9a0f78df13109ac6fa5db149b, data reload: false

query1	1370	1071	1058	1058
query2	6342	1708	1746	1708
query3	11013	4512	4372	4372
query4	53985	24075	23259	23259
query5	5197	476	438	438
query6	353	201	194	194
query7	5037	490	275	275
query8	330	246	232	232
query9	6582	2555	2565	2555
query10	408	346	258	258
query11	15227	15034	14748	14748
query12	154	118	103	103
query13	1082	509	380	380
query14	10660	6202	6192	6192
query15	187	208	187	187
query16	6968	648	510	510
query17	1100	775	612	612
query18	1522	418	317	317
query19	205	197	177	177
query20	126	133	121	121
query21	207	124	109	109
query22	4419	4418	4301	4301
query23	34060	33427	33402	33402
query24	6003	2418	2383	2383
query25	452	492	415	415
query26	736	272	149	149
query27	2010	479	346	346
query28	2954	2140	2121	2121
query29	567	600	425	425
query30	281	219	192	192
query31	842	864	766	766
query32	73	69	63	63
query33	467	353	317	317
query34	774	851	549	549
query35	795	848	759	759
query36	943	992	898	898
query37	117	99	79	79
query38	4211	4129	4102	4102
query39	1474	1441	1427	1427
query40	212	119	103	103
query41	57	53	58	53
query42	121	111	109	109
query43	505	509	462	462
query44	1301	799	810	799
query45	179	176	175	175
query46	828	1042	646	646
query47	1779	1891	1812	1812
query48	374	413	305	305
query49	688	517	406	406
query50	647	699	398	398
query51	4232	4178	4141	4141
query52	109	106	106	106
query53	217	250	179	179
query54	572	589	514	514
query55	79	82	79	79
query56	332	323	283	283
query57	1173	1191	1152	1152
query58	267	264	288	264
query59	2667	2709	2745	2709
query60	338	336	319	319
query61	138	131	136	131
query62	737	719	660	660
query63	224	186	185	185
query64	1793	1064	714	714
query65	4412	4223	4234	4223
query66	724	448	305	305
query67	15701	15579	15432	15432
query68	7207	829	511	511
query69	548	299	256	256
query70	1202	1090	1103	1090
query71	504	304	294	294
query72	5890	4851	5080	4851
query73	1202	689	337	337
query74	9246	9077	8782	8782
query75	3764	3183	2693	2693
query76	4330	1194	755	755
query77	609	414	269	269
query78	10021	9994	9341	9341
query79	3394	799	548	548
query80	654	506	437	437
query81	487	246	214	214
query82	482	123	96	96
query83	342	245	224	224
query84	290	104	77	77
query85	812	356	314	314
query86	402	309	298	298
query87	4369	4415	4259	4259
query88	3251	2173	2193	2173
query89	410	304	274	274
query90	1920	204	206	204
query91	143	143	109	109
query92	74	59	54	54
query93	2412	914	576	576
query94	670	403	300	300
query95	360	289	274	274
query96	520	556	269	269
query97	3134	3228	3113	3113
query98	219	207	199	199
query99	1418	1387	1289	1289
Total cold run time: 299925 ms
Total hot run time: 191643 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.42 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 68423e239aa934a9a0f78df13109ac6fa5db149b, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.10	0.12
query3	0.25	0.19	0.19
query4	1.60	0.19	0.19
query5	0.59	0.58	0.58
query6	1.20	0.73	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.57	0.51	0.50
query10	0.58	0.58	0.57
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.60	0.60	0.59
query14	0.77	0.78	0.78
query15	0.86	0.84	0.86
query16	0.37	0.39	0.38
query17	1.00	1.06	1.03
query18	0.20	0.20	0.20
query19	1.84	1.80	1.79
query20	0.02	0.01	0.01
query21	15.41	0.92	0.56
query22	0.77	1.28	0.76
query23	14.74	1.36	0.62
query24	6.91	0.97	1.38
query25	0.46	0.30	0.06
query26	0.50	0.17	0.14
query27	0.05	0.05	0.05
query28	10.15	0.85	0.43
query29	12.52	4.03	3.40
query30	0.25	0.10	0.07
query31	2.82	0.57	0.37
query32	3.22	0.54	0.46
query33	3.11	3.01	3.08
query34	15.76	5.04	4.48
query35	4.46	4.47	4.51
query36	0.66	0.49	0.50
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.23 s
Total hot run time: 29.42 s

@924060929 924060929 merged commit d0731c8 into apache:master May 12, 2025
26 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] It does not prune partitions about querying one tablet of the table
5 participants