Skip to content

[Feature](MTMV) add mv dependence system table #50411

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

qzsee
Copy link
Contributor

@qzsee qzsee commented Apr 25, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

A system table has been added, through which the dependencies of the materialized view can be viewed

mysql> select * from information_schema.mv_dependence where src_database = 'test_mv_dependence_db' order by src_catalog,src_database,src_table;
+-------------+-----------------------+-----------+--------------+---------------+------------+-------------------+
| SRC_CATALOG | SRC_DATABASE          | SRC_TABLE | FROM_CATALOG | FROM_DATABASE | FROM_TABLE | FROM_TABLE_TYPE   |
+-------------+-----------------------+-----------+--------------+---------------+------------+-------------------+
| internal    | test_mv_dependence_db | mv_a      | internal     | test          | stu        | OLAP              |
| internal    | test_mv_dependence_db | mv_b      | internal     | test          | mv_a       | MATERIALIZED_VIEW |
| internal    | test_mv_dependence_db | mv_c      | internal     | test          | stu        | OLAP              |
| internal    | test_mv_dependence_db | mv_c      | internal     | test          | grade      | OLAP              |
+-------------+-----------------------+-----------+--------------+---------------+------------+-------------------+
4 rows in set (0.02 sec)

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dqz123
Copy link

dqz123 commented Apr 25, 2025

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 83.06% (1098/1322)
Line Coverage: 65.87% (18378/27900)
Region Coverage: 65.31% (9109/13948)
Branch Coverage: 55.30% (4903/8866)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d3c06d78c2417b49ae6f6600f61dba939dd148e5_d3c06d78c2417b49ae6f6600f61dba939dd148e5_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 34437 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d3c06d78c2417b49ae6f6600f61dba939dd148e5, data reload: false

------ Round 1 ----------------------------------
q1	26604	5047	5060	5047
q2	2094	303	198	198
q3	10348	1300	715	715
q4	10239	1016	567	567
q5	7510	2434	2412	2412
q6	179	164	133	133
q7	950	756	619	619
q8	9335	1352	1135	1135
q9	6832	5118	5142	5118
q10	6895	2317	1875	1875
q11	497	284	278	278
q12	364	366	229	229
q13	17779	3689	3130	3130
q14	223	233	220	220
q15	524	501	507	501
q16	464	458	401	401
q17	622	876	388	388
q18	7663	7231	7172	7172
q19	1366	969	591	591
q20	344	340	228	228
q21	4337	3389	2507	2507
q22	1057	1021	973	973
Total cold run time: 116226 ms
Total hot run time: 34437 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5179	5429	5499	5429
q2	240	333	243	243
q3	2158	2650	2267	2267
q4	1503	1904	1498	1498
q5	4505	4424	4472	4424
q6	215	167	128	128
q7	2001	1924	1774	1774
q8	2634	2681	2604	2604
q9	7324	7085	7218	7085
q10	3019	3197	2798	2798
q11	592	502	500	500
q12	711	776	637	637
q13	3531	3943	3388	3388
q14	295	292	260	260
q15	547	478	473	473
q16	470	523	477	477
q17	1189	1594	1402	1402
q18	7838	7659	7470	7470
q19	907	881	957	881
q20	1915	2012	1891	1891
q21	5385	4917	4895	4895
q22	1135	1060	1034	1034
Total cold run time: 53293 ms
Total hot run time: 51558 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192902 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d3c06d78c2417b49ae6f6600f61dba939dd148e5, data reload: false

query1	1413	1094	1068	1068
query2	6276	1925	1920	1920
query3	11030	4512	4286	4286
query4	56077	25608	23356	23356
query5	5031	606	456	456
query6	330	204	190	190
query7	4893	541	291	291
query8	318	255	242	242
query9	5465	2596	2584	2584
query10	475	334	261	261
query11	15269	15050	14750	14750
query12	163	110	107	107
query13	1033	533	403	403
query14	10005	6292	6233	6233
query15	204	207	176	176
query16	7142	665	495	495
query17	1089	747	555	555
query18	1675	411	303	303
query19	218	184	164	164
query20	135	130	120	120
query21	208	124	112	112
query22	4470	4520	4361	4361
query23	34178	33397	33476	33397
query24	6553	2472	2482	2472
query25	496	503	396	396
query26	690	277	154	154
query27	2180	504	336	336
query28	2976	2177	2184	2177
query29	575	574	462	462
query30	273	224	203	203
query31	899	857	785	785
query32	75	67	69	67
query33	459	391	323	323
query34	808	863	535	535
query35	770	833	767	767
query36	968	1018	885	885
query37	115	102	83	83
query38	4189	4288	4197	4197
query39	1468	1440	1419	1419
query40	212	124	133	124
query41	56	53	52	52
query42	126	107	107	107
query43	505	531	517	517
query44	1315	825	813	813
query45	176	172	169	169
query46	858	1047	650	650
query47	1878	1867	1823	1823
query48	398	428	306	306
query49	700	508	439	439
query50	678	710	420	420
query51	4228	4248	4211	4211
query52	112	108	96	96
query53	247	263	188	188
query54	588	580	507	507
query55	82	82	76	76
query56	325	303	279	279
query57	1178	1189	1172	1172
query58	268	269	261	261
query59	2835	2927	2793	2793
query60	368	352	340	340
query61	154	152	155	152
query62	732	758	682	682
query63	241	195	202	195
query64	2030	1152	700	700
query65	4386	4266	4255	4255
query66	721	403	303	303
query67	15770	15652	15371	15371
query68	7298	900	520	520
query69	538	302	266	266
query70	1168	1091	1081	1081
query71	493	313	305	305
query72	5903	4807	4933	4807
query73	1492	685	356	356
query74	9061	9189	8946	8946
query75	4006	3196	2722	2722
query76	4259	1206	758	758
query77	643	375	276	276
query78	10219	10030	9298	9298
query79	4586	794	549	549
query80	643	503	420	420
query81	489	256	214	214
query82	521	132	103	103
query83	334	251	237	237
query84	298	105	85	85
query85	768	352	319	319
query86	346	307	281	281
query87	4402	4398	4356	4356
query88	3747	2258	2236	2236
query89	442	312	284	284
query90	1888	210	210	210
query91	136	138	127	127
query92	73	59	57	57
query93	2888	943	573	573
query94	665	415	310	310
query95	376	294	277	277
query96	489	573	272	272
query97	3149	3282	3115	3115
query98	251	203	199	199
query99	1483	1392	1251	1251
Total cold run time: 304812 ms
Total hot run time: 192902 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.23 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d3c06d78c2417b49ae6f6600f61dba939dd148e5, data reload: false

query1	0.04	0.04	0.03
query2	0.14	0.11	0.11
query3	0.34	0.20	0.20
query4	1.60	0.20	0.20
query5	0.61	0.60	0.62
query6	1.17	0.73	0.72
query7	0.02	0.02	0.01
query8	0.06	0.04	0.05
query9	0.63	0.52	0.52
query10	0.58	0.58	0.57
query11	0.26	0.12	0.13
query12	0.26	0.13	0.14
query13	0.64	0.63	0.60
query14	1.22	1.22	1.23
query15	0.96	0.87	0.86
query16	0.36	0.37	0.38
query17	1.08	1.05	1.04
query18	0.19	0.19	0.19
query19	1.94	1.77	1.82
query20	0.02	0.02	0.01
query21	15.39	0.97	0.67
query22	0.93	1.05	0.82
query23	14.70	1.54	0.76
query24	5.47	0.56	0.28
query25	0.16	0.10	0.09
query26	0.55	0.22	0.19
query27	0.10	0.09	0.08
query28	11.02	1.19	0.56
query29	12.56	4.01	3.40
query30	0.30	0.09	0.06
query31	2.80	0.62	0.42
query32	3.24	0.59	0.50
query33	3.03	3.12	3.07
query34	16.70	5.12	4.47
query35	4.47	4.49	4.38
query36	0.64	0.50	0.50
query37	0.19	0.17	0.17
query38	0.18	0.17	0.15
query39	0.05	0.04	0.04
query40	0.19	0.16	0.16
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.05	0.05
Total cold run time: 105 s
Total hot run time: 30.23 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 0.00% (0/78) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.75% (14595/27153)
Line Coverage 42.58% (126691/297534)
Region Coverage 41.37% (64768/156539)
Branch Coverage 35.94% (32561/90604)

@hello-stephen
Copy link
Contributor

BE Regression P0 && UT Coverage Report

Increment line coverage 0.00% (0/78) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 55.18% (14713/26663)
Line Coverage 44.63% (132579/297045)
Region Coverage 41.71% (76315/182987)
Branch Coverage 35.82% (36953/103168)

@dqz123
Copy link

dqz123 commented Apr 25, 2025

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 83.06% (1098/1322)
Line Coverage: 65.89% (18389/27909)
Region Coverage: 65.31% (9114/13954)
Branch Coverage: 55.30% (4905/8870)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a000ceac66f2e0e027d15cf31ecd664ade250e81_a000ceac66f2e0e027d15cf31ecd664ade250e81_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 34133 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a000ceac66f2e0e027d15cf31ecd664ade250e81, data reload: false

------ Round 1 ----------------------------------
q1	26830	5084	5293	5084
q2	2077	295	211	211
q3	10353	1263	745	745
q4	10231	1006	529	529
q5	7539	2370	2328	2328
q6	181	163	135	135
q7	932	777	610	610
q8	9317	1235	1112	1112
q9	6955	5192	5133	5133
q10	6819	2323	1877	1877
q11	473	286	265	265
q12	343	368	227	227
q13	17751	3726	3129	3129
q14	226	228	212	212
q15	517	501	486	486
q16	452	442	412	412
q17	591	851	382	382
q18	7546	7056	7148	7056
q19	1099	962	552	552
q20	346	330	237	237
q21	4310	3435	2451	2451
q22	1038	1006	960	960
Total cold run time: 115926 ms
Total hot run time: 34133 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5100	5114	5078	5078
q2	236	329	237	237
q3	2174	2617	2297	2297
q4	1433	1862	1513	1513
q5	4515	4436	4460	4436
q6	212	169	131	131
q7	1985	1941	1778	1778
q8	2638	2513	2568	2513
q9	7336	7211	7123	7123
q10	3004	3142	2753	2753
q11	571	513	503	503
q12	689	770	591	591
q13	3587	3892	3317	3317
q14	314	293	267	267
q15	515	466	463	463
q16	466	520	475	475
q17	1157	1576	1396	1396
q18	7787	7487	7520	7487
q19	783	884	1016	884
q20	2003	1962	1858	1858
q21	5346	5019	4769	4769
q22	1074	1058	1031	1031
Total cold run time: 52925 ms
Total hot run time: 50900 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191898 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a000ceac66f2e0e027d15cf31ecd664ade250e81, data reload: false

query1	1424	1072	1059	1059
query2	6059	1891	1881	1881
query3	11021	4462	4549	4462
query4	53505	24660	23056	23056
query5	5370	478	459	459
query6	414	190	178	178
query7	5236	491	277	277
query8	326	230	234	230
query9	7281	2574	2567	2567
query10	422	317	281	281
query11	15163	15000	14778	14778
query12	162	112	105	105
query13	1271	510	396	396
query14	9942	6277	6188	6188
query15	217	191	188	188
query16	7069	661	488	488
query17	1084	724	592	592
query18	1552	407	336	336
query19	194	188	165	165
query20	124	129	124	124
query21	205	130	107	107
query22	4391	4575	4386	4386
query23	34108	33251	33449	33251
query24	6500	2395	2380	2380
query25	454	452	416	416
query26	670	278	156	156
query27	2215	511	338	338
query28	3364	2151	2197	2151
query29	587	580	452	452
query30	271	227	196	196
query31	870	862	765	765
query32	79	65	68	65
query33	507	391	309	309
query34	772	843	525	525
query35	772	838	740	740
query36	984	1002	900	900
query37	116	100	80	80
query38	4156	4257	4270	4257
query39	1476	1463	1442	1442
query40	226	125	110	110
query41	61	53	51	51
query42	121	114	101	101
query43	500	526	498	498
query44	1308	809	813	809
query45	188	185	169	169
query46	852	1025	644	644
query47	1847	1863	1772	1772
query48	378	413	300	300
query49	708	521	423	423
query50	668	686	403	403
query51	4187	4229	4191	4191
query52	114	110	95	95
query53	225	260	188	188
query54	578	574	501	501
query55	84	82	81	81
query56	294	295	285	285
query57	1162	1164	1127	1127
query58	271	266	255	255
query59	2805	2770	2667	2667
query60	321	310	296	296
query61	121	122	120	120
query62	740	749	687	687
query63	230	183	177	177
query64	1441	1000	687	687
query65	4337	4226	4226	4226
query66	722	390	303	303
query67	15704	15713	15541	15541
query68	7571	879	508	508
query69	543	288	264	264
query70	1214	1041	1148	1041
query71	490	312	290	290
query72	5541	4800	5015	4800
query73	1169	670	350	350
query74	9353	9122	8690	8690
query75	3816	3213	2687	2687
query76	4267	1205	758	758
query77	636	363	283	283
query78	10107	10092	9291	9291
query79	3123	834	570	570
query80	637	528	436	436
query81	491	257	219	219
query82	501	131	97	97
query83	388	250	244	244
query84	292	108	85	85
query85	803	352	316	316
query86	423	287	318	287
query87	4437	4346	4325	4325
query88	3450	2223	2217	2217
query89	400	317	281	281
query90	1950	218	220	218
query91	139	143	109	109
query92	72	63	58	58
query93	2600	957	582	582
query94	651	369	264	264
query95	375	294	279	279
query96	490	559	277	277
query97	3149	3236	3119	3119
query98	232	210	212	210
query99	1519	1435	1310	1310
Total cold run time: 301355 ms
Total hot run time: 191898 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.44 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a000ceac66f2e0e027d15cf31ecd664ade250e81, data reload: false

query1	0.04	0.04	0.02
query2	0.14	0.11	0.11
query3	0.35	0.19	0.20
query4	1.59	0.20	0.09
query5	0.54	0.55	0.55
query6	1.19	0.73	0.73
query7	0.02	0.02	0.01
query8	0.06	0.05	0.05
query9	0.62	0.54	0.53
query10	0.58	0.57	0.59
query11	0.25	0.13	0.13
query12	0.25	0.14	0.13
query13	0.64	0.63	0.62
query14	1.21	1.26	1.22
query15	0.97	0.87	0.89
query16	0.38	0.38	0.39
query17	1.04	1.05	1.01
query18	0.24	0.22	0.21
query19	1.98	1.87	1.83
query20	0.01	0.02	0.01
query21	15.38	0.99	0.67
query22	0.93	1.08	0.89
query23	14.68	1.55	0.75
query24	5.45	0.58	0.29
query25	0.16	0.09	0.08
query26	0.54	0.21	0.19
query27	0.09	0.08	0.08
query28	10.97	1.15	0.60
query29	12.53	4.06	3.46
query30	0.30	0.09	0.07
query31	2.81	0.65	0.42
query32	3.24	0.59	0.52
query33	3.20	3.12	3.09
query34	16.42	5.13	4.44
query35	4.43	4.50	4.46
query36	0.64	0.51	0.50
query37	0.20	0.18	0.18
query38	0.17	0.16	0.15
query39	0.04	0.04	0.04
query40	0.21	0.15	0.17
query41	0.11	0.05	0.06
query42	0.06	0.05	0.05
query43	0.06	0.05	0.05
Total cold run time: 104.72 s
Total hot run time: 30.44 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 0.00% (0/78) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 54.13% (14684/27129)
Line Coverage 43.00% (127876/297420)
Region Coverage 41.84% (65479/156481)
Branch Coverage 36.45% (33016/90582)

@hello-stephen
Copy link
Contributor

BE Regression P0 && UT Coverage Report

Increment line coverage 0.00% (0/78) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 54.98% (14643/26635)
Line Coverage 44.49% (132092/296928)
Region Coverage 41.52% (75959/182924)
Branch Coverage 35.63% (36747/103144)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants