Skip to content

[opt](nereids) avoid printing duplicated table stats in profile #51354

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

englefly
Copy link
Contributor

What problem does this PR solve?

Query related tables are collected by function org.apache.doris.common.profile.Profile#updateSummary().
This function is invoked multiple times during a query execution. And hence, tables are added to Profile.physicalRelations multiple times.
In this pr, Profile.physicalRelations is defined as Set instead of List to remove duplications.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@englefly
Copy link
Contributor Author

run buildall

@Thearas
Copy link
Contributor

Thearas commented May 29, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@doris-robot
Copy link

TPC-H: Total hot run time: 34049 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a5b1565b7c283492388d391c692281b402b9a1b5, data reload: false

------ Round 1 ----------------------------------
q1	26135	5097	5009	5009
q2	1944	286	182	182
q3	10294	1278	680	680
q4	10228	1017	546	546
q5	7532	2372	2331	2331
q6	179	169	135	135
q7	900	746	621	621
q8	9290	1280	1094	1094
q9	6841	5115	5076	5076
q10	6851	2325	1881	1881
q11	497	288	281	281
q12	341	358	229	229
q13	17907	3775	3113	3113
q14	241	249	212	212
q15	575	473	499	473
q16	432	444	374	374
q17	589	869	364	364
q18	7784	7257	7432	7257
q19	1674	993	592	592
q20	357	349	239	239
q21	4147	3197	2413	2413
q22	1048	1026	947	947
Total cold run time: 115786 ms
Total hot run time: 34049 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5261	5149	5185	5149
q2	233	329	221	221
q3	2194	2671	2332	2332
q4	1371	1825	1408	1408
q5	4496	4489	4430	4430
q6	230	174	137	137
q7	2021	1912	1768	1768
q8	2610	2573	2568	2568
q9	7214	7349	7303	7303
q10	3064	3269	2883	2883
q11	585	532	495	495
q12	726	793	632	632
q13	3876	4266	3553	3553
q14	322	307	280	280
q15	523	473	480	473
q16	425	493	451	451
q17	1185	1575	1413	1413
q18	7939	7563	7474	7474
q19	1061	886	954	886
q20	2004	2067	1947	1947
q21	4738	4429	4323	4323
q22	1103	1067	1028	1028
Total cold run time: 53181 ms
Total hot run time: 51154 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192547 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a5b1565b7c283492388d391c692281b402b9a1b5, data reload: false

query1	1388	1080	1088	1080
query2	6448	1792	1776	1776
query3	11028	4485	4398	4398
query4	52587	24154	23005	23005
query5	5121	610	451	451
query6	327	205	231	205
query7	4865	535	292	292
query8	282	233	207	207
query9	5214	2639	2631	2631
query10	424	326	251	251
query11	15232	14978	14949	14949
query12	164	117	116	116
query13	1035	557	425	425
query14	10093	6371	6348	6348
query15	221	197	179	179
query16	7040	647	519	519
query17	1098	756	632	632
query18	1568	424	324	324
query19	201	219	171	171
query20	131	132	171	132
query21	212	125	106	106
query22	4390	4437	4281	4281
query23	34280	33688	33625	33625
query24	6661	2432	2410	2410
query25	483	472	407	407
query26	704	283	150	150
query27	2321	522	349	349
query28	3035	2185	2160	2160
query29	573	570	430	430
query30	274	219	188	188
query31	879	872	767	767
query32	72	69	62	62
query33	460	354	321	321
query34	797	866	562	562
query35	798	839	748	748
query36	951	993	888	888
query37	114	105	75	75
query38	4233	4321	4305	4305
query39	1532	1469	1439	1439
query40	215	120	108	108
query41	58	57	55	55
query42	137	116	111	111
query43	511	513	481	481
query44	1321	833	852	833
query45	197	183	173	173
query46	935	1051	661	661
query47	1869	1846	1808	1808
query48	412	441	328	328
query49	651	498	392	392
query50	668	703	457	457
query51	4243	4226	4285	4226
query52	117	100	103	100
query53	229	268	186	186
query54	584	588	518	518
query55	86	85	86	85
query56	316	320	288	288
query57	1135	1192	1144	1144
query58	275	269	260	260
query59	2753	2736	2648	2648
query60	345	323	333	323
query61	128	128	140	128
query62	757	747	696	696
query63	232	191	194	191
query64	1744	1034	665	665
query65	4211	4149	4155	4149
query66	738	448	319	319
query67	15838	15479	15318	15318
query68	6840	889	528	528
query69	544	320	275	275
query70	1184	1088	1160	1088
query71	490	324	301	301
query72	5736	4903	5021	4903
query73	1345	676	357	357
query74	9097	9160	8725	8725
query75	3733	3207	2730	2730
query76	4094	1183	746	746
query77	710	351	283	283
query78	10164	10020	9435	9435
query79	5129	784	557	557
query80	849	518	455	455
query81	494	256	228	228
query82	558	130	96	96
query83	298	246	232	232
query84	299	110	101	101
query85	758	359	313	313
query86	372	313	283	283
query87	4489	4490	4380	4380
query88	3790	2253	2273	2253
query89	446	320	297	297
query90	1787	218	211	211
query91	142	139	114	114
query92	74	64	60	60
query93	3575	923	572	572
query94	704	394	298	298
query95	373	292	288	288
query96	492	576	279	279
query97	2738	2753	2635	2635
query98	241	207	202	202
query99	1450	1428	1307	1307
Total cold run time: 300936 ms
Total hot run time: 192547 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.73 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a5b1565b7c283492388d391c692281b402b9a1b5, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.10	0.11
query3	0.25	0.20	0.19
query4	1.60	0.19	0.19
query5	0.45	0.46	0.45
query6	1.16	0.67	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.61	0.52	0.52
query10	0.57	0.58	0.58
query11	0.15	0.11	0.10
query12	0.15	0.12	0.11
query13	0.62	0.60	0.59
query14	0.80	0.80	0.81
query15	0.87	0.85	0.88
query16	0.38	0.40	0.37
query17	1.06	1.05	1.06
query18	0.23	0.21	0.20
query19	1.96	1.80	1.81
query20	0.01	0.01	0.01
query21	15.42	0.87	0.54
query22	0.75	1.25	0.70
query23	14.85	1.41	0.60
query24	7.57	1.20	0.48
query25	0.50	0.21	0.09
query26	0.59	0.16	0.14
query27	0.05	0.05	0.05
query28	9.28	0.92	0.45
query29	12.54	4.02	3.30
query30	0.26	0.09	0.06
query31	2.81	0.58	0.38
query32	3.24	0.55	0.46
query33	3.01	3.14	3.06
query34	15.78	5.10	4.46
query35	4.56	4.51	4.49
query36	0.68	0.50	0.49
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.14
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.48 s
Total hot run time: 28.73 s

@englefly
Copy link
Contributor Author

run feut

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants