Skip to content

[improve](routine load) adjust default values to make routine load more convenient to use #42491

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Oct 31, 2024
Merged

[improve](routine load) adjust default values to make routine load more convenient to use #42491

merged 1 commit into from
Oct 31, 2024

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Oct 25, 2024 •
edited
Loading

For a routine load job, it will be divided into many tasks, each of which is a transaction. Currently, the default time consumed(max_batch_interval) is 10 seconds. The benefits of increasing this value are:

  1. Larger batch consumption can lead to better performance.
  2. Reducing the number of transactions can alleviate the pressure of compaction and the conflicts of concurrent transaction submissions.

related doc: https://github.com/apache/doris-website/pull/1236/files

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui
Copy link
Contributor Author

sollhui commented Oct 25, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.46% (9713/25932)
Line Coverage: 28.71% (80546/280547)
Region Coverage: 28.14% (41656/148018)
Branch Coverage: 24.70% (21161/85688)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d6d13291f2e5ddfaaf77c05bfc1c7e8e324672fe_d6d13291f2e5ddfaaf77c05bfc1c7e8e324672fe/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 41476 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d6d13291f2e5ddfaaf77c05bfc1c7e8e324672fe, data reload: false

------ Round 1 ----------------------------------
q1	17592	7542	7348	7348
q2	2029	285	272	272
q3	12349	1060	1185	1060
q4	10565	833	819	819
q5	7732	3082	3071	3071
q6	239	149	147	147
q7	1013	614	592	592
q8	9358	2008	2004	2004
q9	6651	6491	6480	6480
q10	7084	2420	2444	2420
q11	443	239	255	239
q12	413	214	218	214
q13	17797	3032	3021	3021
q14	243	214	206	206
q15	556	512	516	512
q16	659	593	591	591
q17	971	587	545	545
q18	7342	6787	6678	6678
q19	1345	1137	964	964
q20	479	184	179	179
q21	3998	3121	3222	3121
q22	1130	1007	993	993
Total cold run time: 109988 ms
Total hot run time: 41476 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7789	7332	7305	7305
q2	354	233	245	233
q3	3116	3016	2968	2968
q4	2073	1778	1797	1778
q5	5734	5790	5853	5790
q6	229	141	141	141
q7	2279	1844	1784	1784
q8	3420	3446	3543	3446
q9	8985	8795	8868	8795
q10	3621	3555	3557	3555
q11	594	505	490	490
q12	817	618	619	618
q13	10627	3175	3207	3175
q14	327	270	283	270
q15	576	511	534	511
q16	698	663	650	650
q17	1867	1633	1612	1612
q18	8334	7723	7492	7492
q19	1757	1663	1590	1590
q20	2145	1847	1896	1847
q21	5524	5565	5443	5443
q22	1146	1075	1077	1075
Total cold run time: 72012 ms
Total hot run time: 60568 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191919 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d6d13291f2e5ddfaaf77c05bfc1c7e8e324672fe, data reload: false

query1	840	417	404	404
query2	6263	2187	2126	2126
query3	8674	193	208	193
query4	35186	23589	23713	23589
query5	5007	467	463	463
query6	305	182	178	178
query7	4210	297	291	291
query8	305	244	236	236
query9	9659	2664	2657	2657
query10	503	271	259	259
query11	18346	15192	15148	15148
query12	146	107	95	95
query13	1583	453	418	418
query14	9416	6815	6927	6815
query15	228	166	167	166
query16	7974	502	478	478
query17	1442	560	560	560
query18	2128	297	306	297
query19	226	145	148	145
query20	120	112	121	112
query21	206	106	108	106
query22	4912	4689	4587	4587
query23	34927	34390	34048	34048
query24	10973	2823	2793	2793
query25	631	407	395	395
query26	1212	165	172	165
query27	2324	282	278	278
query28	7246	2413	2439	2413
query29	842	434	413	413
query30	271	167	166	166
query31	1055	781	773	773
query32	98	54	58	54
query33	755	279	281	279
query34	919	500	510	500
query35	1024	880	884	880
query36	1119	961	973	961
query37	133	87	84	84
query38	4417	4197	4267	4197
query39	1461	1582	1451	1451
query40	197	101	101	101
query41	49	46	47	46
query42	110	99	96	96
query43	533	497	497	497
query44	1258	822	828	822
query45	181	166	168	166
query46	1141	720	728	720
query47	1933	1816	1869	1816
query48	419	324	321	321
query49	920	396	406	396
query50	824	394	391	391
query51	7091	6996	7042	6996
query52	106	88	95	88
query53	255	179	191	179
query54	1133	405	411	405
query55	76	74	79	74
query56	255	252	272	252
query57	1305	1169	1140	1140
query58	236	211	203	203
query59	3489	2965	3158	2965
query60	289	250	254	250
query61	127	119	117	117
query62	835	679	664	664
query63	222	194	185	185
query64	4116	631	601	601
query65	3295	3224	3223	3223
query66	794	296	308	296
query67	15909	15561	15549	15549
query68	4411	553	561	553
query69	432	255	265	255
query70	1184	1137	1135	1135
query71	336	252	247	247
query72	6271	3984	4035	3984
query73	766	363	361	361
query74	10310	8959	9146	8959
query75	3451	2658	2682	2658
query76	2810	980	1059	980
query77	380	269	277	269
query78	10604	9563	9589	9563
query79	1852	593	589	589
query80	894	433	411	411
query81	568	243	245	243
query82	502	114	119	114
query83	224	135	143	135
query84	245	72	71	71
query85	1301	325	308	308
query86	402	282	292	282
query87	4836	4708	4779	4708
query88	3543	2206	2179	2179
query89	412	286	291	286
query90	2105	186	190	186
query91	136	101	98	98
query92	62	49	48	48
query93	2256	552	544	544
query94	958	307	295	295
query95	349	247	245	245
query96	623	273	283	273
query97	2920	2703	2769	2703
query98	217	195	201	195
query99	1596	1318	1310	1310
Total cold run time: 301140 ms
Total hot run time: 191919 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.45 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d6d13291f2e5ddfaaf77c05bfc1c7e8e324672fe, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.03
query3	0.22	0.07	0.07
query4	1.64	0.10	0.09
query5	0.42	0.39	0.40
query6	1.16	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.56	0.51	0.49
query10	0.56	0.55	0.56
query11	0.16	0.11	0.11
query12	0.13	0.10	0.12
query13	0.61	0.60	0.59
query14	2.70	2.75	2.75
query15	0.89	0.82	0.84
query16	0.39	0.37	0.39
query17	1.01	1.01	1.05
query18	0.20	0.20	0.19
query19	1.97	1.77	1.94
query20	0.02	0.00	0.02
query21	15.36	0.60	0.58
query22	2.62	1.71	2.73
query23	16.89	1.09	0.76
query24	3.63	1.53	0.95
query25	0.26	0.18	0.16
query26	0.44	0.14	0.14
query27	0.04	0.03	0.03
query28	10.01	1.10	1.07
query29	12.59	3.34	3.34
query30	0.24	0.06	0.06
query31	2.86	0.39	0.39
query32	3.27	0.46	0.46
query33	3.03	3.01	3.06
query34	17.18	4.46	4.52
query35	4.53	4.50	4.56
query36	0.67	0.48	0.47
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.12	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.87 s
Total hot run time: 32.45 s

@sollhui
Copy link
Contributor Author

sollhui commented Oct 28, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.46% (9715/25936)
Line Coverage: 28.74% (80603/280491)
Region Coverage: 28.16% (41675/148000)
Branch Coverage: 24.72% (21173/85662)
Coverage Report: http://coverage.selectdb-in.cc/coverage/84b3bb4c4a6d22648f0384f4cc57ebc75219cd49_84b3bb4c4a6d22648f0384f4cc57ebc75219cd49/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 41145 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 84b3bb4c4a6d22648f0384f4cc57ebc75219cd49, data reload: false

------ Round 1 ----------------------------------
q1	17871	7686	7304	7304
q2	2046	170	166	166
q3	10688	1073	1251	1073
q4	10437	865	865	865
q5	7777	3080	3101	3080
q6	237	153	146	146
q7	1031	614	583	583
q8	9359	1976	2053	1976
q9	6591	6414	6457	6414
q10	7028	2437	2405	2405
q11	454	247	228	228
q12	404	212	208	208
q13	17842	3019	3017	3017
q14	248	205	215	205
q15	579	532	504	504
q16	633	594	581	581
q17	989	507	559	507
q18	7459	6878	6646	6646
q19	1343	1055	912	912
q20	481	190	198	190
q21	4010	3322	3143	3143
q22	1118	1008	992	992
Total cold run time: 108625 ms
Total hot run time: 41145 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7340	7284	7273	7273
q2	322	227	230	227
q3	3074	2977	3000	2977
q4	2081	1890	1820	1820
q5	5764	5748	5809	5748
q6	224	140	141	140
q7	2245	1829	1806	1806
q8	3357	3567	3478	3478
q9	8973	8976	8954	8954
q10	3581	3611	3551	3551
q11	591	508	495	495
q12	850	639	629	629
q13	11642	3170	3205	3170
q14	324	280	268	268
q15	583	525	530	525
q16	667	662	653	653
q17	1899	1647	1610	1610
q18	8481	7881	7670	7670
q19	1724	1543	1530	1530
q20	2126	1866	1894	1866
q21	5733	5580	5695	5580
q22	1152	1118	1038	1038
Total cold run time: 72733 ms
Total hot run time: 61008 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197740 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 84b3bb4c4a6d22648f0384f4cc57ebc75219cd49, data reload: false

query1	1248	1021	1000	1000
query2	6220	2085	1987	1987
query3	11546	4918	4849	4849
query4	33742	23713	23571	23571
query5	4916	462	435	435
query6	268	182	162	162
query7	3980	300	300	300
query8	287	231	242	231
query9	9857	2735	2730	2730
query10	495	255	249	249
query11	18323	15170	15386	15170
query12	150	107	105	105
query13	1561	439	405	405
query14	9782	7572	7475	7475
query15	265	180	198	180
query16	7972	491	489	489
query17	1566	587	575	575
query18	2030	307	300	300
query19	366	157	164	157
query20	127	122	129	122
query21	213	109	106	106
query22	4823	4467	4263	4263
query23	34909	34070	34271	34070
query24	10916	2874	2869	2869
query25	635	416	399	399
query26	1085	159	163	159
query27	2268	280	283	280
query28	7374	2472	2452	2452
query29	787	426	419	419
query30	251	166	165	165
query31	1006	812	810	810
query32	100	56	58	56
query33	779	306	280	280
query34	924	525	518	518
query35	1038	899	873	873
query36	1092	937	963	937
query37	142	89	74	74
query38	4435	4318	4338	4318
query39	1500	1407	1418	1407
query40	201	102	103	102
query41	50	48	46	46
query42	109	95	102	95
query43	526	477	471	471
query44	1211	812	822	812
query45	185	166	170	166
query46	1146	686	712	686
query47	1905	1817	1821	1817
query48	431	344	322	322
query49	919	397	396	396
query50	805	407	389	389
query51	7111	7010	6945	6945
query52	103	90	89	89
query53	254	183	184	183
query54	1186	421	418	418
query55	84	78	76	76
query56	266	275	243	243
query57	1300	1147	1159	1147
query58	215	201	207	201
query59	3277	3017	3052	3017
query60	297	261	244	244
query61	105	101	96	96
query62	847	680	689	680
query63	217	191	186	186
query64	4000	632	626	626
query65	3335	3212	3200	3200
query66	786	306	306	306
query67	16038	15834	15759	15759
query68	4584	567	567	567
query69	442	258	258	258
query70	1197	1116	1123	1116
query71	389	265	261	261
query72	6547	3994	3987	3987
query73	778	364	362	362
query74	10159	8941	9017	8941
query75	3438	2673	2688	2673
query76	2671	944	984	944
query77	378	290	278	278
query78	10819	9664	9496	9496
query79	1287	601	594	594
query80	931	419	432	419
query81	574	246	244	244
query82	1029	123	119	119
query83	240	139	137	137
query84	240	69	71	69
query85	1384	304	285	285
query86	415	310	297	297
query87	4766	4666	4642	4642
query88	3218	2237	2178	2178
query89	413	291	297	291
query90	1940	187	185	185
query91	138	97	100	97
query92	57	47	49	47
query93	1394	541	544	541
query94	871	306	293	293
query95	357	243	245	243
query96	611	277	283	277
query97	2908	2719	2738	2719
query98	213	199	196	196
query99	1719	1324	1319	1319
Total cold run time: 301994 ms
Total hot run time: 197740 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 84b3bb4c4a6d22648f0384f4cc57ebc75219cd49, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.40	0.38	0.40
query6	1.15	0.66	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.55	0.50	0.49
query10	0.56	0.54	0.54
query11	0.14	0.10	0.10
query12	0.14	0.11	0.13
query13	0.61	0.59	0.59
query14	2.71	2.74	2.76
query15	0.91	0.82	0.82
query16	0.38	0.37	0.40
query17	1.00	1.07	1.00
query18	0.20	0.19	0.20
query19	1.86	1.82	1.88
query20	0.01	0.01	0.01
query21	15.36	0.61	0.60
query22	2.67	1.98	1.72
query23	17.04	1.21	0.63
query24	3.13	0.93	1.14
query25	0.35	0.05	0.22
query26	0.36	0.13	0.14
query27	0.05	0.04	0.04
query28	10.68	1.10	1.06
query29	12.57	3.31	3.32
query30	0.25	0.06	0.06
query31	2.89	0.38	0.38
query32	3.28	0.45	0.46
query33	2.96	3.01	3.04
query34	16.72	4.53	4.48
query35	4.51	4.50	4.53
query36	0.66	0.51	0.49
query37	0.08	0.05	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.6 s
Total hot run time: 32.22 s

@sollhui
Copy link
Contributor Author

sollhui commented Oct 28, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

liaoxin01
liaoxin01 previously approved these changes Oct 28, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 41097 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e48f9d4aa8212dbe60c00c36cf8db36fca862a7d, data reload: false

------ Round 1 ----------------------------------
q1	17562	7421	7309	7309
q2	2053	169	164	164
q3	10563	1077	1222	1077
q4	10547	828	807	807
q5	7762	3068	3059	3059
q6	231	148	144	144
q7	1011	597	606	597
q8	9358	1946	1973	1946
q9	6564	6412	6444	6412
q10	7123	2400	2450	2400
q11	457	232	248	232
q12	409	218	212	212
q13	17817	2999	2991	2991
q14	240	212	219	212
q15	575	508	510	508
q16	648	590	593	590
q17	979	562	611	562
q18	7392	6708	6633	6633
q19	1338	1081	946	946
q20	477	190	181	181
q21	4012	3197	3119	3119
q22	1111	996	1003	996
Total cold run time: 108229 ms
Total hot run time: 41097 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7367	7219	7285	7219
q2	324	224	224	224
q3	2956	2869	3020	2869
q4	2074	1879	1816	1816
q5	5754	5729	5797	5729
q6	230	141	140	140
q7	2215	1862	1802	1802
q8	3348	3566	3451	3451
q9	8973	8911	8908	8908
q10	3592	3567	3579	3567
q11	590	479	491	479
q12	846	654	602	602
q13	9341	3210	3210	3210
q14	305	277	283	277
q15	572	523	528	523
q16	692	643	653	643
q17	1866	1618	1631	1618
q18	8251	7964	7505	7505
q19	1732	1645	1554	1554
q20	2144	1854	1858	1854
q21	5615	5379	5346	5346
q22	1170	1073	1055	1055
Total cold run time: 69957 ms
Total hot run time: 60391 ms

@sollhui
Copy link
Contributor Author

sollhui commented Oct 28, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 41565 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2d1e767d4a796cb54f8d2868ebbab2478e18de9d, data reload: false

------ Round 1 ----------------------------------
q1	17585	7417	7297	7297
q2	2045	156	158	156
q3	10574	1124	1138	1124
q4	10240	835	885	835
q5	7743	3119	3091	3091
q6	234	157	148	148
q7	1033	619	616	616
q8	9371	2034	2015	2015
q9	7839	6463	6482	6463
q10	7104	2447	2510	2447
q11	465	243	244	243
q12	491	213	203	203
q13	17779	2991	2996	2991
q14	242	208	212	208
q15	564	518	512	512
q16	664	585	572	572
q17	986	534	561	534
q18	7384	6775	6692	6692
q19	1338	988	977	977
q20	492	186	176	176
q21	3990	3327	3284	3284
q22	1100	981	1018	981
Total cold run time: 109263 ms
Total hot run time: 41565 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7319	7280	7243	7243
q2	328	236	223	223
q3	2968	2756	2784	2756
q4	1960	1730	1701	1701
q5	5486	5503	5539	5503
q6	222	143	142	142
q7	2124	1707	1690	1690
q8	3246	3419	3413	3413
q9	8565	8563	8507	8507
q10	3500	3476	3442	3442
q11	581	486	474	474
q12	810	590	594	590
q13	9632	3016	3006	3006
q14	293	270	262	262
q15	556	520	512	512
q16	665	635	611	611
q17	1828	1594	1566	1566
q18	7794	7623	7548	7548
q19	1684	1476	1519	1476
q20	2062	1807	1841	1807
q21	5384	5223	5158	5158
q22	1119	1018	1016	1016
Total cold run time: 68126 ms
Total hot run time: 58646 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191917 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2d1e767d4a796cb54f8d2868ebbab2478e18de9d, data reload: false

query1	990	376	374	374
query2	6301	2075	2064	2064
query3	6240	215	229	215
query4	32913	23609	23806	23609
query5	3663	463	445	445
query6	240	182	165	165
query7	3691	290	296	290
query8	269	233	233	233
query9	6761	2760	2741	2741
query10	434	253	252	252
query11	18083	15460	15153	15153
query12	151	106	102	102
query13	1084	443	414	414
query14	9151	7591	6855	6855
query15	222	186	184	184
query16	6929	470	451	451
query17	1190	569	560	560
query18	1608	297	298	297
query19	255	151	155	151
query20	117	110	107	107
query21	207	104	99	99
query22	4503	4402	4131	4131
query23	34893	34241	34135	34135
query24	9425	2849	2901	2849
query25	625	388	424	388
query26	1199	162	172	162
query27	2640	289	288	288
query28	7631	2483	2447	2447
query29	832	424	422	422
query30	328	165	159	159
query31	1009	816	855	816
query32	95	57	62	57
query33	781	284	275	275
query34	977	535	542	535
query35	1031	915	885	885
query36	1101	927	962	927
query37	134	74	75	74
query38	4380	4316	4250	4250
query39	1516	1419	1416	1416
query40	205	100	102	100
query41	52	47	47	47
query42	107	98	99	98
query43	534	505	509	505
query44	1281	841	817	817
query45	184	166	165	165
query46	1150	709	734	709
query47	1947	1853	1859	1853
query48	414	322	324	322
query49	1124	415	404	404
query50	837	406	414	406
query51	7144	6971	7009	6971
query52	103	95	89	89
query53	266	183	186	183
query54	1101	414	411	411
query55	81	78	81	78
query56	270	259	244	244
query57	1379	1159	1186	1159
query58	241	213	208	208
query59	3278	3022	3149	3022
query60	270	247	243	243
query61	104	96	102	96
query62	862	694	675	675
query63	216	189	186	186
query64	4650	620	586	586
query65	3271	3234	3222	3222
query66	1059	306	299	299
query67	16119	15848	15747	15747
query68	4451	582	572	572
query69	434	259	263	259
query70	1182	1139	1087	1087
query71	318	249	261	249
query72	6367	3990	4024	3990
query73	773	359	371	359
query74	10245	8943	9021	8943
query75	3475	2676	2692	2676
query76	2505	1047	1092	1047
query77	397	266	276	266
query78	10506	9543	9526	9526
query79	1167	596	599	596
query80	741	426	434	426
query81	524	242	242	242
query82	786	121	115	115
query83	210	142	140	140
query84	254	67	73	67
query85	1125	286	274	274
query86	323	289	306	289
query87	4766	4689	4649	4649
query88	3331	2243	2157	2157
query89	408	303	293	293
query90	1978	188	189	188
query91	133	99	102	99
query92	58	50	48	48
query93	1064	562	544	544
query94	784	294	276	276
query95	349	250	247	247
query96	617	278	277	277
query97	2876	2702	2753	2702
query98	213	199	193	193
query99	1679	1307	1299	1299
Total cold run time: 285887 ms
Total hot run time: 191917 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.33 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2d1e767d4a796cb54f8d2868ebbab2478e18de9d, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.04	0.03
query3	0.22	0.07	0.07
query4	1.64	0.11	0.11
query5	0.43	0.40	0.41
query6	1.14	0.65	0.66
query7	0.02	0.01	0.01
query8	0.03	0.03	0.03
query9	0.56	0.49	0.49
query10	0.55	0.55	0.54
query11	0.17	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.59	0.60
query14	2.71	2.74	2.80
query15	0.90	0.83	0.82
query16	0.38	0.37	0.38
query17	1.09	1.04	1.06
query18	0.19	0.19	0.19
query19	1.92	1.86	2.00
query20	0.01	0.02	0.01
query21	15.36	0.58	0.59
query22	2.39	2.88	1.78
query23	17.11	0.83	0.84
query24	3.52	1.09	0.65
query25	0.43	0.16	0.13
query26	0.37	0.14	0.14
query27	0.05	0.04	0.04
query28	10.65	1.08	1.06
query29	12.58	3.32	3.29
query30	0.25	0.06	0.05
query31	2.88	0.39	0.38
query32	3.26	0.46	0.45
query33	2.97	3.05	3.04
query34	17.18	4.51	4.45
query35	4.54	4.49	4.46
query36	0.65	0.49	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.46 s
Total hot run time: 32.33 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.46% (9716/25937)
Line Coverage: 28.72% (80576/280539)
Region Coverage: 28.15% (41674/148045)
Branch Coverage: 24.71% (21169/85680)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2d1e767d4a796cb54f8d2868ebbab2478e18de9d_2d1e767d4a796cb54f8d2868ebbab2478e18de9d/report/index.html

dataroaring
dataroaring approved these changes Oct 29, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@liaoxin01 liaoxin01 merged commit 19016b1 into apache:master Oct 31, 2024
25 of 28 checks passed
github-actions bot pushed a commit that referenced this pull request Oct 31, 2024
@sollhui
[improve](routine load) adjust default values to make routine load mo…
bb3dd53 
…re convenient to use (#42491)

For a routine load job, it will be divided into many tasks, each of
which is a transaction. Currently, the default time
consumed(max_batch_interval) is 10 seconds. The benefits of increasing
this value are:
1. Larger batch consumption can lead to better performance.
2. Reducing the number of transactions can alleviate the pressure of
compaction and the conflicts of concurrent transaction submissions.

related doc: https://github.com/apache/doris-website/pull/1236/files
@github-actions github-actions bot mentioned this pull request Oct 31, 2024
Merged
liaoxin01 added a commit that referenced this pull request Nov 4, 2024
@github-actions @sollhui @liaoxin01
[improve](routine load) adjust default values to make routine load mo…
e679fbe 
…re convenient to use (#42968)

PR Body: For a routine load job, it will be divided into many tasks,
each of which is a transaction. Currently, the default time
consumed(max_batch_interval) is 10 seconds. The benefits of increasing
this value are:
1. Larger batch consumption can lead to better performance.
2. Reducing the number of transactions can alleviate the pressure of
compaction and the conflicts of concurrent transaction submissions.

related doc: https://github.com/apache/doris-website/pull/1236/files
 
 Cherry-picked from #42491

Co-authored-by: hui lai <[email protected]>
Co-authored-by: Xin Liao <[email protected]>
sollhui added a commit to sollhui/doris that referenced this pull request Nov 21, 2024
@sollhui
[improve](routine load) adjust default values to make routine load mo…
85c1bdf 
…re convenient to use (apache#42491)

For a routine load job, it will be divided into many tasks, each of
which is a transaction. Currently, the default time
consumed(max_batch_interval) is 10 seconds. The benefits of increasing
this value are:
1. Larger batch consumption can lead to better performance.
2. Reducing the number of transactions can alleviate the pressure of
compaction and the conflicts of concurrent transaction submissions.

related doc: https://github.com/apache/doris-website/pull/1236/files
@sollhui sollhui mentioned this pull request Nov 21, 2024
Merged
dataroaring pushed a commit that referenced this pull request Nov 21, 2024
@sollhui
[improve](routine load) adjust default values to make routine load mo…
346b89e 
…re convenient to use (#42491) (#44377)

pick (#42491)

For a routine load job, it will be divided into many tasks, each of
which is a transaction. Currently, the default time
consumed(max_batch_interval) is 10 seconds. The benefits of increasing
this value are:
1. Larger batch consumption can lead to better performance.
2. Reducing the number of transactions can alleviate the pressure of
compaction and the conflicts of concurrent transaction submissions.

related doc: https://github.com/apache/doris-website/pull/1236/files
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.3-merged kind/behavior-changed reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sollhui @doris-robot @liaoxin01 @dataroaring