Fix slow clone/fetch from git remotes

[elianddb]

Reading a git-backed table file went chunk by chunk, and because git cat-file cannot serve a byte range, each read re-streamed the whole object from the start. Fetching a large table spawned thousands of git processes and stalled.

• Spool a git-backed table file to a local temp file on open and serve its chunk reads locally
• Look up the sizes of a chunked object's parts in a single git cat-file --batch-check call
• Detect an early reader close by end of stdout, so a partial read is torn down the same way on every platform
  Fix git remote clone/fetch stalls importing refs/dolt/data #11236

[coffeegoddd]

@elianddb DOLT

read_tests	from_latency	to_latency	percent_change
covering_index_scan	2.3	2.3	0.0
groupby_scan	142.39	139.85	-1.78
index_join	1.93	1.93	0.0
index_join_scan	1.32	1.32	0.0
index_scan	215.44	215.44	0.0
oltp_point_select	0.25	0.25	0.0
oltp_read_only	4.91	4.91	0.0
select_random_points	0.5	0.5	0.0
select_random_ranges	0.63	0.63	0.0
table_scan	207.82	204.11	-1.79
types_table_scan	475.79	475.79	0.0

write_tests	from_latency	to_latency	percent_change
oltp_delete_insert	6.09	6.09	0.0
oltp_insert	3.13	3.13	0.0
oltp_read_write	11.04	11.04	0.0
oltp_update_index	3.3	3.25	-1.52
oltp_update_non_index	3.02	2.97	-1.66
oltp_write_only	6.21	6.21	0.0
types_delete_insert	6.79	6.79	0.0

[coffeegoddd]

@elianddb DOLT

comparing_percentages
100.000000 to 100.000000

version	result	total
edb6187	ok	5937471

version	total_tests
edb6187	5937471

correctness_percentage
100.0

[coffeegoddd]

@elianddb DOLT

test_name	from_latency_p95	to_latency_p95	percent_change
tpcc-scale-factor-1	44.98	44.98	0.0

test_name	from_server_name	from_server_version	from_tps	to_server_name	to_server_version	to_tps	percent_change
tpcc-scale-factor-1	dolt	2a5c6c2	53.18	dolt	184769f	53.26	0.15

[coffeegoddd]

@elianddb DOLT

comparing_percentages
100.000000 to 100.000000

version	result	total
566875a	ok	5937471

version	total_tests
566875a	5937471

correctness_percentage
100.0

[coffeegoddd]

@elianddb DOLT

comparing_percentages
100.000000 to 100.000000

version	result	total
fa67791	ok	5937471

version	total_tests
fa67791	5937471

correctness_percentage
100.0

[coffeegoddd]

@elianddb DOLT

comparing_percentages
100.000000 to 100.000000

version	result	total
3c6f0ac	ok	5937471

version	total_tests
3c6f0ac	5937471

correctness_percentage
100.0

[coffeegoddd]

@elianddb DOLT

comparing_percentages
100.000000 to 100.000000

version	result	total
b907f04	ok	5937471

version	total_tests
b907f04	5937471

correctness_percentage
100.0

[coffeegoddd]

@elianddb DOLT

comparing_percentages
100.000000 to 100.000000

version	result	total
0d7e368	ok	5937471

version	total_tests
0d7e368	5937471

correctness_percentage
100.0

[macneale4]

This smells very wrong to me. I don't think changes which are only related to git+ remote types should be so invasive to bs_persister.go.

Ref counting of anything is almost always a sign that something is wrong, and given the sentence above I think it merrits some discussion about that the actual problem is.

[coffeegoddd]

@elianddb DOLT

comparing_percentages
100.000000 to 100.000000

version	result	total
95042e2	ok	5937471

version	total_tests
95042e2	5937471

correctness_percentage
100.0

[coffeegoddd]

@elianddb DOLT

comparing_percentages
100.000000 to 100.000000

version	result	total
1af2831	ok	5937471

version	total_tests
1af2831	5937471

correctness_percentage
100.0

[coffeegoddd]

@elianddb DOLT

comparing_percentages
100.000000 to 100.000000

version	result	total
ae5f527	ok	5937471

version	total_tests
ae5f527	5937471

correctness_percentage
100.0

[reltuk]

Looks good to me. One comment about the Reader() implementation which I think is a pretty near-at-hand fix.

[coffeegoddd]

@elianddb DOLT

comparing_percentages
100.000000 to 100.000000

version	result	total
184769f	ok	5937471

version	total_tests
184769f	5937471

correctness_percentage
100.0

[reltuk]

LGTM!

[github-actions]

@coffeegoddd DOLT

name	add_cnt	delete_cnt	update_cnt	latency
adds_only	60000	0	0	0.59
adds_updates_deletes	60000	60000	60000	2.82
deletes_only	0	60000	0	1.44
updates_only	0	0	60000	1.68

[github-actions]

@coffeegoddd DOLT

name	detail	mean_mult
dolt_blame_basic	system table	1.23
dolt_blame_commit_filter	system table	1.19
dolt_commit_ancestors_commit_filter	system table	0.69
dolt_commits_commit_filter	system table	1.05
dolt_diff_log_join_from_commit	system table	3.08
dolt_diff_log_join_to_commit	system table	3.05
dolt_diff_table_from_commit_filter	system table	1.24
dolt_diff_table_to_commit_filter	system table	1.18
dolt_diffs_commit_filter	system table	1.14
dolt_history_commit_filter	system table	1.42
dolt_log_commit_filter	system table	1.2

[github-actions]

@coffeegoddd DOLT

name	add_cnt	delete_cnt	update_cnt	latency
adds_only	60000	0	0	0.6
adds_updates_deletes	60000	60000	60000	2.92
deletes_only	0	60000	0	1.45
updates_only	0	0	60000	1.73

[github-actions]

@coffeegoddd DOLT

name	detail	mean_mult
dolt_blame_basic	system table	1.28
dolt_blame_commit_filter	system table	1.23
dolt_commit_ancestors_commit_filter	system table	0.69
dolt_commits_commit_filter	system table	1.16
dolt_diff_log_join_from_commit	system table	3.01
dolt_diff_log_join_to_commit	system table	3.05
dolt_diff_table_from_commit_filter	system table	1.15
dolt_diff_table_to_commit_filter	system table	1.15
dolt_diffs_commit_filter	system table	1.22
dolt_history_commit_filter	system table	1.41
dolt_log_commit_filter	system table	1.15

[github-actions]

@coffeegoddd DOLT

name	add_cnt	delete_cnt	update_cnt	latency
adds_only	60000	0	0	0.62
adds_updates_deletes	60000	60000	60000	3.03
deletes_only	0	60000	0	1.5
updates_only	0	0	60000	1.76

[github-actions]

@coffeegoddd DOLT

name	detail	mean_mult
dolt_blame_basic	system table	1.29
dolt_blame_commit_filter	system table	1.16
dolt_commit_ancestors_commit_filter	system table	0.67
dolt_commits_commit_filter	system table	1.18
dolt_diff_log_join_from_commit	system table	2.81
dolt_diff_log_join_to_commit	system table	2.83
dolt_diff_table_from_commit_filter	system table	1.23
dolt_diff_table_to_commit_filter	system table	1.2
dolt_diffs_commit_filter	system table	1.15
dolt_history_commit_filter	system table	1.5
dolt_log_commit_filter	system table	1.38

[github-actions]

@coffeegoddd DOLT

name	add_cnt	delete_cnt	update_cnt	latency
adds_only	60000	0	0	0.6
adds_updates_deletes	60000	60000	60000	2.88
deletes_only	0	60000	0	1.44
updates_only	0	0	60000	1.71

[github-actions]

@coffeegoddd DOLT

name	detail	mean_mult
dolt_blame_basic	system table	1.36
dolt_blame_commit_filter	system table	1.26
dolt_commit_ancestors_commit_filter	system table	0.69
dolt_commits_commit_filter	system table	1.1
dolt_diff_log_join_from_commit	system table	3.13
dolt_diff_log_join_to_commit	system table	3.11
dolt_diff_table_from_commit_filter	system table	1.21
dolt_diff_table_to_commit_filter	system table	1.18
dolt_diffs_commit_filter	system table	1.14
dolt_history_commit_filter	system table	1.63
dolt_log_commit_filter	system table	1.32

[github-actions]

@coffeegoddd DOLT

name	add_cnt	delete_cnt	update_cnt	latency
adds_only	60000	0	0	0.73
adds_updates_deletes	60000	60000	60000	2.44
deletes_only	0	60000	0	1.3
updates_only	0	0	60000	1.45

[github-actions]

@coffeegoddd DOLT

name	detail	mean_mult
dolt_blame_basic	system table	1.28
dolt_blame_commit_filter	system table	1.19
dolt_commit_ancestors_commit_filter	system table	0.76
dolt_commits_commit_filter	system table	1.26
dolt_diff_log_join_from_commit	system table	3.14
dolt_diff_log_join_to_commit	system table	3.16
dolt_diff_table_from_commit_filter	system table	1.18
dolt_diff_table_to_commit_filter	system table	1.22
dolt_diffs_commit_filter	system table	1.07
dolt_history_commit_filter	system table	1.48
dolt_log_commit_filter	system table	1.21