Skip to content

perf(files_external): pipeline SFTP stream writes#62377

Open
susnux wants to merge 1 commit into
masterfrom
perf/sftp
Open

perf(files_external): pipeline SFTP stream writes#62377
susnux wants to merge 1 commit into
masterfrom
perf/sftp

Conversation

@susnux

@susnux susnux commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Before

SFTPWriteStream sent a single SSH_FXP_WRITE packet and then blocked on its
acknowledgement before sending the next one. That made every write a full round trip.

After

Mirror phpseclib's SFTP::put() instead: send WRITE packets sized to the
negotiated maximum without waiting for each acknowledgement, and drain the
acknowledgements in batches.

delay file size before (sync) after (pipelined) phpseclib sftp://
LAN 1 MB 1.8 MB/s 307 MB/s 28 MB/s
LAN 8 MB 1.9 MB/s 332 MB/s 38 MB/s
LAN 32 MB 1.9 MB/s 337 MB/s 37 MB/s
~20ms 1 MB 1.3 MB/s 6.3 MB/s 0.24 MB/s
~20ms 8 MB 1.7 MB/s 20.8 MB/s 0.24 MB/s

Checklist

AI (if applicable)

  • The content of this PR was partly or fully generated using AI

ℹ️ AI was used for creating unit tests and creating the benchmark setup.

SFTPWriteStream sent a single SSH_FXP_WRITE packet and then blocked on its
acknowledgement before sending the next one. That made every write a full
round trip, and on low-latency links it additionally triggered a
write-then-read (Nagle / delayed-ACK) stall, capping throughput at ~2 MB/s.

Mirror phpseclib's SFTP::put() instead: send WRITE packets sized to the
negotiated maximum without waiting for each acknowledgement, and drain the
acknowledgements in batches (at NET_SFTP_UPLOAD_QUEUE_SIZE, and on flush /
close). Throughput becomes bandwidth-bound instead of round-trip-bound.

This also fixes a latent issue where the entire flush buffer was sent as a
single SSH_FXP_WRITE that could exceed the negotiated maximum packet size.

Write throughput, 8 KiB stream chunks, OpenSSH sftp-server (indicative,
custom read stream and reads are unchanged as they were already pipelined):

| delay | file size | before (sync) | after (pipelined) | phpseclib sftp:// |
--------|-----------|---------------|-------------------|-------------------|
|   LAN |      1 MB |      1.8 MB/s |          307 MB/s |           28 MB/s |
|   LAN |      8 MB |      1.9 MB/s |          332 MB/s |           38 MB/s |
|   LAN |     32 MB |      1.9 MB/s |          337 MB/s |           37 MB/s |
| ~20ms |      1 MB |      1.3 MB/s |          6.3 MB/s |         0.24 MB/s |
| ~20ms |      8 MB |      1.7 MB/s |         20.8 MB/s |         0.24 MB/s |

Assisted-by: ClaudeCode:claude-opus-4-8
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
@susnux susnux added this to the Nextcloud 35 milestone Jul 21, 2026
@susnux
susnux marked this pull request as ready for review July 22, 2026 09:29
@susnux
susnux requested a review from a team as a code owner July 22, 2026 09:29
@susnux
susnux requested review from Altahrim, come-nc, leftybournes and salmart-dev and removed request for a team July 22, 2026 09:29
@susnux susnux added 3. to review Waiting for reviews and removed 2. developing Work in progress labels Jul 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants