[Bug]: SIGSEGV in `MemcpyWorkerPool::workerThread` under TCP transport with concurrent put/get

[Dogacel]

Bug Report

Summary

On a TCP-only host, MC_STORE_MEMCPY auto-enables and creates a MemcpyWorkerPool. Under concurrent put/get from a multi-threaded Python client, a worker thread segfaults inside glibc __memcpy_avx512_unaligned_erms. Setting MC_STORE_MEMCPY=0 reliably avoids the crash.

Regression PR: #1936

Reproduction

1. Run a multi-process workload where one Mooncake client puts host buffers and another gets them concurrently over TCP.
2. After a few seconds (first batches arriving), a memcpy worker thread crashes.

I've used https://github.com/torchspec-project/TorchSpec's Qwen 8B training pipeline.

Crash signature

gdb on a core dump shows the crashing thread:

#0  __memcpy_avx512_unaligned_erms ()
    at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:265
#1  mooncake::MemcpyWorkerPool::workerThread ()
    from .../site-packages/mooncake/store.so
#2  std::execute_native_thread_routine
#3  start_thread
#4  clone3

Workaround

export MC_STORE_MEMCPY=0

This bypasses MemcpyWorkerPool and the crash disappears completely. Switching transport to RDMA also avoids it (RDMA path doesn't use the pool).

---

Environment

• mooncake-transfer-engine 0.3.10.post2 (pip)
• Python 3.12.13 (miniconda)
• Ubuntu, kernel 6.8.0-1043-aiext, glibc default, system libstdc++.so.6
• Transport: TCP only (RDMA hardware available but not selected)
• Workload: speculative-decoding training pipeline (TorchSpec). Inference workers call put_from, trainer workers call get_into host buffers concurrently.
• Host buffer pool size: 8 × 1 GB; segment size 16 GB; local buffer 4 GB
• MC_STORE_MEMCPY not set → log shows: auto-detected: TCP-only environment, memcpy enabled

Before submitting...

• Ensure you searched for relevant issues and read the [documentation]

[fang-tech]

I'm working on this problem, plz assign it to me.

[ykwd]

I'm working on this problem, plz assign it to me.

Thanks a lot!

[stmatengss]

I'm working on this problem, plz assign it to me.

Please let me know if you have any fixes. Thanks.

[fang-tech]

Root Cause Analysis

The branch point that leads to the memcpy path is selectStrategy() in mooncake-store/src/transfer_task.cpp. When MC_STORE_MEMCPY=1 (automatically set on TCP-only hosts) and isLocalTransfer() returns true, selectStrategy() returns TransferStrategy::LOCAL_MEMCPY.

The intended semantics of isLocalTransfer is to determine whether buffer_address_ can be directly dereferenced in the current process's virtual address space — i.e., whether memcpy(dest, buffer_address_, size) can be executed safely without going through the network. This condition holds only when the buffer was allocated by the current process itself. However, the current implementation only checks whether the buffer resides on the same host, not whether it belongs to the same process:

bool TransferSubmitter::isLocalTransfer(
    const AllocatedBuffer::Descriptor& handle) const {
    std::string local_ep = engine_.getLocalIpAndPort();
    std::string local_ip = extractIpAddress(local_ep);

    if (!local_ep.empty()) {
        std::string handle_ip = extractIpAddress(handle.transport_endpoint_);
        return !handle.transport_endpoint_.empty() && handle_ip == local_ip;
    }

    // Without a local IP we cannot prove locality; disable memcpy.
    return false;
}

Fix

Restore isLocalTransfer to compare the full IP:port instead of IP only.

Question

Historically the logic did perform a full IP:port comparison, but PR #1226 changed it to the current IP-only form. Was there a specific consideration behind that change? Understanding the intent would help confirm that this fix does not introduce any unintended regressions.
cc @stmatengss @ykwd

[fang-tech]

Reproduction

Terminal 1

mooncake_master  \
--enable_http_metadata_server=true  \
 --http_metadata_server_host=0.0.0.0  \
 --http_metadata_server_port=8080

Terminal 2

put_worker.py

import numpy as np
import time
from mooncake.store import MooncakeDistributedStore

HOST_IP = "172.17.0.3"          # Your actual IP
METADATA = f"http://{HOST_IP}:8080/metadata"
MASTER   = f"{HOST_IP}:50051"

store = MooncakeDistributedStore()
ret = store.setup(
    f"{HOST_IP}:12345",   
    METADATA,
    4 * 1024**3,          # global_segment_size
    1 * 1024**3,          # local_buffer_size
    "tcp",
    "",      
    MASTER,
    )
assert ret == 0, f"setup failed: {ret}"
print("put_worker ready")

buf = np.random.rand(128 * 1024 * 1024).astype(np.float32)  # 512 MB
i = 0
while True:
    key = f"key_{i % 4}"
    store.put_tensor_from(key, buf.ctypes.data, buf.nbytes)
    print(f"put {key}")
    i += 1

Terminal 3

get_worker.py

import numpy as np
import time
from mooncake.store import MooncakeDistributedStore

HOST_IP = "172.17.0.3"         # Your actual IP
METADATA = f"http://{HOST_IP}:8080/metadata"
MASTER   = f"{HOST_IP}:50051"

store = MooncakeDistributedStore()
ret = store.setup(
    f"{HOST_IP}:12346",   
    METADATA,
    4 * 1024**3,
    1 * 1024**3,
    "tcp",
    "",
    MASTER,
    )
assert ret == 0, f"setup failed: {ret}"
print("get_worker ready, waiting for data...")
time.sleep(3)  

out = np.empty(128 * 1024 * 1024, dtype=np.float32)
i = 0
while True:
    key = f"key_{i % 4}"
    try:
        store.get_tensor_into(key, out.ctypes.data, out.nbytes)
        print(f"got {key}")
    except Exception as e:
        print(f"error: {e}")
    i += 1

Crash Signature

Thread 55 "python3" received signal SIGSEGV, Segmentation fault.

Thread 55 (Thread 0x7fe788ff9640 (LWP 103854) "python3"):
#0 __memmove_evex_unaligned_erms ()
at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:317
#1 mooncake::MemcpyWorkerPool::workerThread ()
from .../site-packages/mooncake/store.so
#2 std::execute_native_thread_routine
#3 start_thread
#4 clone

[fang-tech]

And if the above fix looks correct, I'd be happy to submit a PR addressing this, along with relevant unit tests.

[00fish0]

@fang-tech Thanks a lot for the clean reproduction script and the careful root-cause writeup. I tried it on my side and reproduced the segfault almost immediately with your three-terminal setup — with MC_STORE_MEMCPY=1 the get_worker SIGSEGVs on the first read, and with MC_STORE_MEMCPY=0 it stays alive, exactly as you described.

Your analysis on isLocalTransfer() reads correct to me. The IP-only comparison was introduced in #1226, and combined with the per-process unique ip:port from #852's AutoPortBinder, it can no longer distinguish processes on the same host — so the LOCAL_MEMCPY path ends up dereferencing another process's virtual address. Restoring the full ip:port comparison (as you proposed) is the right fix.

For what it's worth, #1995 was opened independently and seems to land on the same conclusion — it restores the full ip:port comparison in isLocalTransfer(), and additionally makes MemcpyWorkerPool::workerThread() GPU-pointer-aware (a related issue when buffers are cudaMalloc'd). So this issue should be resolved once #1995 merges.

Thanks again for the thorough investigation!

[fang-tech]

@00fish0 I’m glad this is helpful.
However I found an additional issue. If we simply revert to a strict IP:port check, for example handle.transport_endpoint_ == engine_.getLocalIpAndPort(), we may unexpectedly fall back from LOCAL_MEMCPY to TRANSFER_ENGINE.

In P2P mode, reverting to IP:port is correct: handle.transport_endpoint_ is derived from engine_.getLocalIpAndPort(), and the endpoint is uniquely bound to the local transfer engine instance.

However, in non-P2P mode, handle.transport_endpoint_ comes from local_hostname_, which is a client initialization parameter and may be a logical name such as localhost. In that case, comparing it against engine_.getLocalIpAndPort() will fail by construction, because the latter is the transfer engine's actual runtime endpoint, for example 192.168.1.23:34127. Even if local_hostname_ is an exact IP address, the comparison can still fail because its port may differ from the port assigned by the transfer engine.

In practice, this means the strict IP:port check almost always fails in non-P2P deployments, causing the path to fall back from LOCAL_MEMCPY to TRANSFER_ENGINE. This may also help explain why #1226 switched back to an IP-only comparison.

[fang-tech]

I noticed that 1995 has already been merged. I will submit a new pull request to fix this issue.

[00fish0]

I noticed that 1995 has already been merged. I will submit a new pull request to fix this issue.

Sorry for the late reply! I'll take a look at your PR in the next few days. Thanks for the contribution!