#1558
Issues/modal-labs/modal-examples/#1558

GPU snapshot restore fails in ministral3_inference.py example on H100: 9p volume missing file + NCCL TCPStore broken pipe

Openchenmiaomingopened 5mo ago

Summary

I’m using the official ministral3_inference.py example with GPU snapshots enabled on H100. Snapshot creation succeeds, but restore intermittently fails with:

  • 9p volume file missing
  • NCCL / TCPStore “Broken pipe” errors after restore

This looks like an example/infrastructure issue when GPU snapshots interact with 9p volumes + vLLM + torch.compile artifacts.

Steps to Reproduce

  1. Use the official example file: https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/llm-serving/ministral3_inference.py
  2. Deploy with modal deploy ministral3_inference.py
  3. Observe logs
  4. run python3 ministral3_inference.py
  5. Observe logs

Expected

Snapshot restore should succeed without missing file errors and without NCCL/TCPStore failures.

Activity

4
chenmiaoming5mo ago

log1:

May 10 17:12:07.812 Creating GPU memory snapshot for Function. May 10 17:12:10.620 vllm serve --uvicorn-log-level=info mistralai/Ministral-3-8B-Instruct-2512 --served-model-name mistralai/Ministral-3-8B-Instruct-2512 llm --host 0.0.0.0 --port 8000 --gpu_memory_utilization 0.95 --tensor-parallel-size 1 --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice --enable-sleep-mode --max-num-seqs 2 --max-model-len 12288 --max-num-batched-tokens 12288 May 10 17:12:27.451 WARNING 05-10 09:12:27 [api_server.py:643] SECURITY WARNING: Development endpoints are enabled! This should NOT be used in production! May 10 17:12:27.565 (APIServer pid=4) INFO 05-10 09:12:27 [api_server.py:1351] vLLM API server version 0.13.0 May 10 17:12:27.568 (APIServer pid=4) INFO 05-10 09:12:27 [utils.py:253] non-default args: {'model_tag': 'mistralai/Ministral-3-8B-Instruct-2512', 'host': '0.0.0.0', 'enable_auto_tool_choice': True, 'tool_call_parser': 'mistral', 'model': 'mistralai/Ministral-3-8B-Instruct-2512', 'tokenizer_mode': 'mistral', 'max_model_len': 12288, 'served_model_name': ['mistralai/Ministral-3-8B-Instruct-2512', 'llm'], 'config_format': 'mistral', 'enable_sleep_mode': True, 'load_format': 'mistral', 'gpu_memory_utilization': 0.95, 'max_num_batched_tokens': 12288, 'max_num_seqs': 2} May 10 17:12:42.274 (APIServer pid=4) INFO 05-10 09:12:42 [model.py:514] Resolved architecture: PixtralForConditionalGeneration May 10 17:12:42.274 (APIServer pid=4) INFO 05-10 09:12:42 [model.py:1661] Using max model len 12288 May 10 17:12:42.876 (APIServer pid=4) INFO 05-10 09:12:42 [scheduler.py:230] Chunked prefill is enabled with max_num_batched_tokens=12288. May 10 17:12:56.880 (EngineCore_DP0 pid=49) INFO 05-10 09:12:56 [core.py:93] Initializing a V1 LLM engine (v0.13.0) with config: model='mistralai/Ministral-3-8B-Instruct-2512', speculative_config=None, tokenizer='mistralai/Ministral-3-8B-Instruct-2512', skip_tokenizer_init=False, tokenizer_mode=mistral, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=12288, download_dir=None, load_format=mistral, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=fp8, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False), seed=0, served_model_name=mistralai/Ministral-3-8B-Instruct-2512, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [12288], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 4, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False}, 'local_cache_dir': None} May 10 17:12:57.198 (EngineCore_DP0 pid=49) [2026-05-10 09:12:57] INFO tekken.py:204: Non special vocabulary size is 130072 with 1000 special tokens. May 10 17:12:58.798 (EngineCore_DP0 pid=49) INFO 05-10 09:12:58 [parallel_state.py:1203] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.20.14.147:46989 backend=nccl May 10 17:12:58.823 (EngineCore_DP0 pid=49) INFO 05-10 09:12:58 [parallel_state.py:1411] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0 May 10 17:12:59.895 (EngineCore_DP0 pid=49) INFO 05-10 09:12:59 [gpu_model_runner.py:3562] Starting to load model mistralai/Ministral-3-8B-Instruct-2512... May 10 17:13:22.245 (EngineCore_DP0 pid=49) INFO 05-10 09:13:22 [cuda.py:351] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION') May 10 17:14:20.224 (EngineCore_DP0 pid=49) INFO 05-10 09:14:20 [weight_utils.py:487] Time spent downloading weights for mistralai/Ministral-3-8B-Instruct-2512: 57.652300 seconds May 10 17:14:20.247 (EngineCore_DP0 pid=49) INFO 05-10 09:14:20 [weight_utils.py:527] No consolidated.safetensors.index.json found in remote. May 10 17:14:20.249 Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s] May 10 17:14:22.500 Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:02<00:00, 2.25s/it] May 10 17:14:22.500 Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:02<00:00, 2.25s/it] May 10 17:14:22.500 (EngineCore_DP0 pid=49) May 10 17:14:22.868 (EngineCore_DP0 pid=49) INFO 05-10 09:14:22 [default_loader.py:308] Loading weights took 2.62 seconds May 10 17:14:23.493 (EngineCore_DP0 pid=49) INFO 05-10 09:14:23 [gpu_model_runner.py:3659] Model loading took 9.7684 GiB memory and 82.817632 seconds May 10 17:14:23.897 (EngineCore_DP0 pid=49) INFO 05-10 09:14:23 [gpu_model_runner.py:4446] Encoder cache will be initialized with a budget of 12288 tokens, and profiled with 4 image items of the maximum feature size. May 10 17:14:24.039 (EngineCore_DP0 pid=49) WARNING 05-10 09:14:24 [processing.py:1153] PixtralProcessorAdapter did not return BatchFeature. Make sure to match the behaviour of ProcessorMixin when implementing custom processors. May 10 17:14:39.639 (EngineCore_DP0 pid=49) INFO 05-10 09:14:39 [backends.py:643] Using cache directory: /root/.cache/vllm/torch_compile_cache/ed6a7686a0/rank_0_0/backbone for vLLM's torch.compile May 10 17:14:39.642 (EngineCore_DP0 pid=49) INFO 05-10 09:14:39 [backends.py:703] Dynamo bytecode transform time: 12.70 s May 10 17:14:54.836 (EngineCore_DP0 pid=49) INFO 05-10 09:14:54 [backends.py:261] Cache the graph of compile range (1, 12288) for later use May 10 17:15:02.534 (EngineCore_DP0 pid=49) INFO 05-10 09:15:02 [backends.py:278] Compiling a graph for compile range (1, 12288) takes 12.59 s May 10 17:15:02.537 (EngineCore_DP0 pid=49) INFO 05-10 09:15:02 [monitor.py:34] torch.compile takes 25.29 s in total May 10 17:15:03.792 (EngineCore_DP0 pid=49) INFO 05-10 09:15:03 [gpu_worker.py:375] Available KV cache memory: 59.58 GiB May 10 17:15:04.179 (EngineCore_DP0 pid=49) INFO 05-10 09:15:04 [kv_cache_utils.py:1291] GPU KV cache size: 459,344 tokens May 10 17:15:04.179 (EngineCore_DP0 pid=49) INFO 05-10 09:15:04 [kv_cache_utils.py:1296] Maximum concurrency for 12,288 tokens per request: 37.38x May 10 17:15:04.210 (EngineCore_DP0 pid=49) 2026-05-10 09:15:04,210 - INFO - autotuner.py:256 - flashinfer.jit: [Autotuner]: Autotuning process starts ... May 10 17:15:04.227 (EngineCore_DP0 pid=49) 2026-05-10 09:15:04,227 - INFO - autotuner.py:262 - flashinfer.jit: [Autotuner]: Autotuning process ends May 10 17:15:04.727 Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 3/3 [00:00<00:00, 25.97it/s] May 10 17:15:04.829 Capturing CUDA graphs (decode, FULL): 100%|██████████| 2/2 [00:00<00:00, 19.81it/s] May 10 17:15:05.266 (EngineCore_DP0 pid=49) INFO 05-10 09:15:05 [gpu_model_runner.py:4587] Graph capturing finished in 1 secs, took -1.83 GiB May 10 17:15:05.282 (EngineCore_DP0 pid=49) INFO 05-10 09:15:05 [core.py:259] init engine (profile, create kv cache, warmup model) took 41.78 seconds May 10 17:15:06.249 (APIServer pid=4) WARNING 05-10 09:15:06 [api_router.py:55] SECURITY WARNING: Development endpoints are enabled! This should NOT be used in production! May 10 17:15:06.332 (APIServer pid=4) INFO 05-10 09:15:06 [api_server.py:1099] Supported tasks: ['generate'] May 10 17:15:06.412 (APIServer pid=4) INFO 05-10 09:15:06 [serving_engine.py:270] "auto" tool choice has been enabled. May 10 17:15:06.412 (APIServer pid=4) INFO 05-10 09:15:06 [serving_engine.py:270] "auto" tool choice has been enabled. May 10 17:15:06.477 (APIServer pid=4) INFO 05-10 09:15:06 [serving_engine.py:270] "auto" tool choice has been enabled. May 10 17:15:06.509 (APIServer pid=4) INFO 05-10 09:15:06 [api_server.py:1425] Starting vLLM API server 0 on http://0.0.0.0:8000 May 10 17:15:06.509 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:38] Available routes are: May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /docs, Methods: HEAD, GET May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /redoc, Methods: HEAD, GET May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /sleep, Methods: POST May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /wake_up, Methods: POST May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /is_sleeping, Methods: GET May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /tokenize, Methods: POST May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /detokenize, Methods: POST May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /inference/v1/generate, Methods: POST May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /pause, Methods: POST May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /resume, Methods: POST May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /is_paused, Methods: GET May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /metrics, Methods: GET May 10 17:15:06.510 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /health, Methods: GET May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /load, Methods: GET May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /v1/models, Methods: GET May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /version, Methods: GET May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /v1/responses, Methods: POST May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /v1/messages, Methods: POST May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /v1/chat/completions, Methods: POST May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /v1/completions, Methods: POST May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /v1/audio/translations, Methods: POST May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /server_info, Methods: GET May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /reset_prefix_cache, Methods: POST May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /reset_mm_cache, Methods: POST May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /collective_rpc, Methods: POST May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /ping, Methods: GET May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /ping, Methods: POST May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /invocations, Methods: POST May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /classify, Methods: POST May 10 17:15:06.511 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /v1/embeddings, Methods: POST May 10 17:15:06.512 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /score, Methods: POST May 10 17:15:06.512 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /v1/score, Methods: POST May 10 17:15:06.512 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /rerank, Methods: POST May 10 17:15:06.512 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /v1/rerank, Methods: POST May 10 17:15:06.512 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /v2/rerank, Methods: POST May 10 17:15:06.512 (APIServer pid=4) INFO 05-10 09:15:06 [launcher.py:46] Route: /pooling, Methods: POST May 10 17:15:06.567 (APIServer pid=4) INFO: Started server process [4] May 10 17:15:06.567 (APIServer pid=4) INFO: Waiting for application startup. May 10 17:15:06.846 (APIServer pid=4) INFO: Application startup complete. May 10 17:15:06.851 (APIServer pid=4) INFO 05-10 09:15:06 [chat_utils.py:590] Detected the chat template content format to be 'string'. You can set --chat-template-content-format to override this. May 10 17:15:06.876 (APIServer pid=4) /usr/local/lib/python3.12/site-packages/mistral_common/tokens/tokenizers/tekken.py:471: FutureWarning: get_control_token is deprecated. Use get_special_token instead. (APIServer pid=4) warnings.warn("get_control_token is deprecated. Use get_special_token instead.", FutureWarning) May 10 17:15:06.962 (APIServer pid=4) INFO: 127.0.0.1:21014 - "POST /v1/chat/completions HTTP/1.1" 200 OK May 10 17:15:07.048 (APIServer pid=4) INFO: 127.0.0.1:55202 - "POST /v1/chat/completions HTTP/1.1" 200 OK May 10 17:15:07.132 (APIServer pid=4) INFO: 127.0.0.1:30381 - "POST /v1/chat/completions HTTP/1.1" 200 OK May 10 17:15:07.139 (EngineCore_DP0 pid=49) INFO 05-10 09:15:07 [block_pool.py:447] Successfully reset prefix cache May 10 17:15:11.090 (EngineCore_DP0 pid=49) INFO 05-10 09:15:11 [cumem.py:239] CuMemAllocator: sleep freed 69.41 GiB memory in total, of which 9.78 GiB is backed up in CPU and the rest 59.63 GiB is discarded directly. May 10 17:15:11.091 (EngineCore_DP0 pid=49) INFO 05-10 09:15:11 [gpu_worker.py:132] Sleep mode freed 69.40 GiB memory, 0.92 GiB memory is still in use. May 10 17:15:11.091 (EngineCore_DP0 pid=49) INFO 05-10 09:15:11 [abstract.py:306] It took 3.950532 seconds to fall asleep. May 10 17:15:11.092 (APIServer pid=4) INFO: 127.0.0.1:33263 - "POST /sleep?level=1 HTTP/1.1" 200 OK May 10 17:15:51.763 Snapshot created. Restoring Function from memory snapshot. May 10 17:15:52.096 Restoring Function from memory snapshot. May 10 17:15:57.031 starting container: restoring container "ta-01KR8JGZXK9N638M5GS0NVAES4": failed to load kernel: vfs.CompleteRestore() failed: failed to complete restore for filesystem type "9p": failed to walk "vo-Kb4nVrzSbcwBj6jBJt813V/xet/logs/xet_20260510T091242999+0000_4.log" of type 8000: no such file or directory May 10 17:15:59.159 Runner failed with exit code: 128

chenmiaoming5mo ago

log2:

May 10 17:16:02.031 Restoring Function from memory snapshot. May 10 17:16:06.020 (APIServer pid=4) INFO 05-10 09:16:06 [loggers.py:248] Engine 000: Avg prompt throughput: 0.4 tokens/s, Avg generation throughput: 0.7 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0% May 10 17:16:08.851 [rank0]:[W510 09:16:08.466112605 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:08.851 [rank0]:[W510 09:16:08.507875726 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:09.857 [rank0]:[W510 09:16:09.508229886 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:09.857 [rank0]:[W510 09:16:09.513620968 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:10.866 [rank0]:[W510 09:16:10.514129066 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

[rank0]:[W510 09:16:10.522108327 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:12.594 [rank0]:[W510 09:16:11.523292326 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:12.594 [rank0]:[W510 09:16:12.251031150 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:13.605 [rank0]:[W510 09:16:13.255634794 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:13.605 [rank0]:[W510 09:16:13.261968919 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:14.618 [rank0]:[W510 09:16:14.262374491 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:14.620 [rank0]:[W510 09:16:14.274726817 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:15.624 [rank0]:[W510 09:16:15.275661063 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:15.624 [rank0]:[W510 09:16:15.281062449 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:15.625 (APIServer pid=4) INFO 05-10 09:16:15 [api_router.py:38] wake up the engine with tags: None May 10 17:16:16.020 (APIServer pid=4) INFO 05-10 09:16:16 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0% May 10 17:16:16.263 (EngineCore_DP0 pid=49) INFO 05-10 09:16:16 [abstract.py:324] It took 0.628759 seconds to wake up tags {'weights', 'kv_cache'}. May 10 17:16:16.268 (APIServer pid=4) INFO: 127.0.0.1:36721 - "POST /wake_up HTTP/1.1" 200 OK May 10 17:16:16.410 (APIServer pid=4) INFO: 172.20.9.75:58283 - "POST /v1/chat/completions HTTP/1.1" 200 OK May 10 17:16:16.632 [rank0]:[W510 09:16:16.282222950 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:16.632 [rank0]:[W510 09:16:16.288300802 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:17.112 GET /v1/chat/completions -> 200 OK (duration: 100.6 s, execution: 704.9 ms) May 10 17:16:17.640 [rank0]:[W510 09:16:17.290856179 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:17.640 [rank0]:[W510 09:16:17.296343129 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:18.646 [rank0]:[W510 09:16:18.297452594 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:18.646 [rank0]:[W510 09:16:18.302605457 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:19.652 [rank0]:[W510 09:16:19.303857532 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:19.652 [rank0]:[W510 09:16:19.308843845 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:20.658 [rank0]:[W510 09:16:20.309461308 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:20.658 [rank0]:[W510 09:16:20.314353788 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:21.664 [rank0]:[W510 09:16:21.315585968 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

[rank0]:[W510 09:16:21.320820731 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:22.671 [rank0]:[W510 09:16:22.321426002 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:22.671 [rank0]:[W510 09:16:22.328042071 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:23.677 [rank0]:[W510 09:16:23.328747549 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:23.677 [rank0]:[W510 09:16:23.333720954 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:24.683 [rank0]:[W510 09:16:24.334636153 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:24.683 [rank0]:[W510 09:16:24.339697053 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:25.689 [rank0]:[W510 09:16:25.340287020 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:25.689 [rank0]:[W510 09:16:25.345312151 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:26.022 (APIServer pid=4) INFO 05-10 09:16:26 [loggers.py:248] Engine 000: Avg prompt throughput: 0.8 tokens/s, Avg generation throughput: 4.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0% May 10 17:16:26.695 [rank0]:[W510 09:16:26.346710080 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:26.695 [rank0]:[W510 09:16:26.351920569 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:27.701 [rank0]:[W510 09:16:27.352641823 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:27.701 [rank0]:[W510 09:16:27.357689076 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:28.707 [rank0]:[W510 09:16:28.358951040 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:28.707 [rank0]:[W510 09:16:28.363946529 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:29.713 [rank0]:[W510 09:16:29.364849260 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:29.713 [rank0]:[W510 09:16:29.369974884 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:30.719 [rank0]:[W510 09:16:30.370225789 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:30.719 [rank0]:[W510 09:16:30.375263709 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:31.725 [rank0]:[W510 09:16:31.375612686 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:31.725 [rank0]:[W510 09:16:31.382031630 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:32.731 [rank0]:[W510 09:16:32.383116366 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:32.732 [rank0]:[W510 09:16:32.388187202 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:33.739 [rank0]:[W510 09:16:33.389762510 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:33.739 [rank0]:[W510 09:16:33.395368149 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:34.744 [rank0]:[W510 09:16:34.395619793 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:34.744 [rank0]:[W510 09:16:34.400709670 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:35.750 [rank0]:[W510 09:16:35.401416980 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:35.750 [rank0]:[W510 09:16:35.406838982 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:36.023 (APIServer pid=4) INFO 05-10 09:16:36 [loggers.py:248] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%, MM cache hit rate: 0.0% May 10 17:16:36.756 [rank0]:[W510 09:16:36.407980796 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:36.756 [rank0]:[W510 09:16:36.413084421 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:37.762 [rank0]:[W510 09:16:37.413317221 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:37.762 [rank0]:[W510 09:16:37.418342070 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:38.767 [rank0]:[W510 09:16:38.418573636 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:38.767 [rank0]:[W510 09:16:38.423640263 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:39.773 [rank0]:[W510 09:16:39.424581832 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:39.773 [rank0]:[W510 09:16:39.429497611 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:40.778 [rank0]:[W510 09:16:40.430033047 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:40.778 [rank0]:[W510 09:16:40.435124744 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:41.784 [rank0]:[W510 09:16:41.436250961 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:41.784 [rank0]:[W510 09:16:41.441118175 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:42.790 [rank0]:[W510 09:16:42.442320200 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:42.790 [rank0]:[W510 09:16:42.447101101 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:43.796 [rank0]:[W510 09:16:43.447770186 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:43.796 [rank0]:[W510 09:16:43.453056672 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:44.801 [rank0]:[W510 09:16:44.453434092 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:44.802 [rank0]:[W510 09:16:44.458191695 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:45.807 [rank0]:[W510 09:16:45.458621822 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:45.807 [rank0]:[W510 09:16:45.463643324 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:46.813 [rank0]:[W510 09:16:46.464464498 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:46.813 [rank0]:[W510 09:16:46.469271750 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:47.818 [rank0]:[W510 09:16:47.469943391 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:47.818 [rank0]:[W510 09:16:47.474928365 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:48.825 [rank0]:[W510 09:16:48.476178676 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:48.825 [rank0]:[W510 09:16:48.481225690 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:49.830 [rank0]:[W510 09:16:49.482226303 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

[rank0]:[W510 09:16:49.486926413 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:50.836 [rank0]:[W510 09:16:50.487862524 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:50.836 [rank0]:[W510 09:16:50.492703185 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:51.841 [rank0]:[W510 09:16:51.493136913 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:51.841 [rank0]:[W510 09:16:51.498042952 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:52.847 [rank0]:[W510 09:16:52.498878819 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:52.847 [rank0]:[W510 09:16:52.503628075 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:53.852 [rank0]:[W510 09:16:53.503984818 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

May 10 17:16:53.852 [rank0]:[W510 09:16:53.508862322 ProcessGroupNCCL.cpp:1771] [PG ID 0 PG GUID 0 Rank 0] Failed to check the "should dump" flag on TCPStore, (maybe TCPStore server has shut down too early), with error: Broken pipe May 10 17:16:54.858 [rank0]:[W510 09:16:54.509978292 TCPStore.cpp:106] [c10d] sendBytes failed on SocketImpl(fd=62, addr=[::ffff:172.20.14.147]:54761, remote=[::ffff:172.20.14.147]:46989): Broken pipe Exception raised from sendBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:668 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x80 (0x2b3303b51b80 in /usr/local/lib/python3.12/site-packages/torch/lib/libc10.so) frame #1: + 0x5ffd531 (0x2b33187fd531 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #2: + 0x5ffddc2 (0x2b33187fddc2 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #3: + 0x5fff8ce (0x2b33187ff8ce in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #4: c10d::TCPStore::check(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&) + 0x30e (0x2b33187fa1ee in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cpu.so) frame #5: c10d::ProcessGroupNCCL::HeartbeatMonitor::runLoop() + 0x3c8 (0x2b33280546b8 in /usr/local/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so) frame #6: + 0xdc253 (0x2b330b2dc253 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6) frame #7: + 0x94ac3 (0x2b3302e94ac3 in /usr/lib/x86_64-linux-gnu/libc.so.6) frame #8: + 0x126850 (0x2b3302f26850 in /usr/lib/x86_64-linux-gnu/libc.so.6)

rehan2433mo ago

Hm, ran into something similar with NCCL errors on restore, are you using the latest vLLM version? I've found that torch.compile can sometimes cause issues with snapshot restore, especially when you're using it with GPU snapshots and 9p volumes like in ministral3_inference.py. I'd try disabling torch.compile or experimenting with different vLLM versions to see if that resolves the "Broken pipe" errors you're seeing. I recall having to tweak my container config to get around a related issue, turns out the docs don't mention this specific gotcha. What's your Modal version, by the way?

georgebaher27d ago

I hit both errors from this issue (vLLM 0.21.0, Modal client 1.6.1, L40S, GPU snapshots + vLLM sleep mode) and narrowed down each cause.

1. failed to walk ".../torch_compile_cache" (9p)

Cause: when the snapshotting container restores, it sees the Volume as it was when the container started, not when the snapshot was taken. If the process holds open a file or directory created on the Volume during snap=True, the restore can't find it and fails. Modal's retry works because the new container mounts the latest Volume.

vLLM creates torch_compile_cache on the vllm-cache Volume during snap=True, and the restore then fails on that directory, same as the repro below. We saw it on the first snapshot build with an empty cache.

CPU-only repro (fresh, empty Volume):

import os, modal

app = modal.App("snapshot-volume-repro")
vol = modal.Volume.from_name("snapshot-volume-repro", create_if_missing=True)

@app.cls(image=modal.Image.debian_slim(), volumes={"/vol": vol}, enable_memory_snapshot=True)
class Repro:
    @modal.enter(snap=True)
    def build(self):
        os.makedirs("/vol/newdir", exist_ok=True)
        with open("/vol/newdir/data.txt", "w") as f:
            f.write("x")
        vol.commit()
        self.f = open("/vol/newdir/data.txt")  # held open across the snapshot

    @modal.method()
    def ping(self):
        return "ok"

Variants (each on a new, empty Volume):

During snap=TrueRestoreNew file visible after restore
Read onlyOKn/a
Write, closeOKNo
Write, close, commit()OKNo
Write, close, commit(), wait 60 sOKNo
Write, commit(), keep openFails (failed to walk)In retry container: yes
Same + wait 60 sFailsIn retry container: yes

So vol.commit() before the snapshot (as in #1615) doesn't help, and neither does waiting.

Workaround: don't mount a Volume at vLLM's cache dir on snapshotted Functions; vLLM then uses local disk, which the snapshot captures. Verified twice on the setup above, both with a freshly compiled cache: first-try restore.

2. NCCL TCPStore ... Broken pipe every second after restore

Cause: PyTorch's ProcessGroupNCCL heartbeat monitor polls the TCPStore for a "should dump" flag every second (TORCH_NCCL_COORD_CHECK_MILSEC), but the TCPStore socket doesn't survive the snapshot. Harmless, but it floods the logs.

Fix: set TORCH_NCCL_DUMP_ON_TIMEOUT=0 in the image env; the polling is gated on it (checkDumpSignal = dumpOnTimeoutOrEx_ && uid == 0). TORCH_NCCL_ENABLE_MONITORING=0 does not stop it; it only controls whether the monitor kills the process. Verified: 0 warnings after restore, vs. one per second before.