PyTorch 2.15 makes rewritten NCCL backend the default and adds FP8 on Apple Silicon
PyTorch 2.15 is out, and the rewritten NCCL backend now runs every nccl process group by default. The release also adds one-sided transports to torch.distributed, finer memory control in FSDP2 and pipelining, FP8 support on Apple Silicon and wider hardware coverage.
- The rewritten NCCL backend is now the default for every nccl process group, with TORCH_DIST_USE_NCCL2=0 and nccl-legacy as rollbacks.
- torch.distributed adds one-sided transports with NIXL as the first backend, capturable in CUDA graphs.
- Apple Silicon gains FP8 and faster decode, including 1.32 to 1.34x MoE decode throughput on an M3 Ultra.
- CUDA 13.2 becomes the default wheel, with CUDA 13.4 and ROCm 10.0 wheels also shipping.
- The release has 3,117 commits from 583 contributors since PyTorch 2.14.
A new default for distributed training
According to the PyTorch team, every job that uses nccl now runs on the rewritten backend by default. PyTorch 2.14 previewed it as the opt-in nccl2 backend, ported from torchcomms. In 2.15 it backs every nccl process group, with TORCH_DIST_USE_NCCL2=0 and the nccl-legacy backend name available as rollback paths.
Communicators are now created eagerly with the process group. Reconfiguration can opt into a shrink/grow fast path (NCCL 2.29 or newer) so surviving ranks skip a full bootstrap. Reconfigure for fault tolerance is also enabled on ROCm with RCCL 2.30.7.
For debugging in production, a backend-agnostic flight recorder writes a post-mortem to disk when a collective hangs. Watchdog health is published to the debug server, so a timeout on any rank triggers flight-recorder dumps from every rank. The team notes that at cluster scale a single stalled GPU can halt an entire run, and that finding the cause has often meant reproducing the failure by hand.
torch.distributed also gains a Python transport API for one-sided data movement, with NIXL as the first backend. Transports connect to a peer through a store, without a process group or collectives. Transport.read_stream and write_stream enqueue transfers on a CUDA stream so they can be captured with torch.cuda.graph. Both the NCCL work and the transport API are marked API Unstable.
FSDP2 and pipeline parallelism get finer control
FSDP2 now lets precision-sensitive parameters, such as MoE router gates and norm weights, stay in fp32 while the rest of the model trains in bf16, still with one all-gather per fully_shard. This is configured through MixedPrecisionPolicy(param_dtype_override_fn=...). A full gradient-accumulation step can be captured in a single CUDA graph, and gradient precision can be set per parameter. Gradients accumulate directly in .grad and follow each parameter's grad_dtype, which changes the default behavior when reduce_dtype=None. The release notes list this under backwards-incompatible changes.
Pipeline parallelism now allocates receive buffers just before each receive instead of for every microbatch up front, and releases forward outputs once their sends complete. Schedules accept a max_outstanding_sends budget, and FSDP unshards are staggered across each rank's startup bubble. Each unique edge between pipeline-parallel ranks gets a dedicated P2P communicator, and FSDP gradient reduction can overlap with later compute. According to the team, these changes make PyTorch pipelining cudagraphable and deterministic, with an optimized memory profile.
Apple Silicon, NVIDIA Blackwell and other hardware
On Apple Silicon, more MPS operators move from MPSGraph to native Metal kernels. The team reports that a single-row softmax over a 128,256-entry vocabulary is 8x faster, upsample backward is up to 21.7x faster, and norm full reductions are more than 100x faster. F.linear now uses existing gemv kernels, with a 1.23x geometric-mean speedup and up to 5x on small projections. The new MPS grouped_mm has a gemv path that gives 1.32 to 1.34x higher MoE decode throughput on an M3 Ultra. QR of a 512 x 512 matrix falls from about 7 seconds to 1.5 ms. The release also brings float8_e4m3fn, _scaled_mm and public torch.mps.Stream APIs.
On NVIDIA Blackwell GPUs, NVGEMM, the CuTeDSL GEMM backend added in 2.14, now folds the scalar dequantization factor of ModelOpt NVFP4 linear layers into the GEMM. This removes a separate pointwise launch per layer during decode. A bounded, shape-aware candidate policy covers Blackwell NVFP4 decode shapes that nvMatmulHeuristics misses.
Other platform changes include FlyDSL kernels on AMD gfx950, Tensor Data Mover support for gfx1250 (MI450), and simpler SYCL kernel authoring and multi-GPU profiling on Intel GPUs.
Packaging and what it means for users
The default PyTorch wheel moves to CUDA 13.2 (13.2.2 toolkit). CUDA 13.4 GA wheels ship alongside it for Linux x86_64, Linux aarch64 and Windows, and add experimental support for NVIDIA Rubin GPUs. ROCm 10.0 wheels arrive next to the ROCm 7.14 wheels from 2.14. All ROCm wheels are now built on AMD's TheRock packages and install ROCm as a pip dependency instead of bundling the libraries. PyTorch 2.15 supports Python 3.11 through 3.15, including free-threaded 3.14t and 3.15t builds, and Dynamo adds Python 3.15 support.
For teams running multi-GPU training on NVIDIA hardware, the NCCL change applies without code changes, and the rollback options exist if problems appear. The release is composed of 3,117 commits from 583 contributors since PyTorch 2.14. A Q&A webinar is scheduled for Thursday, November 5, 2026, at 11 AM PT, with Meta engineers presenting the release.
PyTorch: PyTorch 2.15 Release Blog
