nccl

NumPy & SciPy for GPU

Safe rust wrapper around CUDA toolkit

Best practices & guides on how to write distributed pytorch training code

An open collection of methodologies to help with successful training of large language models.

An open collection of implementation tips, tricks and resources for training large language models

Efficient Distributed GPU Programming for Exascale, an SC/ISC Tutorial

Distributed and decentralized training framework for PyTorch over graph

Federated Learning Utilities and Tools for Experimentation

NCCL Fast Socket is a transport layer plugin to improve NCCL collective communication performance on Google Cloud.

CUDA编程练习项目-Hands-on CUDA kernels and performance optimization, covering GEMM, FlashAttention, Tensor Cores, CUTLASS, quantization, KV cache, NCCL, and profiling.

Switchless RoCE collectives, NCCL profiles and fabric tooling for DGX Spark inference recipes

Experimental Explicit Communications API for Kokkos

Curated resources on RDMA, collectives, and AI cluster fabrics for GPU performance engineers

N-Ways to Multi-GPU Programming

Sample examples of how to call collective operation functions on multi-GPU environments. A simple example of using broadcast, reduce, allGather, reduceScatter and sendRecv operations.

Communication-layer expert-parallel MoE dispatch/combine optimization on H100 with PyTorch all-to-all and DeepEP

A Julia wrapper for the NVIDIA Collective Communications Library.

Python Distributed Non Negative Matrix Factorization with custom clustering

🎹 Instruct.KR 2025 Summer Meetup: 오픈소스 LLM, vLLM으로 Production까지 🎹

Build, load, and verify NCCL for switchless DGX Spark RoCE cycles