tensor-parallelism

🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

A scalable, agentic-first, and HuggingFace-native RL framework for research (9k lines).

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

Slicing a PyTorch Tensor Into Parallel Shards

Decentralized LLMs fine-tuning and inference with offloading

Qwen3.8-Flash-Next (MoE) and Qwen3.8-27B on one or two 16 GB RTX 50-series GPUs: experts offloaded to CPU, two-GPU tensor parallel. 64 GB of RAM is enough for Flash-Next. Formerly ninfer-tp2.

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

An Efficient and Versatile Inference Engine for Distributed LLM Serving

A custom LLM inference engine for low-cost NVIDIA GPUs: Quadro K420 and K1200, two K1200s joined into one 8 GB card, GTX 1080 and RTX 3080. Runs Bonsai 2B, 4B, 9B and 27B models with an OpenAI-compatible server.

llama.cpp optimized for NVIDIA Tesla V100 (SM70): Qwen3.8-Flash-Next with MTP (prefill 3402 tok/s at 200K context on 6x V100-16GB; also runs on 2x V100-16GB) and Qwen3.8-27B with DFlash2 (decode up to 220 tok/s); 2-6 GPUs, 262K-512K context, multimodal, concurrent serving, built-in web UI.

Switchless RoCE collectives, NCCL profiles and fabric tooling for DGX Spark inference recipes

Simulate distributed LLM training and inference across GPU clusters in your browser. Memory, throughput, cost, and parallelism strategy planning.

JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)

A distributed training framework for large language models powered by Lightning.

Tensor-parallel DeepSeek V4 Flash inference on dual AMD Strix Halo over OdinLink USB4/TB5 RDMA or Mellanox RoCE v2.

A decision library for running local LLMs on multi-GPU machines without NVLink: parallelism, communication, quantization, speculative decoding, KV cache. Every conclusion carries its evidence grade and reproducible conditions.

vLLM for NVIDIA CMP 170HX (Ampere sm_80): fast, repeatable model serving on one card or several

RDMA fork version of Strix Halo inference engine. Qwen Flash Next Q4_K_XL: >2100pp …

Tencent Hy3 295B MoE (NVFP4) on 2x NVIDIA DGX Spark — TP2 over 200GbE, 256K context, 26 tok/s end-to-end. Tuned, benchmarked, agent-ready.

DeepSeek-V4-Flash-0731 tuned on 4x DGX Spark (GB10) at TP=4 — 123.13 tok/s decode. Every variable measured one at a time, with the values that lost.