Rhythm Syed

I just found the fastest serving engine for LLMs: (fully open-source & makes vLLM 7x faster) LMCache is an LLM serving engine designed to reduce time-to-first-token and increase throughput, especially under long-context scenarios. At a high level, LMCache is like a CDN of KV caches. It sits between the LLM serving engine, like vLLM, and the external data by storing the KV caches of reusable texts across various locations like GPU, CPU, and Local Disk. This way, it can boost vLLM with 7x faster access to 100x more KV caches. Find a link to the GitHub repo in the comments! ____ Find me → Avi Chawla Every day, I share tutorials and insights on DS, ML, LLMs, and RAGs.

1,240

37 Comments

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer PhD, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here: https://lnkd.in/g_XEgAuJ

4,377

165 Comments

Clifford neural networks often suffer from longer compute times and limited training memory. This paper applies several techniques from computer science to speed up the mathematical operations. It's always great to see gains from interdisciplinary work! https://lnkd.in/eiJq-rC6

110

Catching up on improving LLMs by scaling inference-time compute, new LLM scaling laws for precision, and more! "Noteworthy AI Research Papers 2024, Part 2" is now formatted and published. It was certainly an eventful year for this fast-moving field: https://lnkd.in/gciNkD5U In this second part, I cover: 1. Some of the major open-weight LLM releases from last year, featuring more sophisticated pre- and post-training pipelines 2. Techniques for improving LLMs by scaling inference-time compute 3. The different multimodal LLM paradigms 4. Replicating OpenAI's reasoning capabilities in the o1 model 5. New LLM scaling laws for precision 6. Insights from learning with synthetic data I hope you had a great start to the year and find these insights useful: Happy reading!

694

27 Comments

Today marks another step toward making Reinforcement Learning (RL) faster, simpler, and scalable for builders harnessing the Essential Cloud for AI. Our support on the development of torchforge - a new PyTorch-native framework - is helping researchers take large-scale RL workloads from experimentation to production. In partnership with Meta and Stanford University, torchforge was validated on CoreWeave’s Cloud Platform proving stable, efficient, and highly performant – ready to power the next generation of RL workloads. torchforge is now supported on SUNK. More in our recent blog: https://hubs.la/Q03PLJC10

136

2/2 papers from my PhD students, Parjanya Prashant and Jiongli Zhu, accepted at NeurIPS 2025. 📑 TOKENSWAP: A LIGHTWEIGHT METHOD TO DISRUPT MEMORIZED SEQUENCES IN LLMS (SPOTLIGHT) 🔎 The Problem: Large language models sometimes reproduce portions of their training data verbatim, raising copyright and legal risks. Importantly, this can occur even under normal, benign prompting, meaning that users may inadvertently generate copyrighted text without intent. Existing solutions to mitigate this require access to proprietary training data or expensive model retraining. 🛠️ Our Solution: TokenSwap works by selectively replacing common grammar tokens (like "the," "and," "of") with predictions from a small auxiliary model during text generation. This disrupts memorized sequences while preserving model capabilities. 📊 Key Results: Up to 10x reduction in verbatim memorization across Llama, Pythia, and OLMo No access to training data or model weights required Easy integration: Install our verbatim-llm library and add just 2 lines to your existing HuggingFace .generate() calls 🌍 Impact: This approach offers a practical path for LLM users to mitigate copyright risks without needing to retrain models or access proprietary datasets. 📄 Paper: https://lnkd.in/gqhFVxpV 💻 Code: https://lnkd.in/gE4Dqb_S 📑 KAIROS: SCALABLE MODEL-AGNOSTIC DATA VALUATION 🔎 The Problem: Training data quality shapes model accuracy, compliance, and AI asset value. Data valuation provides a systematic way to measure and address data quality. However, existing methods fall short: model-based scores inherit bias, Shapley-based methods are infeasible at scale, and Wasserstein surrogates (LAVA, SAVA) misrank examples and run slowly. 🛠️ Our Solution: KAIROS computes each example’s contribution to dataset quality using a closed-form Maximum Mean Discrepancy (MMD) influence score. It requires no retraining, guarantees faithful leave-one-out rankings, and supports efficient online updates for streaming data. 📊 Key Results: Up to 50× faster than State-of-the-Art (LAVA/SAVA) Rankings that are 99% faithful to exact leave-one-out (vs. ~60% for LAVA) Robust across label noise, feature noise, and malicious attack detection Scales to million-example datasets like ImageNet in minutes 🌍 Impact: KAIROS provides a principled and scalable solution for data valuation, helping practitioners detect corrupted or mislabeled data, and support data pricing. Its model-agnostic nature is also well-suited with regulatory compliance. 📄 Paper: https://lnkd.in/g_Fbd-bQ 💻 Code: https://lnkd.in/gBfEYY-w

104

Some signs that Waymo has jumped on the end-to-end machine learning bandwagon. Junko Yoshida has a scoop from attending The Autonomous: signs that Waymo is doing tentative deployments of end-to-end machine learning in their fleet. Complete with a quote from Waymo that seems to support this (or at least does not deny it). Such a change has deep implications for scalability as well as safety. More about this in the article. I fully expect that Waymo will be smarter than a simplistic pixels-in/driving-out approach, as indicated in the story. But what that will look like is speculation at this point. https://lnkd.in/e4vTmRTs

88

9 Comments

🚀𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗟𝗟𝗠 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝘄𝗶𝘁𝗵 𝗞𝗩 𝗖𝗮𝗰𝗵𝗲 𝗤𝘂𝗮𝗻𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻 We’re excited to share our latest work on improving inference-time efficiency for LLMs through KV cache quantization—a critical step toward making long-context reasoning more scalable and memory-efficient. 🧠𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗺𝗼𝗱𝗲𝗹𝘀 & 𝗶𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲-𝘁𝗶𝗺𝗲 𝘀𝗰𝗮𝗹𝗶𝗻𝗴 Modern reasoning models often require long responses to “think” through problems before arriving at a final answer. Inference-time scaling methods make this even more compute-intensive. While these approaches improve model performance, they incur higher latency and demand more GPU memory. 💡𝗞𝗩 𝗖𝗮𝗰𝗵𝗲 𝗾𝘂𝗮𝗻𝘁𝗶𝘇𝗮𝘁𝗶𝗼𝗻 The KV cache stores the intermediate representations from previous tokens to accelerate autoregressive decoding. Think of it as the model’s short-term memory—just as humans recall previous parts of a conversation to respond quickly, the KV cache helps models maintain and build on prior context. For long sequences, the KV cache can consume more GPU memory than the model weights. During inference, LLM decoding becomes memory-bound, with most of the time spent on data transfer rather than computation. This has led to active research on KV cache quantization but quantization errors can accumulate as more tokens are generated, causing later tokens to deviate from expected outputs. ✨𝗪𝗵𝗮𝘁’𝘀 𝗻𝗲𝘄 𝗶𝗻 𝘁𝗵𝗶𝘀 𝘄𝗼𝗿𝗸? We introduce 𝚂̲𝚀̲𝚞̲𝚊̲𝚝̲ (Subspace-orthogonal KV cache quantization)—a new method that significantly reduces memory overhead and latency while maintaining model accuracy. SQuat constructs a subspace that captures critical task-relevant information, then enforces quantization errors to lie orthogonal to this subspace, minimizing their effect on the output of the attention mechanism. 𝗛𝗶𝗴𝗵𝗹𝗶𝗴𝗵𝘁𝘀: ✅Training-free: No fine-tuning or calibration data needed ✅On-the-fly: Runs during inference without modifying the model ✅Theory-grounded: Built on a theoretical foundation we developed ⚡𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁: • Reduces GPU peak memory by 2.17× to 2.82× • Improves throughput by 2.45× to 3.60× • outperform existing KV cache quantization methods on benchmark tasks 📄paper: https://lnkd.in/emKhAVZu 💻 code: https://lnkd.in/e8TJ7N3R 👏 Joint work with my amazing co-authors Ligong Han, Kai Xu, Akash Srivastava, at the Red Hat AI innovation team (https://lnkd.in/exd6QDbk) Chris Wright, Ruchir Puri, Steven Huels, Joe Fernandes, Tushar Katarki, Ritika Gunnar, Jason McGee, Máirín Duffy, Luke Inglis, Mark Kurtz, Nick Hill, Tyler Michael Smith, vLLM 

206

1 Comment

Stanford CS234: Reinforcement Learning These lectures look like a nice introduction to reinforcement learning (RL). After the impact of RL in recent models like DeepSeek-R1 and o1, it's worth learning about RL today. https://lnkd.in/ev-YUEdc -- Learn how to build with LLMs, RAGs, and AI agents in my new courses: https://lnkd.in/eug3D2-h Use code AGENT20 for 20% off.

1,098

22 Comments

Recent research by UIUC and Intel Labs has introduced a new jailbreak technique for Large Language Models (LLMs) known as InfoFlood. This method takes advantage of a vulnerability termed "Information Overload," where excessive linguistic complexity can circumvent safety mechanisms without the need for traditional adversarial prefixes or suffixes.   InfoFlood operates through a three-stage process: Linguistic Saturation, Rejection Analysis, and Saturation Refinement. Initially, it reformulates potentially harmful queries into more complex structures. If the first attempt does not succeed, the system analyzes the response to iteratively refine the query until a successful jailbreak is achieved. Empirical validation across four notable LLMs—GPT-4o, GPT-3.5-turbo, Gemini 2.0, and LLaMA 3.1—indicates that InfoFlood significantly surpasses existing methods, achieving success rates up to three times higher on various benchmarks.   The study underscores significant vulnerabilities in current AI safety measures, as widely used defenses, such as OpenAI’s Moderation API, proved ineffective against InfoFlood attacks. This situation raises important concerns regarding the robustness of AI alignment systems and highlights the necessity for more resilient safety interventions. As LLMs become increasingly integrated into diverse applications, addressing these vulnerabilities is crucial for ensuring the responsible deployment of AI technologies and enhancing their safety against emerging adversarial techniques.  Arxiv: https://lnkd.in/eBty6G7z

53

4 Comments

DeepSeek recently published a paper on "Native Sparse Attention". Here is why it matters. The attention mechanism is core to the transformer architecture, which in turn is a building block for today's LLMs. Unlike recurrent networks, the attention mechanism can find causal relationships even in long inputs. Simply put, if you feed a 10,000 word book to a recurrent network, it is heavily compressed. Some of the causal relationships in input are lost. Example: Why did Harry Potter get his scar? This long range dependency could be lost. The attention mechanism is able to avoid this. It takes the entire input sequence together, and computes their relationships together, conserving long-range dependencies. DeepSeek has just published how they sped up this mechanism, using GPU-specific optimizations. Instead of computing attention in a dense manner (all-to-all O(N²) time complexity), DeepSeek uses SPARSE attention. Think of this as a sliding window of blocks, reducing time complexity to O(N*W). The unique contribution of this paper is the native support for this kind of sparse attention compute. DeepSeek is going down to the level of hardware optimisations to speed up training. Let me know your thoughts around this. Cheers! #AI #LLMs #DeepSeek

133

2 Comments

Excited to share about two manuscripts to be presented at ICASSP 2025! 🎉 1️⃣ Speech-N-LlaMA: Solving the Exposure Bias Problem in Speech LLMs 🤖 🤝 Research done with LlaMA Speech team, AI at Meta The challenge: LLMs are sequence-to-sequence networks that use teacher forcing (golden output) during training, but inference happens auto-regressively → creating an "exposure bias" gap! Our solution: N-pass decoding during training to bridge this gap Results: Up to 18% relative WER improvement with zero additional inference compute! 2️⃣ DiffSSD: Catching Up to Diffusion-Based Speech Generators 🤝 Done during and w/ PhD collaborators at Purdue University and ISPL - Politecnico di Milano The problem: Showed current synthetic speech detectors are falling behind! They don't generalize to recent diffusion-based generators Our solution: DiffSSD - a new 200-hours dataset featuring speech from both open-source and commercial diffusion-based generators Impact: Detectors trained with this dataset can counter synthetic speech from the latest speech generator tools ICASSP proceedings can be found in comment. Thanks to collaborators Gil, Desh, Ozlem, and all peers from LlaMA Speech team, Kratika, Prof. Delp, and Prof. Bestagini. #AI #SpeechLLMs #GenerativeAI #Deepfake #LLMs

177

5 Comments

For years, more training compute meant more capable AI models. That used to be true. Toby Ord’s new technical report shows why this rule is breaking and why inference compute now tells the real story. 🔗 Read the report: https://lnkd.in/eMk4hawd ▶︎ How is AI progress changing? AI progress is no longer mainly about training ever-bigger models. The biggest performance gains now come from how much compute a model uses while running. This shift changes what counts as “frontier” AI and who can build or control it. ▶︎ What does “inference scaling” mean? It means spending more compute when the model answers a question instead of when it’s first trained. More inference compute lets the same model reason longer or try more options, boosting quality without retraining. It’s like giving an existing brain more thinking time instead of building a bigger one. ▶︎ Who could gain or lose power if AI depends on inference compute? Those who control large amounts of compute will hold more power. Access to compute will shape who gets top performance, creating bigger gaps between users and companies. AI could become more like a pay-to-play service than an open technology. ▶︎ Why might today’s AI rules stop working? Current laws focus on how much compute goes into training a model. But if capability comes from inference, smaller models could still become highly powerful. Policymakers will need new ways to measure and govern real-world capability, not just training size. ▶︎ Could AI start improving itself – and what happens then? Yes. Developers can use inference-heavy models to train better versions of themselves, repeating the process to climb quickly in capability. This could speed up progress toward general-purpose systems while making it harder for outsiders to see how far things have advanced. If training compute has become a weaker proxy for capability, how should policy adapt? Should we still rely on training compute thresholds and other model-based governance mechanisms? What else could we do?

84

7 Comments

DeepSeek's LLMs made a big splash, but more interesting is their recent research papers. Shayan Mohanty writes an overview of them, outlining their three main arcs: efficiency, HPC Co-Design, and RL for emergent reasoning. https://lnkd.in/enifbYaU

965

13 Comments