After the release, OpenClaw quickly added support for it. From the pricing structure, Flash is actually extremely cheap, while Pro is relatively expensive. The gap between the two is around 12x.

V4 also quickly appeared on OpenRouter’s usage ranking, which shows that people were already trying it in real workflows.

Then, on the second day, DeepSeek launched a 75% discount for V4-Pro API.

The DeepSeek-V4-Pro API is now available at a limited-time 75% discount, ending at 15:59 UTC on May 5, 2026.

My guess is that the initial pricing of Pro was indeed a bit too high. For many people doing coding or agentic coding, they would naturally use Pro or Pro Max-like settings. After trying it for a day, it could easily cost dozens of RMB. Compared with a fixed coding plan, that is simply too expensive.

DeepSeek may have noticed this quickly, so it introduced the limited-time discount.

This also puts OpenRouter in an awkward position. If OpenRouter cannot adjust prices in time, many users will simply go back to the official API.

Then, on the third day, DeepSeek announced another price cut:

Starting immediately, the input cache hit price for all DeepSeek API products has been reduced to one-tenth of the original price. Build applications more efficiently at a lower cost.

This cache price cut is the real killer move.

Once the cache price drops to one-tenth of the original level, few vendors can really follow. DeepSeek is clearly starting to grab market share. My feeling is that this price may actually be close to the normal price level after Huawei’s inference servers begin shipping at scale.

It almost feels intentional.

First, set the price high.
Then cut it twice.
Get trending twice.
And finally land at a price point that looks extremely competitive.

The result is that DeepSeek-V4 suddenly looks like one of the highest cost-performance options on the market.

This is not just a model release.

It is a pricing campaign, a developer acquisition campaign, and probably the beginning of a new round of inference cost competition.

Inference System Design Principles

DeepSeek first introduced the design principles of the inference system, with optimization goals focused on: greater throughput and lower latency.

The solution currently adopted by DeepSeek: Large-Scale Node Expert Parallelism (EP)

• EP significantly increases batch size, thereby improving the efficiency of GPU matrix multiplication and boosting throughput.

• EP distributes experts across different GPUs, with each GPU only needing to compute a small number of experts (thus reducing memory access demands), lowering latency.

Introduced Complexity:

• EP introduces cross-node transmission. To optimize throughput, a suitable computation workflow must be designed to allow transmission and computation to occur synchronously.

• EP involves multiple nodes, naturally requiring Data Parallelism (DP), with load balancing needed between different DP instances.

Large-Scale Cross-Node Expert Parallelism

DeepSeek employs a multi-machine, multi-card expert parallelism strategy:

• Prefill: Router Expert EP32, MLA, and Shared Expert DP32. One deployment unit consists of 4 nodes, 32 redundant router experts, with 9 router experts and 1 shared expert per card.

• Decode: Router Expert EP144, MLA, and Shared Expert DP144. One deployment unit consists of 18 nodes, 32 redundant router experts, with 2 router experts and 1 shared expert per card.

The multi-machine, multi-card expert parallelism introduces significant communication overhead, so dual-batch overlapping is used to mask communication costs and improve overall throughput.

• In the prefill phase, computation and communication of two batches are interleaved—one batch’s computation can mask the communication overhead of the other.

• In the decode phase, execution times vary across stages, so the attention part is split into two stages, creating a 5-stage pipeline to overlap computation and communication.

Due to the use of large-scale parallelism (including expert parallelism and data parallelism), some GPUs may become overloaded, necessitating computational and communication load balancing.

• Prefill Load Balancer

  • Core Issue: Variations in the number and length of requests across different Data Parallelism (DP) instances lead to differences in core-attention computation and dispatch transmission volumes.

  • Optimization Goal: Ensure roughly equal computation load across GPUs (core-attention load balancing) and similar token input volumes (dispatch transmission load balancing) to avoid prolonged processing times on some GPUs.

• Decode Load Balancer

  • Core Issue: Variations in the number and length of requests across DP instances result in differences in core-attention computation (related to KVCache usage) and dispatch transmission volumes.

  • Optimization Goal: Ensure roughly equal KVCache usage across GPUs (core-attention load balancing) and similar request volumes (dispatch transmission load balancing).

• Expert-Parallel Load Balancer

  • Core Issue: For a given MoE model, certain naturally high-load experts exist, leading to uneven expert computation loads across GPUs.

  • Optimization Goal: Balance expert computation across GPUs (i.e., minimize the maximum dispatch reception volume across all GPUs).

Real-World Statistics of the Online Inference System

All DeepSeek R1/V3 services use H800 GPUs. Matrix computations and dispatch transmissions use FP8 format consistent with training, while core-attention computations and combine transmissions use BF16 consistent with training, maximizing service performance.

Since service loads are high during the day and low at night, all servers handle inference during peak load times, while some machines are freed up for research and training during low-load periods.

Over the past 24 hours (Beijing time, 2025/02/27 12:00 to 2025/02/28 12:00), the total node usage for DeepSeek V3 and R1 inference services peaked at 278 nodes and averaged 226.75 nodes (each node with 8 H800 GPUs). Assuming a GPU rental cost of $2/hour, the total daily cost is $87,072.

Within this 24-hour statistical period, DeepSeek V3 and R1 recorded:

• Total input tokens: 608 billion, of which 342 billion tokens (56.3%) hit the KVCache disk cache.

• Total output tokens: 168 billion. The average output rate was 20–22 tokens per second (tps), with an average KVCache length of 4,989 per output token.

• Average throughput per H800 GPU:

  • For prefill tasks: ~73.7k tokens/s input throughput (including cache hits).

  • For decode tasks: ~14.8k tokens/s output throughput.

These statistics encompass all loads from web, app, and API usage. If all tokens were priced according to DeepSeek R1’s rates, the theoretical daily revenue would be $562,027, yielding a cost-profit margin of 545%.

In reality, DeepSeek doesn’t generate this much revenue because V3’s pricing is lower, paid services only account for a portion of usage, and discounts are offered at night.

What Can We Infer?

Rumors previously suggested that DeepSeek’s official deployment consisted of a 320-H800 inference cluster. Now, it appears to be 278 nodes—2,224 H800s. Officially, DeepSeek acknowledges owning at least 10,000 H800s, meaning the GPUs used for inference are relatively few.

• Cost: Average of 226.75 nodes (1,814 GPUs), at $2/hour per GPU, yields a daily cost of $87,072.

• Revenue: Input: 608B tokens; Output: 168B tokens, resulting in a daily revenue of $562,027.

• Gross Daily Profit: $474,955 = ~3,457,672.4 RMB/day.

However, the above calculation has flaws. Even if the 6x profit margin is halved to 3x, the profit margin remains very high. Many domestic vendors deploying DeepSeek have shut down API services due to losses, raising questions about where the problem lies.

In an interview, Liang Wenfeng said: “We just do things at our own pace, then calculate costs and set prices. Our principle is not to lose money.”

This suggests DeepSeek is currently profitable, with earnings likely reinvested into R&D. We look forward to R2’s release soon.

Subscribe now

According to the official introduction, the Fire-Flyer File System (3FS) is a parallel file system designed to fully utilize the bandwidth of modern SSDs and RDMA networks.

• Achieves an aggregate read throughput of 6.6 TiB/s in a 180-node cluster.

• Delivers 3.66 TiB/minute throughput in the GraySort benchmark on a 25-node cluster.

• Provides peak throughput of over 40 GiB/s per client node in KVCache lookups.

• Features a distributed architecture with strong consistency semantics.

• Used for training data preprocessing, dataset loading, checkpoint saving/reloading, embedding vector search, and KVCache lookups in inference for V3/R1.

Smallpond, on the other hand, is a data processing framework built on top of 3FS.

Fire-Flyer File System (3FS)

3FS is part of the Fire-Flyer AI-HPC developed by DeepSeek. It is detailed in the paper Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning.

Fire-Flyer AI-HPC consists of three components: the HAI Platform (open-sourced two years ago), 3FS (open-sourced today), and HaiScale (yet to be open-sourced).

In summary, 3FS has several key features:

1. High-Performance Design: 3FS is tailored to leverage the high IOPS (input/output operations per second) and throughput of NVMe SSDs, as well as RDMA networks. This design enables it to efficiently handle large-scale data requests, meeting the demands of deep learning and large-scale computing.

2. System Architecture: The 3FS system comprises four roles: cluster manager, metadata service, storage service, and client. The metadata and storage services periodically send heartbeat signals to the cluster manager to ensure system stability and efficiency. Multiple cluster managers ensure high availability.

3. Request Control Mechanism: 3FS implements a request transmission control mechanism to alleviate network congestion. Upon receiving a read request, the storage service asks the client for permission to transfer data. This limits the number of concurrent senders, maintaining good performance under high load.

4. Strong Consistency with Chain Replication: 3FS adopts the Chain Replication and Allocate Query (CRAQ) approach to provide strong consistency. File contents are split into blocks and replicated across a series of storage targets, fully unleashing the throughput and IOPS of all SSDs.

5. High Throughput: By optimizing batch write and read operations, 3FS achieves write speeds exceeding 10 GiB/s per node, accelerating checkpoint saving and loading, and reducing latency during training.

6. 3FS-KV System: 3FS also supports 3FS-KV, a shared-storage distributed data processing system built on 3FS. It supports key-value storage, message queues, and object storage models, further enhancing system flexibility and performance.

3FS provides robust storage support for deep learning and large-scale computing, effectively meeting demands for high throughput and low latency.

Description from the Paper:

Performance

Peak Throughput

Read throughput test results for a 3FS cluster: The cluster consists of 180 storage nodes, each equipped with 2×200Gbps InfiniBand NICs and 16×14TiB NVMe SSDs. Over 500+ client nodes, each with a 1×200Gbps InfiniBand NIC, were used for the read stress test. Under background traffic from training jobs, the aggregate read throughput reached approximately 6.6 TiB/s.

Sorting Performance

The test cluster consists of 25 storage nodes (2 NUMA domains per node, 1 storage service per NUMA, 2×400Gbps NICs per node) and 50 compute nodes (2 NUMA domains, 192 physical cores, 2.2 TiB RAM, and 1×200Gbps NIC per node). Sorting 110.5 TiB of data across 8192 partitions took 30 minutes and 14 seconds, achieving an average throughput of 3.66 TiB/minute.

KVCache

KVCache is designed to optimize the LLM inference process by caching keys and value vectors from previous tokens in the decoder layers, avoiding redundant computation. The figure above shows the read throughput for all KVCache clients, highlighting peak and average values, with a peak throughput of up to 40 GiB/s. The figure below shows the IOPS of delete operations during garbage collection (GC) over the same period.

Why is a Specialized File System Like 3FS Needed?

In LLM scenarios, there’s a need for highly concurrent, high-throughput, and scalable distributed file systems that also demand strong consistency, intelligent routing, and cache management. Systems like 3FS, which offer high-performance solutions tailored for RDMA and SSDs, flexible metadata design, and asynchronous zero-copy I/O, become highly valuable.

Code Analysis

One notable aspect of 3FS’s implementation is that DeepSeek used Rust to develop the chunk_engine.

The chunk_engine is a core module at the bottom layer of the 3FS storage service, responsible for managing, allocating, and reclaiming physical disk blocks. The upper layers can read and write block data through this engine. It primarily uses cxx to automatically generate C++ bindings, allowing C++ code to directly call Rust code.

In recent years, Rust has gained popularity in the MLSys (Machine Learning Systems) field. For example, Hugging Face’s tokenizers are also implemented in Rust. The DeepSeek team likely chose Rust for the chunk_engine due to its maintainability, memory safety, and excellent performance.

The DeepSeek team may also have used the Rust framework Tokio in backend services, as I found several Rust open-source projects, including Tokio, in Quant AI’s open-source initiatives. I sincerely hope more teams adopt Rust for developing machine learning systems.

Smallpond

Smallpond is a lightweight data processing framework built on top of DuckDB and 3FS. It supports lightweight, high-performance data processing and scales to petabyte-scale datasets.

Installation and usage are straightforward, with a minimal API offering two types: one for dynamically building dataflow graphs and another for static construction.

Installation:

pip install smallpond

Usage Example:

# Download example data
wget https://duckdb.org/data/prices.parquet

import smallpond
# Initialize session
sp = smallpond.init()
# Load data
df = sp.read_parquet("prices.parquet")
# Process data
df = df.repartition(3, hash_by="ticker")
df = sp.partial_sql("SELECT ticker, min(price), max(price) FROM {0} GROUP BY ticker", df)
# Save results
df.write_parquet("output/")
# Show results
print(df.to_pandas())

For performance, refer to the sorting performance of 3FS.

Conclusion of DeepSeek Open Source Week

DeepSeek Open Source Week concludes today. Thank you, DeepSeek, for sharing valuable resources for everyone to learn and use.

Many teams have already taken action and achieved tangible performance improvements. For instance, the vLLM team replaced TRITON_MLA with FLASHMLA, boosting throughput by 2-16%, delivering real results.

These projects open-sourced by DeepSeek will continue to influence us. Our journey of intense learning continues.

more

• day0: https://aigc.openbot.ai/p/deepseek-opensourceweek-is-coming

• day1: https://aigc.openbot.ai/p/deepseek-open-source-week-day-1-in

• day2:https://aigc.openbot.ai/p/day-2-of-deepseek-opensourceweek

• day3:https://aigc.openbot.ai/p/deepseek-opensourceweek-day-3-deepgemm

• day4:https://aigc.openbot.ai/p/deepseek-opensourceweek-day-4-in

Subscribe now

DualPipe is a bidirectional pipeline parallelism algorithm designed for computation-communication overlap in V3/R1 training. Meanwhile, EPLB serves as an expert-parallel load balancer for V3/R1.

The final project, profile-data, primarily releases analytical data from DeepSeek’s infrastructure for training and inference. So far, it includes data on Prefilling for both training and inference, while the Decoding analysis data for inference has yet to be made public.

Today, we’ll dive into an analysis of the DualPipe and EPLB projects, both of which lean toward engineering optimization.

DualPipe

DualPipe is mentioned in the DeepSeek V3 paper as a bidirectional pipeline parallelism communication algorithm, mainly used to optimize data interaction and training efficiency in large-scale models.

Key Features:

• Computation-Communication Overlap
  DualPipe’s design aims to maximize cluster computing performance by achieving full overlap of computation and communication during forward and backward passes, reducing idle wait times typical in traditional pipeline parallelism. This is especially critical for expert parallelism (Expert Parallelism) across nodes in MoE models.

• Bidirectional Scheduling
  DualPipe employs a bidirectional scheduling strategy, feeding data from both ends of the pipeline simultaneously to reuse hardware resources efficiently. It also incorporates a sophisticated yet highly effective 8-step scheduling strategy.

• Memory Optimization
  DualPipe deploys the shallowest layers (including the embedding layer) and the deepest layers (including the output layer) on the same pipeline level (PP Rank), enabling physical sharing of parameters and gradients to further enhance memory efficiency.

Pipeline Bubble and Memory Usage Comparison (Pipeline bubble refers to idle wait time)

For those interested, you can check out the code—it’s not very long and is great for learning.

EPLB

EPLB (Expert Parallelism Load Balancer) is primarily designed to optimize the distributed deployment of MoE models. It ensures load balancing among different experts in the MoE portion by replicating shared experts and fine-grained high-load experts across multiple GPUs in the cluster. This allows GPUs to handle more "hot data" (data sent to shared experts) efficiently.

EPLB isn’t detailed in DeepSeek’s paper, and its code is remarkably concise at just 160 lines.

Key Features:

• Load Balancing Optimization
  It replicates high-load experts (a strategy we can call "redundant expert strategy") and uses heuristic adjustments for expert allocation to ensure balanced workloads across GPUs.

• Hierarchical Load Balancing
  EPLB adopts a three-tier structure: node-level → intra-node expert replication → GPU allocation. It prioritizes assigning experts from the same group to the same node to minimize cross-node data transfers, then ensures load balancing at each layer. This approach, combined with DeepSeek V3’s Group-Limited Expert Routing strategy, significantly boosts distributed training efficiency.

• Dynamic Scheduling Strategy
  EPLB dynamically selects load balancing strategies based on the situation—using a hierarchical strategy during the prefilling phase and a global strategy during the decoding phase.

Let’s Look at the Code:

Redundant Expert Strategy

def replicate_experts(weight: torch.Tensor, num_phy: int):
    # Replicate high-load experts
    for i in range(num_log, num_phy):
        redundant_indices = (weight / logcnt).max(dim=-1).indices
        phy2log[:, i] = redundant_indices
        logcnt[arangen, redundant_indices] += 1

Hierarchical Load Balancing

def rebalance_experts_hierarchical():
    # Step 1: Pack expert groups to nodes
    tokens_per_group = weight.unflatten(-1, (num_groups, group_size)).sum(-1)
    group_pack_index, group_rank_in_pack = balanced_packing(tokens_per_group, num_nodes)

    # Step 2: Build redundant experts within nodes
    tokens_per_mlog = weight.gather(-1, mlog2log).view(-1, num_logical_experts // num_nodes)

    # Step 3: Pack physical experts to GPUs
    tokens_per_phy = (tokens_per_mlog / mlogcnt).gather(-1, phy2mlog)

Dynamic Scheduling Strategy

def rebalance_experts():
    if num_groups % num_nodes == 0:
        # Use hierarchical strategy
        phy2log, phyrank, logcnt = rebalance_experts_hierarchical()
    else:
        # Use global strategy
        phy2log, phyrank, logcnt = replicate_experts()

Interested readers can explore the full code.

Tomorrow is the final day of DeepSeek Open Source Week—will they drop a heavyweight open-source project? Let’s wait and see!

Subscribe now

Today marks the third day of DeepSeek's Open Source Week, with the release of DeepGEMM right on schedule at 9 AM.

As of now, the project has garnered 3.3k stars since its release.

The official introduction describes DeepGEMM as an FP8-supporting GEMM library compatible with both dense and MoE (Mixture of Experts) GEMM operations, designed for training and inference of V3/R1 models.

A Brief Introduction to GEMM

General Matrix Multiplication (GEMM) is one of the most fundamental and critical operations in deep learning and scientific computing. GEMM refers to the multiplication of two matrices, A and B, to produce a result matrix C, typically expressed as C = A × B.

In deep learning, GEMM underpins core components such as fully connected layers, convolutional layers, and attention mechanisms. For instance, in Transformer architectures, both self-attention and feedforward network layers heavily rely on matrix multiplication. As model sizes grow, GEMM operations dominate the computational time in training and inference, making their performance a key factor in the efficiency of deep learning systems.

Modern GPU architectures, like NVIDIA’s Tensor Core technology, are specifically designed to accelerate matrix multiplication. With the ever-increasing scale of models, the demand for high-performance GEMM implementations continues to rise, especially in large language models (LLMs) and MoE frameworks, where efficient GEMM is critical for real-time inference and cost-effective training.

In the paper DeepSeek LLM: Scaling Open-Source Language Models with Longtermism, DeepSeek mentions GEMM, though it ties into their work in another paper, Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning, which introduces the HAI-LLM training system. For those interested, I recommend checking out the Fire-Flyer AI-HPC paper.

Today’s open-sourced DeepGEMM supports FP8 and is tailored for training and inference of DeepSeek’s V3/R1 models. In the V3 paper, DeepSeek details several optimizations for FP8 training.

The main challenges of FP8 training lie in precision and error handling. To tackle these, DeepSeek implemented the following optimizations:

• Fine-Grained Quantization: Data is split into smaller groups, each with a specific multiplier to maintain high precision.

• Online Quantization: Weights are computed online for each 1x128 activation block or 128x128 weight block, with scaling factors inferred on-the-fly, and activations converted to FP8 in real time.

• Improved Accumulation Precision: FP8 accumulation can introduce random errors, so intermediate results are stored in FP32, then converted back after accumulation.

• Low-Precision/Mixed-Precision Storage and Communication: For MoE model training, FP8 is mixed with BF16/FP32 to ensure dynamic model stability.

For a detailed look at these optimizations, check out the DeepSeek V3 paper.

DeepGEMM Overview

Here’s a summary of its key features:

• FP8 Support: DeepGEMM uses CUDA’s two-stage accumulation to address precision issues.

• Grouped GEMM Support: It improves on CUTLASS’s grouped GEMM, with targeted optimizations for MoE models.

• Just-In-Time Compilation: Through JIT technology, code is dynamically generated and optimized at runtime, boosting performance and flexibility.

• FFMA SASS Interleaving: DeepSeek analyzed SASS compilation results in depth, tweaking FFMA/FADD instructions to enhance fine-grained FP8 GEMM efficiency.

Performance

All metrics show improvement, with the highest gain reaching a 2.7x speedup. DeepSeek notes that performance isn’t optimal in some areas and welcomes PRs from those interested in further optimization.

In the DeepGEMM project’s README, the DeepSeek team provides a detailed breakdown of the optimizations. For those interested, it’s worth diving into the code alongside the documentation for a hands-on exploration.

Spotlight: The interleave_ffma.py File

Today, let’s focus on a specific file in the project: interleave_ffma.py under the jit directory. It contains some clever tricks worth exploring.

Here’s the code:

import argparse
import mmap
import os
import re
import subprocess
from torch.utils.cpp_extension import CUDA_HOME

def run_cuobjdump(file_path):
    command = [f'{CUDA_HOME}/bin/cuobjdump', '-sass', file_path]
    result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
    assert result.returncode == 0
    return result.stdout

def extract_ffma(sass):
    lines = sass.splitlines()
    collected = []
    current = []
    arch_name, func_name = 'N/A', 'N/A'
    skip_next_line = False
    for line in lines:
        if 'code for' in line:
            arch_name = line.lstrip().lstrip('code for ').rstrip()
        elif 'Function :' in line:
            func_name = line.lstrip().lstrip('Function :').rstrip()
        elif 'FFMA' in line:
            current.append(line)
            skip_next_line = True
        elif skip_next_line:
            current.append(line)
            skip_next_line = False
        else:
            if len(current) >= 16:
                assert len(current) % 2 == 0
                collected.append((f'{arch_name}::{func_name}', current))
            current = []
    if os.getenv('DG_PRINT_REG_REUSE', None):
        print(f"Found {len(collected)} FFMA segments")
    return collected

def extract_hex_from_line(line):
    match = re.search(r'/\*\s*(0x[0-9a-fA-F]+)\s*\*/', line)
    assert match
    return int(match.group(1), 16)

def validate(m, offset, le_bytes, num_lines):
    assert len(le_bytes) == num_lines // 2
    assert m[offset:offset + 16] == le_bytes[0]
    for i in range(1, num_lines // 2):
        if m[offset + i * 16:offset + i * 16 + 16] != le_bytes[i]:
            return False
    return True

def parse_registers(line):
    import re
    line = re.sub(r'/\*.*?\*/', '', line)
    line = line.replace(';', '')
    tokens = line.strip().split(',')
    registers = []
    for token in tokens:
        token = token.strip()
        words = token.split()
        for word in words:
            if word.startswith('R'):
                reg = word.split('.')[0]
                registers.append(reg)
    return registers

def modify_segment(m, name, ffma_lines):
    num_lines = len(ffma_lines)
    assert num_lines % 2 == 0
    le_bytes, new_le_bytes = [], []
    reused_list = []
    dst_reg_set = set()
    last_reused, last_dst_reg = False, ''
    num_changed = 0
    for i in range(num_lines // 2):
        dst_reg = parse_registers(ffma_lines[i * 2])[-2]
        low_line, high_line = ffma_lines[i * 2], ffma_lines[i * 2 + 1]
        low_hex, high_hex = extract_hex_from_line(low_line), extract_hex_from_line(high_line)
        le_bytes.append(low_hex.to_bytes(8, 'little') + high_hex.to_bytes(8, 'little'))
        reused = (high_hex & 0x0800000000000000) != 0
        if reused:
            is_first_occurred = dst_reg not in dst_reg_set
            if is_first_occurred or (last_reused and dst_reg == last_dst_reg):
                assert high_hex & 0x0800200000000000, f"{hex(high_hex)}"
                high_hex ^= 0x0800200000000000
                reused = False
                num_changed += 1
            else:
                reused_list.append(i)
        dst_reg_set.add(dst_reg)
        new_le_bytes.append(low_hex.to_bytes(8, 'little') + high_hex.to_bytes(8, 'little'))
        last_reused, last_dst_reg = reused, dst_reg
    if os.getenv('DG_PRINT_REG_REUSE', None):
        print(f" > segment `{name}` new reused list ({num_changed} changed): {reused_list}")
    offsets = []
    offset = m.find(le_bytes[0])
    while offset != -1:
        offsets.append(offset)
        offset = m.find(le_bytes[0], offset + 1)
    offsets = list(filter(lambda x: validate(m, x, le_bytes, num_lines), offsets))
    for offset in offsets:
        for i in range(num_lines // 2):
            m[offset + i * 16:offset + i * 16 + 16] = new_le_bytes[i]

def process(path):
    if os.getenv('DG_PRINT_REG_REUSE', None):
        print(f'Processing {path}')
    output = run_cuobjdump(path)
    segments = extract_ffma(output)
    with open(path, 'r+b') as f:
        mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_WRITE)
        for segment in segments:
            modify_segment(mm, *segment)
        mm.close()

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='Interleave FFMA reg reuse')
    parser.add_argument('--so', help='Path to the SO file')
    args = parser.parse_args()
    process(args.so)

This file is designed to optimize the register reuse patterns of FFMA (Fused Floating-point Multiply-Add) instructions in CUDA-compiled assembly code by modifying the binary file, ultimately improving GPU instruction execution efficiency.

Key Function Breakdown:

SASS Code Extraction

def run_cuobjdump(file_path):
    command = [f'{CUDA_HOME}/bin/cuobjdump', '-sass', file_path]

Uses NVIDIA’s cuobjdump tool to extract SASS (assembly) code from the binary.

FFMA Instruction Analysis

def extract_ffma(sass):

Extracts segments of SASS code containing FFMA instructions, collecting architecture and function names along with the instruction sequences.

Register Usage Analysis

def parse_registers(line):

Parses the registers used in each instruction, identifying those starting with 'R'.

Binary Modification

def modify_segment(m, name, ffma_lines):

Modifies the reuse and yield bits of FFMA instructions by tweaking specific bit patterns (e.g., 0x0800200000000000) to optimize register reuse.

Workflow:

• Reads a compiled CUDA shared library (.so file).

• Extracts SASS code using cuobjdump.

• Identifies and collects all FFMA instruction sequences.

• Analyzes register usage patterns in each FFMA instruction.

• Modifies the reuse flags based on specific rules.

• Writes the optimized instructions back to the original file.

Optimization Strategy:

The tool targets:

• First-time register usage.

• Consecutive reuse of the same destination register.
  By tweaking the reuse and yield bits, it optimizes instruction scheduling.

Usage:

python interleave_ffma.py --so path/to/cuda_lib.so

Analysis:

This tool acts as a post-processing optimizer, running after CUDA compilation to enhance GPU instruction efficiency by tweaking the binary. Its focus on FFMA instruction register reuse is particularly impactful for compute-intensive applications like deep learning.

A regex in the file often stumps readers:

def extract_hex_from_line(line):
    match = re.search(r'/\*\s*(0x[0-9a-fA-F]+)\s*\*/', line)
    assert match
    return int(match.group(1), 16)

In CUDA SASS assembly, instructions often appear like this:

FFMA R8, R8, R6, R4;                  /* 0x5c98078000870808 */

This regex extracts 0x5c98078000870808, the hexadecimal machine instruction encoding. The function:

• Extracts the hex code from the assembly line.

• Converts it to an integer for subsequent modification.

• This step is crucial for locating instructions, modifying specific bits (e.g., reuse and yield flags), and writing them back.

Honestly, DeepSeek’s engineers seem to outshine even some NVIDIA folks when it comes to CUDA mastery! Oh, and they’ve slashed their API prices again.

Subscribe now

On the second day of OpenSourceWeek, the official DeepSeek X account posted an article at 10:24, introducing the second open-source project of Open Source Week: DeepEP.

Since the code was released, it has already garnered 4.3K stars.

Many people are not very familiar with MoE models, so this article will first briefly introduce MoE and some of DeepSeek's work on MoE.

MoE Introduction

The Mixture-of-Experts (MoE) model is a simple extension of the Transformer architecture, rapidly becoming the preferred architecture for medium-to-large-scale language models (2 billion to 600 billion parameters).

Key Advantages:

• Faster pre-training speed compared to dense models

• Faster inference speed compared to models with the same number of parameters

Challenges:
It requires significant memory since all expert systems need to be loaded into memory. Additionally, as it is typically used for medium-to-large models, it often requires parallel processing across multiple GPUs, and communication must be highly efficient.

MoE models primarily consist of two key components:

1. Sparse MoE Layers
   These layers replace the feed-forward network (FFN) layers in traditional Transformer models. An MoE layer contains several "experts" (e.g., 8), each of which is an independent neural network.

2. Gating Network or Router
   This component determines which tokens are sent to which experts. Sometimes, a token may even be routed to multiple experts.

Summary:
A notable advantage of Mixture-of-Experts (MoE) models is their ability to perform effective pre-training with far fewer computational resources than dense models. This means that, under the same computational budget, you can significantly scale up the model or dataset size. Especially during pre-training, MoE models typically reach the same quality level faster than dense models.

DeepSeek MoE

On January 11, 2024, DeepSeek released the paper DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models, making it one of the earliest companies to research MoE models.

In the DeepSeekMoE paper, the concepts of Router Expert and Shared Expert were introduced, and experiments were conducted on increasing fine-grained experts.

In the paper, DeepSeekMoE 16B has 2 shared experts and 64 router experts per layer, with each token activating 2 shared experts and 6 router experts. The 145B version, on the other hand, has 4 shared experts and 128 router experts, with each token activating 4 shared experts and 12 router experts.

The DeepSeekMoE paper also mentions today’s topic: Expert Parallelism.

In the DeepSeek V3 paper, two paragraphs provide more detailed data. DeepSeek R1 was trained based on V3, and although no further details are provided, it is largely consistent with V3.

DeepEP Introduction

DeepEP is a communication library designed for MoE models and Expert Parallelism (EP). It provides high-throughput and low-latency all-to-all GPU kernels, also known as MoE data dispatch and combine. Additionally, the library supports low-precision operations, including FP8.

To align with the group-limited gating algorithm proposed in the DeepSeek-V3 paper, DeepEP offers a set of kernels optimized for asymmetric domain bandwidth forwarding (e.g., from NVLink domain to RDMA domain). These kernels deliver high throughput, making them ideal for training and inference prefetching tasks. They also support control over the number of Streaming Multiprocessors (SM).

For latency-sensitive inference decoding tasks, DeepEP includes a set of low-latency kernels using pure RDMA to minimize latency. The library also introduces a hook-based method for overlapping communication and computation without occupying any SM resources.

Note: The implementation of this library may slightly differ from the DeepSeek-V3 paper.

Performance

NVLink and RDMA Forwarding Test for General Kernels

We tested general kernels on H800 (NVLink max bandwidth ~160 GB/s), with each device connected to a CX7 InfiniBand 400 Gb/s RDMA NIC (max bandwidth ~50 GB/s). The test followed the pre-training setup of DeepSeek-V3/R1: 4096 tokens per batch, hidden dimension of 7168, top-4 grouping, top-8 experts, FP8 dispatch, and BF16 combine.

Low-Latency Kernel Test with Pure RDMA

We tested low-latency kernels on H800, with each device connected to a CX7 InfiniBand 400 Gb/s RDMA NIC (max bandwidth ~50 GB/s). The test followed a typical DeepSeek-V3/R1 production environment setup: 128 tokens per batch, hidden dimension of 7168, top-8 experts, FP8 dispatch, and BF16 combine.

Code Analysis

The optimization strategy focuses heavily on low-latency communication. Here, we’ll highlight this aspect, while other features like dynamic scheduling, asynchronous communication, and stream management can be explored in the relevant documentation and code.

Double Buffering

auto buffer = layout.buffers[low_latency_buffer_idx];
auto next_buffer = layout.buffers[low_latency_buffer_idx ^= 1];

• Alternates between two buffers: one for the current operation, another for the next.

• Uses the bitwise operation ^= 1 to efficiently switch buffer indices.

TMA (Tensor Memory Access) Optimization

• Leverages the Hopper architecture’s TMA instructions to accelerate data transfer.

• Supports low-precision formats like FP8 to reduce communication bandwidth requirements.

IBGDA Direct Communication

// Initialize recv queues for low-latency mode AR
ibgda_initialize_recv_queue<<<num_ranks, 128>>>(rank);

• Uses NVSHMEM’s IBGDA technology for GPU-direct RDMA communication.

• Bypasses CPU involvement entirely to reduce latency.

Expert-Level QP Allocation

_buffer = Buffer(group, 0, num_rdma_bytes, low_latency_mode=True,
num_qps_per_rank=num_experts // group.size())

• Assigns independent Queue Pairs (QPs) to each local expert, eliminating resource contention.

DeepEP Use Cases

• Large-scale MoE model training (e.g., models with hundreds of billions of parameters)

• High-concurrency, low-latency real-time inference services

• Heterogeneous computing tasks such as multimodal applications and scientific computing

Notably, at the end of the repository, DeepSeek mentions using an undocumented NVIDIA instruction for optimization—a true hacker spirit worth learning from!

Subscribe now

This morning at 9:34, DeepSeek announced the first project of Open Source Week on X: FlashMLA. This article provides an in-depth analysis of FlashMLA.

The FlashMLA project has gained significant popularity, with its code already reaching 6.8k stars.

Brief Introduction to MLA

MLA (Multi-Head Latent Attention) is an optimization method for Multi-Head Attention (MHA) proposed by DeepSeek in their paper DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model.

In Transformer models, MHA is one of the most computationally intensive modules. To maintain high efficiency in large-scale scenarios, further optimization is necessary.

MLA can be considered a variant of MHA. In its implementation, it borrows some concepts from FlashAttention. The DeepSeek-V2 paper primarily compares it with MHA, GQA, and MQA, with optimization results shown in the figure below:

In some inference frameworks, MLA has also been implemented. As shown below, after integrating MLA into SGLang, throughput increased by 2-3 times.

Using FlashMLA

Environment Requirements:

• Hopper GPUs

• Minimum CUDA 12.3

• Minimum PyTorch 2.0

Installation:

git clone https://github.com/deepseek-ai/FlashMLA  
python setup.py install  

Performance:
The repository provides a Benchmark file that can be run directly. Official results show that on an H800 SXM5 with CUDA 12.6, it achieves speeds of up to 3000 GB/s under memory-bound configurations and 580 TFLOPS under compute-bound configurations.

Code Analysis

FlashMLA’s codebase is relatively small with minimal dependencies.
The primary optimization techniques are as follows:

1. Computation Chunking and Scheduling Optimization

template<int kHeadDim_, int kBlockM_, int kBlockN_, int kNWarps_>  
struct Flash_fwd_kernel_traits_mla {  
    // Fixed block size of 64x64  
    static constexpr int kBlockM = kBlockM_;    
    static constexpr int kBlockN = kBlockN_;    
    
    // Each block uses 8 warps in parallel  
    static constexpr int kNWarps = kNWarps_;    
    static constexpr int kNThreads = kNWarps * 32;  
    
    // Shared memory optimization  
    static constexpr int kBlockKSmem = kHeadDim % 64 == 0 ? 64 : 32;  
};

Key Points:

• Improves computational efficiency through chunking (block size of 64), paged KV caching, and multi-warp parallelism.

2. Memory Access Optimization

struct Flash_fwd_mla_params {  
    using index_t = int64_t;  
    int b, seqlen_q, d, d_v;  
    int h, h_h_k_ratio, ngroups;  
    bool is_causal;  
    float scale_softmax, scale_softmax_log2;  
    int *__restrict__ cu_seqlens_k;  
    void *__restrict__ q_ptr;  
    void *__restrict__ k_ptr;  
    void *__restrict__ v_ptr;  
    void *__restrict__ o_ptr;  
    void *__restrict__ softmax_lse_ptr;  
    index_t q_batch_stride;  
    index_t k_batch_stride;  
    index_t v_batch_stride;  
    index_t o_batch_stride;  
    index_t q_row_stride;  
    index_t k_row_stride;  
    index_t v_row_stride;  
    index_t o_row_stride;  
    index_t q_head_stride;  
    index_t k_head_stride;  
    index_t v_head_stride;  
    index_t o_head_stride;  
    int *__restrict__ block_table;  
    index_t block_table_batch_stride;  
    int page_block_size;  
    int *__restrict__ tile_scheduler_metadata_ptr;  
    int num_sm_parts;  
    int *__restrict__ num_splits_ptr;  
    void *__restrict__ softmax_lseaccum_ptr;  
    void *__restrict__ oaccum_ptr;  
};

Key Points:

• Uses paged KV caching (block_table, page_block_size).

• Optimized memory layout and access strides (stride).

• Scheduling with tile_scheduler_metadata.

3. Softmax Computation Optimization

for (int mi = 0; mi < size<0>(tensor); ++mi) {  
    MaxOp<float> max_op;  
    max(mi) = zero_init ? tensor(mi, 0) : max_op(max(mi), tensor(mi, 0));  
    #pragma unroll  
    for (int ni = 1; ni < size<1>(tensor); ni++) {  
        max(mi) = max_op(max(mi), tensor(mi, ni));  
    }  
    max(mi) = Allreduce<4>::run(max(mi), max_op);  
    const float max_scaled = max(mi) == -INFINITY ? 0.f : max(mi) * scale;  
    sum(mi) = 0;  
    #pragma unroll  
    for (int ni = 0; ni < size<1>(tensor); ++ni)  {  
        tensor(mi, ni) = exp2f(tensor(mi, ni) * scale - max_scaled);  
        sum(mi) += tensor(mi, ni);  
    }  
    SumOp<float> sum_op;  
    sum(mi) = Allreduce<4>::run(sum(mi), sum_op);  
}

Key Points:

• Uses log2/exp2 instead of log/exp.

• Optimizes with FFMA instructions.

• Warp-level reduction for summation optimization.

4. Double Buffering Optimization

struct SharedStorageMLA {  
    union {  
        struct {  
            // Double buffering for K matrix  
            cute::array_aligned<Element, cosize_v<SmemLayoutQ>> smem_q;  
            cute::array_aligned<Element, cosize_v<SmemLayoutK> * 2> smem_k;  // Double buffer  
            cute::array_aligned<Element, cosize_v<SmemLayoutP>> smem_p;  
            cute::array_aligned<ElementAccum, cosize_v<SmemLayoutRow>> smem_scale;  
        };  
    };  
};

Key Points:

• Hides memory latency with double buffering to improve hardware utilization.

Summary of FlashMLA

FlashMLA is essentially a customized version of FlashAttention. Its current applicable scenarios include:

• Environments requiring CUDA 11+ and SM90+ Hopper architecture.

• Inference or training of multi-head attention with BF16 (Q=576, V=512).

• Large-sequence scenarios requiring integration with split-K schemes to boost throughput.

As shown above, there are still many optimization methods from the official team. Looking forward to tomorrow’s project—could it be infra-related? Perhaps something as challenging as MTP?

Subscribe now

On February 21, DeepSeek announced on X the launch of a warm-up for Open Source Week. Starting next week, they will open-source five projects over five consecutive days. For each of these five projects, I will write detailed articles to introduce them on the day they are announced. Feel free to follow me for the latest analyses.

Today, I’ll make some predictions about which projects might be open-sourced. If I guess even one correctly, I’ll consider it a win.

First, there’s definitely going to be something related to infra.
According to the X post, they are a small team within DeepSeek, sharing small but genuine progress, specifically modules used in online services. They particularly emphasized “small,” which likely points to code related to model inference optimization.

The recent release of DeepSeek-R1 has generated significant buzz, but inference optimization still lacks robust support from major frameworks. It feels like they might directly release some official implementations, as this is currently in high demand.

These optimizations could include deployment and inference strategies mentioned in the DeepSeek V3 technical report, such as Prefilling and Decoding.

Second, the official repository index hints at a paper, which is likely related.

Today, DeepSeek created an "open-infra-index" repo on GitHub, which includes a paper: Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning.

This Fire-Flyer AI-HPC paper mainly discusses optimizations for a training cluster with 10,000 A100 GPUs, focusing on reducing system construction costs and energy consumption. Specifically, compared to DGX-A100, it achieves about 80% of the performance while cutting costs by 60% and energy consumption by 50%. I won’t go into the detailed innovations here—interested folks can read the paper themselves.

Why mention this paper? My guess is that they’ll open-source some of the system code referenced in it. Looking at it, here are a few possibilities:

• HFReduce: A library developed specifically for efficient allreduce operations, designed to optimize GPU communication in PCIe architectures. HFReduce overlaps computation and communication through asynchronous allreduce operations, significantly boosting performance.

• HaiScale: A distributed data-parallel training tool that uses HFReduce as its communication backend, enabling asynchronous allreduce operations during backpropagation to improve training efficiency.

• 3FS: A high-performance distributed file system designed to fully leverage the high IOPS and throughput of NVMe SSDs and RDMA networks. The 3FS system supports efficient read/write operations and achieves traffic isolation under high loads.

• 3FS-KV: A shared-storage distributed data processing system built on 3FS, supporting various data models (e.g., key-value storage, message queues, and object storage) with read-write separation and on-demand startup features.

• HAI Platform: A time-sharing scheduling platform that manages and schedules training tasks to ensure efficient GPU resource utilization.

• Checkpoint Manager: A tool for managing checkpoints during large language model training, supporting rapid recovery from hardware failures to ensure training continuity.

Whether they open-source one or several of these is hard to say, but at the very least, it’ll involve systems from this paper—possibly the HAI platform.

Third, an RL training framework? It’s possible.
They might open-source some RL methods from DeepSeek R1. RL is genuinely tough to train. However, I think the likelihood of this is low.

Fourth, an inference framework? The most likely.
DeepSeek’s internal inference optimization is top-notch, and their API pricing is very low. However, inference costs for external vendors remain high. If they open-source an inference framework, it could help major frameworks optimize efficiency more quickly.

For example, the recently announced NAS (Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention) might come with code. It’d be even better if they included something like distributed KV caching.

Fifth, a big guess about the open-source license.
Currently, DeepSeek uses the MIT license, which is the most permissive one. Many vendors’ open-source licenses are stricter—LLaMA, for instance, has changed its license multiple times and is no longer considered a true open-source model.

Here’s a rundown of licenses for some popular models:

• DeepSeek: MIT, fully open.

• Qwen: Apache 2.0 + additional terms for some models.

• Mistral: Apache 2.0.

• LLaMA: Non-commercial research license, the most restrictive.

Recently, Perplexity’s r1-1776 fiasco was clownish behavior, which might push DeepSeek toward a stricter license. Still, I’m guessing these five projects will stick with MIT. I’ll share more details on the licenses once they’re officially announced.

Thank DeepSeek. Next week, we will continue analyzing the five open-sourced projects.

Subscribe now

Scale Mindset

Scale refers to the replication, development, and expansion of something to generate accumulative, positive feedback effects that can achieve exponential growth.

This mindset encourages y…

Read more