Today marks the release of Gemini 3 Pro, the new state-of-the-art model.

Do yourself a favor: take a moment to explore the links below and try out the tools to see what this model can really do.

1. Gemini 3: Introducing the latest Gemini AI model from Google

2. Start building with Gemini 3

3. Gemini 3 brings upgraded smarts and new capabilities to the Gemini app

4. Google Antigravity

5. Generative UI: A rich, custom, visual interactive user experience for any prompt

Subscribe now

In today’s issue:

• AIGC Papers – π*0.6 brings real-world experience into home robots, “Back to Basics” rethinks diffusion as true denoising, and Depth Anything 3 offers a cleaner geometric backbone for future world-models.

• AIGC Projects – Depth Anything 3 lands in ComfyUI, AgentEvolver lets LLM agents train themselves, and a new Qwen upscaling LoRA targets real-world photography.

• AIGC News – Gemini 3 Pro’s powerful model card leaks, xAI rolls out Grok 4.1, and Replicate joins Cloudflare right as a major outage hits the platform.

Today’s AIGC Papers

• π*0.6 shows how a VLA can truly learn from experience, turning home robots into durable, all-day performers through demonstrations, corrections, and self-practice.

• Back to Basics pulls diffusion models back to real denoising, using a pure image-space Transformer to achieve high-fidelity generation with a simpler formulation.

• Depth Anything 3 reconstructs consistent 3D geometry from any view using a single Transformer, giving future world-models a more stable and cleaner geometric backbone.

1. π*0.6: a VLA that Learns from Experience( blog | paper )

   Physical Intelligence presents π*0.6, a VLA combined with advantage-conditioned RL, trained through demonstrations, corrections, and self-experience. The system doubles success rates and throughput on real household tasks—making coffee, folding clothes, packing objects—and supports long, continuous, real-world operation, and the model card for π0.6 is also provided in the paper.

2. Kaiming He just Introduced JiT! Back to Basics: Let Denoising Generative Models Denoise（ paper | code)

   MIT’s Li Tianhong and Kaiming He propose predicting the clean image instead of noise, using a large patch-based ViT (“Just-image Transformers”) in pixel space. The method needs no tokenizer or pretraining yet achieves competitive high-res ImageNet generation, offering a simpler and clearer theoretical view of diffusion.

3. Depth Anything 3: Recovering the Visual Space from Any Views ( webpage | paper | code )

   ByteDance introduces Depth Anything 3, using a single Transformer and depth-ray representation to recover consistent geometry from single images, multi-view inputs, or videos. It surpasses VGGT in camera-pose and geometry accuracy, and outperforms DA2 in monocular depth, enabling high-fidelity 3DGS reconstruction.

Today’s AIGC Projects

• ComfyUI-DepthAnythingV3 turns Depth Anything 3 into drop-in ComfyUI nodes, making DA3 usable in everyday image and video workflows.

• AgentEvolver provides a self-evolving training loop where LLM agents generate their own tasks, rewards, and improvements.

• Qwen-Edit-2509-Upscale-LoRA gives Qwen-Image-Edit a practical, detail-preserving photography upscaler for real-world enhancement.

1. ComfyUI-DepthAnythingV3 ( link )

   ComfyUI-DepthAnythingV3 wraps Depth Anything 3 into ready-made ComfyUI nodes, letting users run DA3 on images and, with custom graphs, on multi-view / video inputs. It exposes DA3’s spatially consistent depth prediction in a visual workflow, so you can feed depth maps directly into ControlNet, 3DGS, or other geometry-aware pipelines.

2. AgentEvolver: Towards Efficient Self-Evolving Agent System ( link )

   AgentEvolver is an end-to-end framework where LLM-based agents teach themselves via three loops: self-questioning (generate new tasks), self-navigating (reuse past trajectories with hybrid policies), and self-attributing (fine-grained credit assignment over states/actions. It cuts dataset engineering cost, improves exploration efficiency, and yields faster capability gains than traditional RL-style agent training.

3. vafipas663/Qwen-Edit-2509-Upscale-LoRA ( link )

   Qwen-Edit-2509-Upscale-LoRA is a LoRA adapter for Qwen-Image-Edit-2509 focused on realistic photography. Trained on UltraHR-100K and Unsplash-lite, it repairs extreme low resolution, oversharpening, strong JPEG artifacts, motion blur, pixelation, and heavy noise—often up to 16×—while preserving composition and structure, making it a practical replacement for many “magic” commercial upscalers.

Today’s AIGC News

• Gemini 3 Pro model card leak – A leaked model card outlines Google’s next-gen sparse MoE multimodal model with 1M context, 64K outputs, RL training, detailed agent benchmarks, safety evals, and a Jan 2025 knowledge cutoff.

• Grok 4.1 announced – xAI unveils Grok 4.1, a new iteration of the Grok family with upgraded capabilities and overall performance.

• Replicate × Cloudflare, amid outage – Replicate is joining Cloudflare just as Cloudflare suffers a major service outage impacting large parts of its platform.

1. The model card for Gemini 3 Pro has reportedly leaked, revealing a model with extremely strong capabilities

   Leaked Gemini 3 Pro model card describing Google’s next-generation sparse MoE multimodal model with 1M-token context, 64K outputs, trained on large-scale web, code and media with RL. It details agentic performance, deployment channels, benchmarks, safety evaluations, frontier safety status, remaining risks, and a January 2025 knowledge cutoff.

2. xAI has announced Grok 4.1, a new iteration of its Grok model family with upgraded capabilities and performance. ( link )

3. Replicate is joining Cloudflare, at the same time that Cloudflare has been experiencing a significant service outage affecting its platform. ( link )

Subscribe now

Always fresh, always live

Real-time AIGC tracker

New models, papers, and projects as they drop — stay ahead of the AI curve.

For deeper insights and long-form analysis, subscribe to our weekly briefings at newsletter.aigc.news.

🚀 See today’s AIGC highlights

That’s it for today.

Keep building, keep thinking for yourself — we’ll be here tracking the next wave.

The aigc.news Team

My last update here was several months ago. During this break, I’ve been focused on one simple question:

In this era of exploding AIGC information, what can I offer that is genuinely useful?

I’ve realized that people don’t need more information. What’s missing are two key things:

1. Someone to filter the noise for you.

2. A way to quickly know “the few things worth seeing today.”

So starting today, I’m tackling this with a much simpler approach.

1. aigc.news: 3 News + 3 Papers + 3 Projects, Every Day

Over the past few months, I’ve been quietly building aigc.news.

It will now be updated daily and is completely free. Each day, you’ll find just three essential sections:

• 3 AIGC News: The developments that actually matter, with one-click access to the original sources.

• 3 AIGC Papers: A single sentence explaining why each one is worth your attention.

• 3 AIGC Projects : Who they’re for and what they do, at a glance.

I’m also building two small extensions:

• A “3x3 daily signal card” that’s easy to save and share.

• A subdomain: live.aigc.news, acting as a simple live radar for paper and project updates.

Think of aigc.news as your: “1-minute scan of the 9 must-see signals in the AIGC world today.”

2. newsletter.aigc.news: Weekly Digests + Occasional Deep Dives

This newsletter will now focus on two things:

• Weekly Digest From the week’s daily signals, I’ll pull out the ~10 most worth revisiting and add my perspective: what’s just hype, and what might shape the coming months.

• Occasional In-Depth Essays When a key question surfaces repeatedly, I’ll explore it in a long-form article you can reference later. For example:

  • What is the true division of labor between open-source and closed-source?

  • How do multimodal models, embodied intelligence, and agents fit together?

  • What opportunities remain for indie developers and small teams in this wave?

In short: aigc.news gives you the daily signals. newsletter.aigc.news helps you see the broader direction.

3. Subscriptions & Pricing (For Now)

It’s very simple:

• aigc.news: Free daily updates.

• newsletter.aigc.news: The weekly digest and most long-form pieces will also be free.

Your subscription status won’t change, and you won’t lose access to content.

My priority right now is to get this system running smoothly and prove its usefulness to you. Everything else can wait.

The AIGC space is noisy enough. I have no intention of adding to it. I just want to do one thing well:'

With a steady, sustainable rhythm, help you notice the few signals that truly matter.

Welcome back. I hope that starting today, you’ll make aigc.news your quick morning radar.

— pxiaoer

Subscribe now

Top Papers of the week

1.) Tracing the thoughts of a large language model ( blog | paper1 | paper2 )

Anthropic's research on the Claude language model reveals key insights:

• Multilingual Ability: Claude uses a shared conceptual space across languages, enabling knowledge transfer and suggesting a universal "language of thought."

• Poetry: It plans ahead for rhyming, showing foresight and flexibility.

• Mental Math: Claude combines approximation and precise calculation to solve problems, reflecting complex internal strategies.

• Reasoning: It performs multi-step reasoning by integrating facts, demonstrating adaptability.

• Hallucinations: Claude avoids guessing to reduce hallucinations but can still falter in some cases.

• Jailbreaks: Specific prompts can bypass safety mechanisms, exploiting coherence-safety conflicts.

2.) Synthetic Video Enhances Physical Fidelity in Video Synthesis ( webpage | paper )

We explore enhancing video generation models using physics-consistent synthetic videos from computer graphics. These videos maintain 3D consistency and improve model fidelity by reducing artifacts. Our method curates synthetic data and transfers its realism, boosting physical consistency across tasks. While not fully understanding physics, this work shows synthetic videos can enhance physical fidelity in video synthesis.

3.) Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback ( paper )

RLHF is essential for aligning large language models with human preferences, but prompt-data construction has been neglected. This paper explores data-driven bottlenecks in RLHF performance scaling, focusing on reward hacking and reduced response diversity. We propose a hybrid reward system combining RTV and GenRM to counter reward hacking and introduce Pre-PPO to maintain response diversity and boost learning efficiency. Prioritizing math and coding tasks early in training also significantly improves performance. Experiments on two model sizes show that RTV is most resistant to reward hacking, followed by GenRM with ground truth and then GenRM with SFT Best-of-N responses. Our methods capture task-specific nuances quickly, enhancing overall RLHF performance. This work highlights the importance of careful data construction and provides practical solutions to overcome performance barriers in RLHF.

4.) Gemini Robotics: Bringing AI into the Physical World ( paper )

Recent advancements in multimodal models have shown promise in digital domains, but translating these capabilities to physical robots remains challenging. This report introduces Gemini Robotics, a new family of AI models built on Gemini 2.0, specifically designed for robotics. Gemini Robotics is a Vision-Language-Action (VLA) model that can directly control robots, performing complex manipulation tasks with smooth, reactive movements. It is robust to variations in objects and environments and can follow diverse instructions. With fine-tuning, it can tackle long-horizon tasks, learn new tasks from few demonstrations, and adapt to novel robot embodiments. This is enabled by Gemini Robotics-ER, an extended model that enhances spatial and temporal reasoning for robotics tasks such as object detection, trajectory prediction, and 3D bounding box predictions. The Gemini Robotics family represents a significant step towards general-purpose robots, addressing safety considerations and unlocking AI's potential in the physical world.

5.) Qwen2.5-Omni Technical Report( paper )

We present Qwen2.5-Omni, an end-to-end multimodal model handling text, images, audio, and video inputs while generating text and speech responses in a streaming manner. It uses block-wise processing for audio and visual inputs, synchronized via TMRoPE. The Thinker-Talker architecture separates text (Thinker) and speech (Talker) generation to prevent interference, with Talker using sliding-window DiT for low-latency audio decoding. Qwen2.5-Omni outperforms Qwen2-Audio, matches Qwen2.5-VL, and sets new benchmarks on Omni-Bench, excelling in speech instruction following, robustness, and naturalness.

6.) Scaling Laws of Synthetic Data for Language Models ( paper )

Large language models (LLMs) rely on high-quality web data, but this resource is depleting. Synthetic data offers a solution, though its scalability remains uncertain. We propose SynthLLM, a framework that creates high-quality synthetic datasets by recombining concepts from pre-training corpora. Key findings include: (1) SynthLLM follows scaling laws reliably, (2) performance plateaus at 300B tokens, and (3) larger models need fewer tokens to optimize. SynthLLM outperforms existing methods, proving synthetic data as a scalable alternative for advancing LLM performance.

7.) GAIA-2: Pushing the Boundaries of Video Generative Models for Safer Assisted and Automated Driving( blog | paper )

Generative models enable environment simulation but lack key features for autonomous driving, like multi-agent interactions and multi-camera consistency. We present GAIA-2, a latent diffusion model that generates controllable, high-resolution, spatiotemporally consistent videos across diverse driving environments. GAIA-2 integrates structured inputs and latent embeddings to simulate complex, scalable driving scenarios, advancing autonomous system development.

8.) ChatAnyone: Stylized Real-time Portrait Video Generation with Hierarchical Motion Diffusion Model( webpage | paper )

Real-time interactive video-chat portraits are advancing but struggle with synchronized body motions and fine control over expressions. We propose a framework for stylized video generation, extending from talking heads to upper-body interaction. Using hierarchical motion diffusion and explicit hand control, our method generates expressive, synchronized videos at 512×768 resolution, 30fps, enabling real-time, natural video chats with rich gestures and realism.

9.) What, How, Where, and How Well? A Survey on Test-Time Scaling in Large Language Models ( paper )

As pretraining-era scaling wanes, test-time scaling (TTS) has gained focus, enhancing LLMs' problem-solving in tasks like math, coding, and open-ended Q&A. This survey introduces a unified framework across four TTS dimensions: what, how, where, and how well to scale. We review methods, applications, and challenges, offering deployment guidelines and future directions for further scaling and broader generalization.

10.) Large Language Model Agent: A Survey on Methodology, Applications and Challenges( paper | repo )

The rise of intelligent agents, powered by large language models (LLMs), marks a step toward artificial general intelligence. This survey presents a taxonomy of LLM agents, exploring their architecture, collaboration, evolution, and emergent behaviors in complex environments. It unifies fragmented research, examines evaluation methods, tools, challenges, and applications, and highlights future research directions.

AIGC News of the week

1.) deepseek-ai/DeepSeek-V3-0324 ( huggingface )

2.) bytedance’s MegaTTS 3 ( repo )

3.) OpenAI Agents SDK support MCP ( link )

4.) Gemini 2.5: Google’s most intelligent AI model ( link )

5.) VGGT: Visual Geometry Grounded Transformer ( repo )

more AI News: live.aigc.news

Inference System Design Principles

DeepSeek first introduced the design principles of the inference system, with optimization goals focused on: greater throughput and lower latency.

The solution currently adopted by DeepSeek: Large-Scale Node Expert Parallelism (EP)

• EP significantly increases batch size, thereby improving the efficiency of GPU matrix multiplication and boosting throughput.

• EP distributes experts across different GPUs, with each GPU only needing to compute a small number of experts (thus reducing memory access demands), lowering latency.

Introduced Complexity:

• EP introduces cross-node transmission. To optimize throughput, a suitable computation workflow must be designed to allow transmission and computation to occur synchronously.

• EP involves multiple nodes, naturally requiring Data Parallelism (DP), with load balancing needed between different DP instances.

Large-Scale Cross-Node Expert Parallelism

DeepSeek employs a multi-machine, multi-card expert parallelism strategy:

• Prefill: Router Expert EP32, MLA, and Shared Expert DP32. One deployment unit consists of 4 nodes, 32 redundant router experts, with 9 router experts and 1 shared expert per card.

• Decode: Router Expert EP144, MLA, and Shared Expert DP144. One deployment unit consists of 18 nodes, 32 redundant router experts, with 2 router experts and 1 shared expert per card.

The multi-machine, multi-card expert parallelism introduces significant communication overhead, so dual-batch overlapping is used to mask communication costs and improve overall throughput.

• In the prefill phase, computation and communication of two batches are interleaved—one batch’s computation can mask the communication overhead of the other.

• In the decode phase, execution times vary across stages, so the attention part is split into two stages, creating a 5-stage pipeline to overlap computation and communication.

Due to the use of large-scale parallelism (including expert parallelism and data parallelism), some GPUs may become overloaded, necessitating computational and communication load balancing.

• Prefill Load Balancer

  • Core Issue: Variations in the number and length of requests across different Data Parallelism (DP) instances lead to differences in core-attention computation and dispatch transmission volumes.

  • Optimization Goal: Ensure roughly equal computation load across GPUs (core-attention load balancing) and similar token input volumes (dispatch transmission load balancing) to avoid prolonged processing times on some GPUs.

• Decode Load Balancer

  • Core Issue: Variations in the number and length of requests across DP instances result in differences in core-attention computation (related to KVCache usage) and dispatch transmission volumes.

  • Optimization Goal: Ensure roughly equal KVCache usage across GPUs (core-attention load balancing) and similar request volumes (dispatch transmission load balancing).

• Expert-Parallel Load Balancer

  • Core Issue: For a given MoE model, certain naturally high-load experts exist, leading to uneven expert computation loads across GPUs.

  • Optimization Goal: Balance expert computation across GPUs (i.e., minimize the maximum dispatch reception volume across all GPUs).

Real-World Statistics of the Online Inference System

All DeepSeek R1/V3 services use H800 GPUs. Matrix computations and dispatch transmissions use FP8 format consistent with training, while core-attention computations and combine transmissions use BF16 consistent with training, maximizing service performance.

Since service loads are high during the day and low at night, all servers handle inference during peak load times, while some machines are freed up for research and training during low-load periods.

Over the past 24 hours (Beijing time, 2025/02/27 12:00 to 2025/02/28 12:00), the total node usage for DeepSeek V3 and R1 inference services peaked at 278 nodes and averaged 226.75 nodes (each node with 8 H800 GPUs). Assuming a GPU rental cost of $2/hour, the total daily cost is $87,072.

Within this 24-hour statistical period, DeepSeek V3 and R1 recorded:

• Total input tokens: 608 billion, of which 342 billion tokens (56.3%) hit the KVCache disk cache.

• Total output tokens: 168 billion. The average output rate was 20–22 tokens per second (tps), with an average KVCache length of 4,989 per output token.

• Average throughput per H800 GPU:

  • For prefill tasks: ~73.7k tokens/s input throughput (including cache hits).

  • For decode tasks: ~14.8k tokens/s output throughput.

These statistics encompass all loads from web, app, and API usage. If all tokens were priced according to DeepSeek R1’s rates, the theoretical daily revenue would be $562,027, yielding a cost-profit margin of 545%.

In reality, DeepSeek doesn’t generate this much revenue because V3’s pricing is lower, paid services only account for a portion of usage, and discounts are offered at night.

What Can We Infer?

Rumors previously suggested that DeepSeek’s official deployment consisted of a 320-H800 inference cluster. Now, it appears to be 278 nodes—2,224 H800s. Officially, DeepSeek acknowledges owning at least 10,000 H800s, meaning the GPUs used for inference are relatively few.

• Cost: Average of 226.75 nodes (1,814 GPUs), at $2/hour per GPU, yields a daily cost of $87,072.

• Revenue: Input: 608B tokens; Output: 168B tokens, resulting in a daily revenue of $562,027.

• Gross Daily Profit: $474,955 = ~3,457,672.4 RMB/day.

However, the above calculation has flaws. Even if the 6x profit margin is halved to 3x, the profit margin remains very high. Many domestic vendors deploying DeepSeek have shut down API services due to losses, raising questions about where the problem lies.

In an interview, Liang Wenfeng said: “We just do things at our own pace, then calculate costs and set prices. Our principle is not to lose money.”

This suggests DeepSeek is currently profitable, with earnings likely reinvested into R&D. We look forward to R2’s release soon.

Subscribe now

According to the official introduction, the Fire-Flyer File System (3FS) is a parallel file system designed to fully utilize the bandwidth of modern SSDs and RDMA networks.

• Achieves an aggregate read throughput of 6.6 TiB/s in a 180-node cluster.

• Delivers 3.66 TiB/minute throughput in the GraySort benchmark on a 25-node cluster.

• Provides peak throughput of over 40 GiB/s per client node in KVCache lookups.

• Features a distributed architecture with strong consistency semantics.

• Used for training data preprocessing, dataset loading, checkpoint saving/reloading, embedding vector search, and KVCache lookups in inference for V3/R1.

Smallpond, on the other hand, is a data processing framework built on top of 3FS.

Fire-Flyer File System (3FS)

3FS is part of the Fire-Flyer AI-HPC developed by DeepSeek. It is detailed in the paper Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning.

Fire-Flyer AI-HPC consists of three components: the HAI Platform (open-sourced two years ago), 3FS (open-sourced today), and HaiScale (yet to be open-sourced).

In summary, 3FS has several key features:

1. High-Performance Design: 3FS is tailored to leverage the high IOPS (input/output operations per second) and throughput of NVMe SSDs, as well as RDMA networks. This design enables it to efficiently handle large-scale data requests, meeting the demands of deep learning and large-scale computing.

2. System Architecture: The 3FS system comprises four roles: cluster manager, metadata service, storage service, and client. The metadata and storage services periodically send heartbeat signals to the cluster manager to ensure system stability and efficiency. Multiple cluster managers ensure high availability.

3. Request Control Mechanism: 3FS implements a request transmission control mechanism to alleviate network congestion. Upon receiving a read request, the storage service asks the client for permission to transfer data. This limits the number of concurrent senders, maintaining good performance under high load.

4. Strong Consistency with Chain Replication: 3FS adopts the Chain Replication and Allocate Query (CRAQ) approach to provide strong consistency. File contents are split into blocks and replicated across a series of storage targets, fully unleashing the throughput and IOPS of all SSDs.

5. High Throughput: By optimizing batch write and read operations, 3FS achieves write speeds exceeding 10 GiB/s per node, accelerating checkpoint saving and loading, and reducing latency during training.

6. 3FS-KV System: 3FS also supports 3FS-KV, a shared-storage distributed data processing system built on 3FS. It supports key-value storage, message queues, and object storage models, further enhancing system flexibility and performance.

3FS provides robust storage support for deep learning and large-scale computing, effectively meeting demands for high throughput and low latency.

Description from the Paper:

Performance

Peak Throughput

Read throughput test results for a 3FS cluster: The cluster consists of 180 storage nodes, each equipped with 2×200Gbps InfiniBand NICs and 16×14TiB NVMe SSDs. Over 500+ client nodes, each with a 1×200Gbps InfiniBand NIC, were used for the read stress test. Under background traffic from training jobs, the aggregate read throughput reached approximately 6.6 TiB/s.

Sorting Performance

The test cluster consists of 25 storage nodes (2 NUMA domains per node, 1 storage service per NUMA, 2×400Gbps NICs per node) and 50 compute nodes (2 NUMA domains, 192 physical cores, 2.2 TiB RAM, and 1×200Gbps NIC per node). Sorting 110.5 TiB of data across 8192 partitions took 30 minutes and 14 seconds, achieving an average throughput of 3.66 TiB/minute.

KVCache

KVCache is designed to optimize the LLM inference process by caching keys and value vectors from previous tokens in the decoder layers, avoiding redundant computation. The figure above shows the read throughput for all KVCache clients, highlighting peak and average values, with a peak throughput of up to 40 GiB/s. The figure below shows the IOPS of delete operations during garbage collection (GC) over the same period.

Why is a Specialized File System Like 3FS Needed?

In LLM scenarios, there’s a need for highly concurrent, high-throughput, and scalable distributed file systems that also demand strong consistency, intelligent routing, and cache management. Systems like 3FS, which offer high-performance solutions tailored for RDMA and SSDs, flexible metadata design, and asynchronous zero-copy I/O, become highly valuable.

Code Analysis

One notable aspect of 3FS’s implementation is that DeepSeek used Rust to develop the chunk_engine.

The chunk_engine is a core module at the bottom layer of the 3FS storage service, responsible for managing, allocating, and reclaiming physical disk blocks. The upper layers can read and write block data through this engine. It primarily uses cxx to automatically generate C++ bindings, allowing C++ code to directly call Rust code.

In recent years, Rust has gained popularity in the MLSys (Machine Learning Systems) field. For example, Hugging Face’s tokenizers are also implemented in Rust. The DeepSeek team likely chose Rust for the chunk_engine due to its maintainability, memory safety, and excellent performance.

The DeepSeek team may also have used the Rust framework Tokio in backend services, as I found several Rust open-source projects, including Tokio, in Quant AI’s open-source initiatives. I sincerely hope more teams adopt Rust for developing machine learning systems.

Smallpond

Smallpond is a lightweight data processing framework built on top of DuckDB and 3FS. It supports lightweight, high-performance data processing and scales to petabyte-scale datasets.

Installation and usage are straightforward, with a minimal API offering two types: one for dynamically building dataflow graphs and another for static construction.

Installation:

pip install smallpond

Usage Example:

# Download example data
wget https://duckdb.org/data/prices.parquet

import smallpond
# Initialize session
sp = smallpond.init()
# Load data
df = sp.read_parquet("prices.parquet")
# Process data
df = df.repartition(3, hash_by="ticker")
df = sp.partial_sql("SELECT ticker, min(price), max(price) FROM {0} GROUP BY ticker", df)
# Save results
df.write_parquet("output/")
# Show results
print(df.to_pandas())

For performance, refer to the sorting performance of 3FS.

Conclusion of DeepSeek Open Source Week

DeepSeek Open Source Week concludes today. Thank you, DeepSeek, for sharing valuable resources for everyone to learn and use.

Many teams have already taken action and achieved tangible performance improvements. For instance, the vLLM team replaced TRITON_MLA with FLASHMLA, boosting throughput by 2-16%, delivering real results.

These projects open-sourced by DeepSeek will continue to influence us. Our journey of intense learning continues.

more

• day0: https://aigc.openbot.ai/p/deepseek-opensourceweek-is-coming

• day1: https://aigc.openbot.ai/p/deepseek-open-source-week-day-1-in

• day2:https://aigc.openbot.ai/p/day-2-of-deepseek-opensourceweek

• day3:https://aigc.openbot.ai/p/deepseek-opensourceweek-day-3-deepgemm

• day4:https://aigc.openbot.ai/p/deepseek-opensourceweek-day-4-in

Subscribe now

DualPipe is a bidirectional pipeline parallelism algorithm designed for computation-communication overlap in V3/R1 training. Meanwhile, EPLB serves as an expert-parallel load balancer for V3/R1.

The final project, profile-data, primarily releases analytical data from DeepSeek’s infrastructure for training and inference. So far, it includes data on Prefilling for both training and inference, while the Decoding analysis data for inference has yet to be made public.

Today, we’ll dive into an analysis of the DualPipe and EPLB projects, both of which lean toward engineering optimization.

DualPipe

DualPipe is mentioned in the DeepSeek V3 paper as a bidirectional pipeline parallelism communication algorithm, mainly used to optimize data interaction and training efficiency in large-scale models.

Key Features:

• Computation-Communication Overlap
  DualPipe’s design aims to maximize cluster computing performance by achieving full overlap of computation and communication during forward and backward passes, reducing idle wait times typical in traditional pipeline parallelism. This is especially critical for expert parallelism (Expert Parallelism) across nodes in MoE models.

• Bidirectional Scheduling
  DualPipe employs a bidirectional scheduling strategy, feeding data from both ends of the pipeline simultaneously to reuse hardware resources efficiently. It also incorporates a sophisticated yet highly effective 8-step scheduling strategy.

• Memory Optimization
  DualPipe deploys the shallowest layers (including the embedding layer) and the deepest layers (including the output layer) on the same pipeline level (PP Rank), enabling physical sharing of parameters and gradients to further enhance memory efficiency.

Pipeline Bubble and Memory Usage Comparison (Pipeline bubble refers to idle wait time)

For those interested, you can check out the code—it’s not very long and is great for learning.

EPLB

EPLB (Expert Parallelism Load Balancer) is primarily designed to optimize the distributed deployment of MoE models. It ensures load balancing among different experts in the MoE portion by replicating shared experts and fine-grained high-load experts across multiple GPUs in the cluster. This allows GPUs to handle more "hot data" (data sent to shared experts) efficiently.

EPLB isn’t detailed in DeepSeek’s paper, and its code is remarkably concise at just 160 lines.

Key Features:

• Load Balancing Optimization
  It replicates high-load experts (a strategy we can call "redundant expert strategy") and uses heuristic adjustments for expert allocation to ensure balanced workloads across GPUs.

• Hierarchical Load Balancing
  EPLB adopts a three-tier structure: node-level → intra-node expert replication → GPU allocation. It prioritizes assigning experts from the same group to the same node to minimize cross-node data transfers, then ensures load balancing at each layer. This approach, combined with DeepSeek V3’s Group-Limited Expert Routing strategy, significantly boosts distributed training efficiency.

• Dynamic Scheduling Strategy
  EPLB dynamically selects load balancing strategies based on the situation—using a hierarchical strategy during the prefilling phase and a global strategy during the decoding phase.

Let’s Look at the Code:

Redundant Expert Strategy

def replicate_experts(weight: torch.Tensor, num_phy: int):
    # Replicate high-load experts
    for i in range(num_log, num_phy):
        redundant_indices = (weight / logcnt).max(dim=-1).indices
        phy2log[:, i] = redundant_indices
        logcnt[arangen, redundant_indices] += 1

Hierarchical Load Balancing

def rebalance_experts_hierarchical():
    # Step 1: Pack expert groups to nodes
    tokens_per_group = weight.unflatten(-1, (num_groups, group_size)).sum(-1)
    group_pack_index, group_rank_in_pack = balanced_packing(tokens_per_group, num_nodes)

    # Step 2: Build redundant experts within nodes
    tokens_per_mlog = weight.gather(-1, mlog2log).view(-1, num_logical_experts // num_nodes)

    # Step 3: Pack physical experts to GPUs
    tokens_per_phy = (tokens_per_mlog / mlogcnt).gather(-1, phy2mlog)

Dynamic Scheduling Strategy

def rebalance_experts():
    if num_groups % num_nodes == 0:
        # Use hierarchical strategy
        phy2log, phyrank, logcnt = rebalance_experts_hierarchical()
    else:
        # Use global strategy
        phy2log, phyrank, logcnt = replicate_experts()

Interested readers can explore the full code.

Tomorrow is the final day of DeepSeek Open Source Week—will they drop a heavyweight open-source project? Let’s wait and see!

Subscribe now

Today marks the third day of DeepSeek's Open Source Week, with the release of DeepGEMM right on schedule at 9 AM.

As of now, the project has garnered 3.3k stars since its release.

The official introduction describes DeepGEMM as an FP8-supporting GEMM library compatible with both dense and MoE (Mixture of Experts) GEMM operations, designed for training and inference of V3/R1 models.

A Brief Introduction to GEMM

General Matrix Multiplication (GEMM) is one of the most fundamental and critical operations in deep learning and scientific computing. GEMM refers to the multiplication of two matrices, A and B, to produce a result matrix C, typically expressed as C = A × B.

In deep learning, GEMM underpins core components such as fully connected layers, convolutional layers, and attention mechanisms. For instance, in Transformer architectures, both self-attention and feedforward network layers heavily rely on matrix multiplication. As model sizes grow, GEMM operations dominate the computational time in training and inference, making their performance a key factor in the efficiency of deep learning systems.

Modern GPU architectures, like NVIDIA’s Tensor Core technology, are specifically designed to accelerate matrix multiplication. With the ever-increasing scale of models, the demand for high-performance GEMM implementations continues to rise, especially in large language models (LLMs) and MoE frameworks, where efficient GEMM is critical for real-time inference and cost-effective training.

In the paper DeepSeek LLM: Scaling Open-Source Language Models with Longtermism, DeepSeek mentions GEMM, though it ties into their work in another paper, Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning, which introduces the HAI-LLM training system. For those interested, I recommend checking out the Fire-Flyer AI-HPC paper.

Today’s open-sourced DeepGEMM supports FP8 and is tailored for training and inference of DeepSeek’s V3/R1 models. In the V3 paper, DeepSeek details several optimizations for FP8 training.

The main challenges of FP8 training lie in precision and error handling. To tackle these, DeepSeek implemented the following optimizations:

• Fine-Grained Quantization: Data is split into smaller groups, each with a specific multiplier to maintain high precision.

• Online Quantization: Weights are computed online for each 1x128 activation block or 128x128 weight block, with scaling factors inferred on-the-fly, and activations converted to FP8 in real time.

• Improved Accumulation Precision: FP8 accumulation can introduce random errors, so intermediate results are stored in FP32, then converted back after accumulation.

• Low-Precision/Mixed-Precision Storage and Communication: For MoE model training, FP8 is mixed with BF16/FP32 to ensure dynamic model stability.

For a detailed look at these optimizations, check out the DeepSeek V3 paper.

DeepGEMM Overview

Here’s a summary of its key features:

• FP8 Support: DeepGEMM uses CUDA’s two-stage accumulation to address precision issues.

• Grouped GEMM Support: It improves on CUTLASS’s grouped GEMM, with targeted optimizations for MoE models.

• Just-In-Time Compilation: Through JIT technology, code is dynamically generated and optimized at runtime, boosting performance and flexibility.

• FFMA SASS Interleaving: DeepSeek analyzed SASS compilation results in depth, tweaking FFMA/FADD instructions to enhance fine-grained FP8 GEMM efficiency.

Performance

All metrics show improvement, with the highest gain reaching a 2.7x speedup. DeepSeek notes that performance isn’t optimal in some areas and welcomes PRs from those interested in further optimization.

In the DeepGEMM project’s README, the DeepSeek team provides a detailed breakdown of the optimizations. For those interested, it’s worth diving into the code alongside the documentation for a hands-on exploration.

Spotlight: The interleave_ffma.py File

Today, let’s focus on a specific file in the project: interleave_ffma.py under the jit directory. It contains some clever tricks worth exploring.

Here’s the code:

import argparse
import mmap
import os
import re
import subprocess
from torch.utils.cpp_extension import CUDA_HOME

def run_cuobjdump(file_path):
    command = [f'{CUDA_HOME}/bin/cuobjdump', '-sass', file_path]
    result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
    assert result.returncode == 0
    return result.stdout

def extract_ffma(sass):
    lines = sass.splitlines()
    collected = []
    current = []
    arch_name, func_name = 'N/A', 'N/A'
    skip_next_line = False
    for line in lines:
        if 'code for' in line:
            arch_name = line.lstrip().lstrip('code for ').rstrip()
        elif 'Function :' in line:
            func_name = line.lstrip().lstrip('Function :').rstrip()
        elif 'FFMA' in line:
            current.append(line)
            skip_next_line = True
        elif skip_next_line:
            current.append(line)
            skip_next_line = False
        else:
            if len(current) >= 16:
                assert len(current) % 2 == 0
                collected.append((f'{arch_name}::{func_name}', current))
            current = []
    if os.getenv('DG_PRINT_REG_REUSE', None):
        print(f"Found {len(collected)} FFMA segments")
    return collected

def extract_hex_from_line(line):
    match = re.search(r'/\*\s*(0x[0-9a-fA-F]+)\s*\*/', line)
    assert match
    return int(match.group(1), 16)

def validate(m, offset, le_bytes, num_lines):
    assert len(le_bytes) == num_lines // 2
    assert m[offset:offset + 16] == le_bytes[0]
    for i in range(1, num_lines // 2):
        if m[offset + i * 16:offset + i * 16 + 16] != le_bytes[i]:
            return False
    return True

def parse_registers(line):
    import re
    line = re.sub(r'/\*.*?\*/', '', line)
    line = line.replace(';', '')
    tokens = line.strip().split(',')
    registers = []
    for token in tokens:
        token = token.strip()
        words = token.split()
        for word in words:
            if word.startswith('R'):
                reg = word.split('.')[0]
                registers.append(reg)
    return registers

def modify_segment(m, name, ffma_lines):
    num_lines = len(ffma_lines)
    assert num_lines % 2 == 0
    le_bytes, new_le_bytes = [], []
    reused_list = []
    dst_reg_set = set()
    last_reused, last_dst_reg = False, ''
    num_changed = 0
    for i in range(num_lines // 2):
        dst_reg = parse_registers(ffma_lines[i * 2])[-2]
        low_line, high_line = ffma_lines[i * 2], ffma_lines[i * 2 + 1]
        low_hex, high_hex = extract_hex_from_line(low_line), extract_hex_from_line(high_line)
        le_bytes.append(low_hex.to_bytes(8, 'little') + high_hex.to_bytes(8, 'little'))
        reused = (high_hex & 0x0800000000000000) != 0
        if reused:
            is_first_occurred = dst_reg not in dst_reg_set
            if is_first_occurred or (last_reused and dst_reg == last_dst_reg):
                assert high_hex & 0x0800200000000000, f"{hex(high_hex)}"
                high_hex ^= 0x0800200000000000
                reused = False
                num_changed += 1
            else:
                reused_list.append(i)
        dst_reg_set.add(dst_reg)
        new_le_bytes.append(low_hex.to_bytes(8, 'little') + high_hex.to_bytes(8, 'little'))
        last_reused, last_dst_reg = reused, dst_reg
    if os.getenv('DG_PRINT_REG_REUSE', None):
        print(f" > segment `{name}` new reused list ({num_changed} changed): {reused_list}")
    offsets = []
    offset = m.find(le_bytes[0])
    while offset != -1:
        offsets.append(offset)
        offset = m.find(le_bytes[0], offset + 1)
    offsets = list(filter(lambda x: validate(m, x, le_bytes, num_lines), offsets))
    for offset in offsets:
        for i in range(num_lines // 2):
            m[offset + i * 16:offset + i * 16 + 16] = new_le_bytes[i]

def process(path):
    if os.getenv('DG_PRINT_REG_REUSE', None):
        print(f'Processing {path}')
    output = run_cuobjdump(path)
    segments = extract_ffma(output)
    with open(path, 'r+b') as f:
        mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_WRITE)
        for segment in segments:
            modify_segment(mm, *segment)
        mm.close()

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='Interleave FFMA reg reuse')
    parser.add_argument('--so', help='Path to the SO file')
    args = parser.parse_args()
    process(args.so)

This file is designed to optimize the register reuse patterns of FFMA (Fused Floating-point Multiply-Add) instructions in CUDA-compiled assembly code by modifying the binary file, ultimately improving GPU instruction execution efficiency.

Key Function Breakdown:

SASS Code Extraction

def run_cuobjdump(file_path):
    command = [f'{CUDA_HOME}/bin/cuobjdump', '-sass', file_path]

Uses NVIDIA’s cuobjdump tool to extract SASS (assembly) code from the binary.

FFMA Instruction Analysis

def extract_ffma(sass):

Extracts segments of SASS code containing FFMA instructions, collecting architecture and function names along with the instruction sequences.

Register Usage Analysis

def parse_registers(line):

Parses the registers used in each instruction, identifying those starting with 'R'.

Binary Modification

def modify_segment(m, name, ffma_lines):

Modifies the reuse and yield bits of FFMA instructions by tweaking specific bit patterns (e.g., 0x0800200000000000) to optimize register reuse.

Workflow:

• Reads a compiled CUDA shared library (.so file).

• Extracts SASS code using cuobjdump.

• Identifies and collects all FFMA instruction sequences.

• Analyzes register usage patterns in each FFMA instruction.

• Modifies the reuse flags based on specific rules.

• Writes the optimized instructions back to the original file.

Optimization Strategy:

The tool targets:

• First-time register usage.

• Consecutive reuse of the same destination register.
  By tweaking the reuse and yield bits, it optimizes instruction scheduling.

Usage:

python interleave_ffma.py --so path/to/cuda_lib.so

Analysis:

This tool acts as a post-processing optimizer, running after CUDA compilation to enhance GPU instruction efficiency by tweaking the binary. Its focus on FFMA instruction register reuse is particularly impactful for compute-intensive applications like deep learning.

A regex in the file often stumps readers:

def extract_hex_from_line(line):
    match = re.search(r'/\*\s*(0x[0-9a-fA-F]+)\s*\*/', line)
    assert match
    return int(match.group(1), 16)

In CUDA SASS assembly, instructions often appear like this:

FFMA R8, R8, R6, R4;                  /* 0x5c98078000870808 */

This regex extracts 0x5c98078000870808, the hexadecimal machine instruction encoding. The function:

• Extracts the hex code from the assembly line.

• Converts it to an integer for subsequent modification.

• This step is crucial for locating instructions, modifying specific bits (e.g., reuse and yield flags), and writing them back.

Honestly, DeepSeek’s engineers seem to outshine even some NVIDIA folks when it comes to CUDA mastery! Oh, and they’ve slashed their API prices again.

Subscribe now

On the second day of OpenSourceWeek, the official DeepSeek X account posted an article at 10:24, introducing the second open-source project of Open Source Week: DeepEP.

Since the code was released, it has already garnered 4.3K stars.

Many people are not very familiar with MoE models, so this article will first briefly introduce MoE and some of DeepSeek's work on MoE.

MoE Introduction

The Mixture-of-Experts (MoE) model is a simple extension of the Transformer architecture, rapidly becoming the preferred architecture for medium-to-large-scale language models (2 billion to 600 billion parameters).

Key Advantages:

• Faster pre-training speed compared to dense models

• Faster inference speed compared to models with the same number of parameters

Challenges:
It requires significant memory since all expert systems need to be loaded into memory. Additionally, as it is typically used for medium-to-large models, it often requires parallel processing across multiple GPUs, and communication must be highly efficient.

MoE models primarily consist of two key components:

1. Sparse MoE Layers
   These layers replace the feed-forward network (FFN) layers in traditional Transformer models. An MoE layer contains several "experts" (e.g., 8), each of which is an independent neural network.

2. Gating Network or Router
   This component determines which tokens are sent to which experts. Sometimes, a token may even be routed to multiple experts.

Summary:
A notable advantage of Mixture-of-Experts (MoE) models is their ability to perform effective pre-training with far fewer computational resources than dense models. This means that, under the same computational budget, you can significantly scale up the model or dataset size. Especially during pre-training, MoE models typically reach the same quality level faster than dense models.

DeepSeek MoE

On January 11, 2024, DeepSeek released the paper DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models, making it one of the earliest companies to research MoE models.

In the DeepSeekMoE paper, the concepts of Router Expert and Shared Expert were introduced, and experiments were conducted on increasing fine-grained experts.

In the paper, DeepSeekMoE 16B has 2 shared experts and 64 router experts per layer, with each token activating 2 shared experts and 6 router experts. The 145B version, on the other hand, has 4 shared experts and 128 router experts, with each token activating 4 shared experts and 12 router experts.

The DeepSeekMoE paper also mentions today’s topic: Expert Parallelism.

In the DeepSeek V3 paper, two paragraphs provide more detailed data. DeepSeek R1 was trained based on V3, and although no further details are provided, it is largely consistent with V3.

DeepEP Introduction

DeepEP is a communication library designed for MoE models and Expert Parallelism (EP). It provides high-throughput and low-latency all-to-all GPU kernels, also known as MoE data dispatch and combine. Additionally, the library supports low-precision operations, including FP8.

To align with the group-limited gating algorithm proposed in the DeepSeek-V3 paper, DeepEP offers a set of kernels optimized for asymmetric domain bandwidth forwarding (e.g., from NVLink domain to RDMA domain). These kernels deliver high throughput, making them ideal for training and inference prefetching tasks. They also support control over the number of Streaming Multiprocessors (SM).

For latency-sensitive inference decoding tasks, DeepEP includes a set of low-latency kernels using pure RDMA to minimize latency. The library also introduces a hook-based method for overlapping communication and computation without occupying any SM resources.

Note: The implementation of this library may slightly differ from the DeepSeek-V3 paper.

Performance

NVLink and RDMA Forwarding Test for General Kernels

We tested general kernels on H800 (NVLink max bandwidth ~160 GB/s), with each device connected to a CX7 InfiniBand 400 Gb/s RDMA NIC (max bandwidth ~50 GB/s). The test followed the pre-training setup of DeepSeek-V3/R1: 4096 tokens per batch, hidden dimension of 7168, top-4 grouping, top-8 experts, FP8 dispatch, and BF16 combine.

Low-Latency Kernel Test with Pure RDMA

We tested low-latency kernels on H800, with each device connected to a CX7 InfiniBand 400 Gb/s RDMA NIC (max bandwidth ~50 GB/s). The test followed a typical DeepSeek-V3/R1 production environment setup: 128 tokens per batch, hidden dimension of 7168, top-8 experts, FP8 dispatch, and BF16 combine.

Code Analysis

The optimization strategy focuses heavily on low-latency communication. Here, we’ll highlight this aspect, while other features like dynamic scheduling, asynchronous communication, and stream management can be explored in the relevant documentation and code.

Double Buffering

auto buffer = layout.buffers[low_latency_buffer_idx];
auto next_buffer = layout.buffers[low_latency_buffer_idx ^= 1];

• Alternates between two buffers: one for the current operation, another for the next.

• Uses the bitwise operation ^= 1 to efficiently switch buffer indices.

TMA (Tensor Memory Access) Optimization

• Leverages the Hopper architecture’s TMA instructions to accelerate data transfer.

• Supports low-precision formats like FP8 to reduce communication bandwidth requirements.

IBGDA Direct Communication

// Initialize recv queues for low-latency mode AR
ibgda_initialize_recv_queue<<<num_ranks, 128>>>(rank);

• Uses NVSHMEM’s IBGDA technology for GPU-direct RDMA communication.

• Bypasses CPU involvement entirely to reduce latency.

Expert-Level QP Allocation

_buffer = Buffer(group, 0, num_rdma_bytes, low_latency_mode=True,
num_qps_per_rank=num_experts // group.size())

• Assigns independent Queue Pairs (QPs) to each local expert, eliminating resource contention.

DeepEP Use Cases

• Large-scale MoE model training (e.g., models with hundreds of billions of parameters)

• High-concurrency, low-latency real-time inference services

• Heterogeneous computing tasks such as multimodal applications and scientific computing

Notably, at the end of the repository, DeepSeek mentions using an undocumented NVIDIA instruction for optimization—a true hacker spirit worth learning from!

Subscribe now

This morning at 9:34, DeepSeek announced the first project of Open Source Week on X: FlashMLA. This article provides an in-depth analysis of FlashMLA.

The FlashMLA project has gained significant popularity, with its code already reaching 6.8k stars.

Brief Introduction to MLA

MLA (Multi-Head Latent Attention) is an optimization method for Multi-Head Attention (MHA) proposed by DeepSeek in their paper DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model.

In Transformer models, MHA is one of the most computationally intensive modules. To maintain high efficiency in large-scale scenarios, further optimization is necessary.

MLA can be considered a variant of MHA. In its implementation, it borrows some concepts from FlashAttention. The DeepSeek-V2 paper primarily compares it with MHA, GQA, and MQA, with optimization results shown in the figure below:

In some inference frameworks, MLA has also been implemented. As shown below, after integrating MLA into SGLang, throughput increased by 2-3 times.

Using FlashMLA

Environment Requirements:

• Hopper GPUs

• Minimum CUDA 12.3

• Minimum PyTorch 2.0

Installation:

git clone https://github.com/deepseek-ai/FlashMLA  
python setup.py install  

Performance:
The repository provides a Benchmark file that can be run directly. Official results show that on an H800 SXM5 with CUDA 12.6, it achieves speeds of up to 3000 GB/s under memory-bound configurations and 580 TFLOPS under compute-bound configurations.

Code Analysis

FlashMLA’s codebase is relatively small with minimal dependencies.
The primary optimization techniques are as follows:

1. Computation Chunking and Scheduling Optimization

template<int kHeadDim_, int kBlockM_, int kBlockN_, int kNWarps_>  
struct Flash_fwd_kernel_traits_mla {  
    // Fixed block size of 64x64  
    static constexpr int kBlockM = kBlockM_;    
    static constexpr int kBlockN = kBlockN_;    
    
    // Each block uses 8 warps in parallel  
    static constexpr int kNWarps = kNWarps_;    
    static constexpr int kNThreads = kNWarps * 32;  
    
    // Shared memory optimization  
    static constexpr int kBlockKSmem = kHeadDim % 64 == 0 ? 64 : 32;  
};

Key Points:

• Improves computational efficiency through chunking (block size of 64), paged KV caching, and multi-warp parallelism.

2. Memory Access Optimization

struct Flash_fwd_mla_params {  
    using index_t = int64_t;  
    int b, seqlen_q, d, d_v;  
    int h, h_h_k_ratio, ngroups;  
    bool is_causal;  
    float scale_softmax, scale_softmax_log2;  
    int *__restrict__ cu_seqlens_k;  
    void *__restrict__ q_ptr;  
    void *__restrict__ k_ptr;  
    void *__restrict__ v_ptr;  
    void *__restrict__ o_ptr;  
    void *__restrict__ softmax_lse_ptr;  
    index_t q_batch_stride;  
    index_t k_batch_stride;  
    index_t v_batch_stride;  
    index_t o_batch_stride;  
    index_t q_row_stride;  
    index_t k_row_stride;  
    index_t v_row_stride;  
    index_t o_row_stride;  
    index_t q_head_stride;  
    index_t k_head_stride;  
    index_t v_head_stride;  
    index_t o_head_stride;  
    int *__restrict__ block_table;  
    index_t block_table_batch_stride;  
    int page_block_size;  
    int *__restrict__ tile_scheduler_metadata_ptr;  
    int num_sm_parts;  
    int *__restrict__ num_splits_ptr;  
    void *__restrict__ softmax_lseaccum_ptr;  
    void *__restrict__ oaccum_ptr;  
};

Key Points:

• Uses paged KV caching (block_table, page_block_size).

• Optimized memory layout and access strides (stride).

• Scheduling with tile_scheduler_metadata.

3. Softmax Computation Optimization

for (int mi = 0; mi < size<0>(tensor); ++mi) {  
    MaxOp<float> max_op;  
    max(mi) = zero_init ? tensor(mi, 0) : max_op(max(mi), tensor(mi, 0));  
    #pragma unroll  
    for (int ni = 1; ni < size<1>(tensor); ni++) {  
        max(mi) = max_op(max(mi), tensor(mi, ni));  
    }  
    max(mi) = Allreduce<4>::run(max(mi), max_op);  
    const float max_scaled = max(mi) == -INFINITY ? 0.f : max(mi) * scale;  
    sum(mi) = 0;  
    #pragma unroll  
    for (int ni = 0; ni < size<1>(tensor); ++ni)  {  
        tensor(mi, ni) = exp2f(tensor(mi, ni) * scale - max_scaled);  
        sum(mi) += tensor(mi, ni);  
    }  
    SumOp<float> sum_op;  
    sum(mi) = Allreduce<4>::run(sum(mi), sum_op);  
}

Key Points:

• Uses log2/exp2 instead of log/exp.

• Optimizes with FFMA instructions.

• Warp-level reduction for summation optimization.

4. Double Buffering Optimization

struct SharedStorageMLA {  
    union {  
        struct {  
            // Double buffering for K matrix  
            cute::array_aligned<Element, cosize_v<SmemLayoutQ>> smem_q;  
            cute::array_aligned<Element, cosize_v<SmemLayoutK> * 2> smem_k;  // Double buffer  
            cute::array_aligned<Element, cosize_v<SmemLayoutP>> smem_p;  
            cute::array_aligned<ElementAccum, cosize_v<SmemLayoutRow>> smem_scale;  
        };  
    };  
};

Key Points:

• Hides memory latency with double buffering to improve hardware utilization.

Summary of FlashMLA

FlashMLA is essentially a customized version of FlashAttention. Its current applicable scenarios include:

• Environments requiring CUDA 11+ and SM90+ Hopper architecture.

• Inference or training of multi-head attention with BF16 (Q=576, V=512).

• Large-sequence scenarios requiring integration with split-K schemes to boost throughput.

As shown above, there are still many optimization methods from the official team. Looking forward to tomorrow’s project—could it be infra-related? Perhaps something as challenging as MTP?

Subscribe now

On February 21, DeepSeek announced on X the launch of a warm-up for Open Source Week. Starting next week, they will open-source five projects over five consecutive days. For each of these five projects, I will write detailed articles to introduce them on the day they are announced. Feel free to follow me for the latest analyses.

Today, I’ll make some predictions about which projects might be open-sourced. If I guess even one correctly, I’ll consider it a win.

First, there’s definitely going to be something related to infra.
According to the X post, they are a small team within DeepSeek, sharing small but genuine progress, specifically modules used in online services. They particularly emphasized “small,” which likely points to code related to model inference optimization.

The recent release of DeepSeek-R1 has generated significant buzz, but inference optimization still lacks robust support from major frameworks. It feels like they might directly release some official implementations, as this is currently in high demand.

These optimizations could include deployment and inference strategies mentioned in the DeepSeek V3 technical report, such as Prefilling and Decoding.

Second, the official repository index hints at a paper, which is likely related.

Today, DeepSeek created an "open-infra-index" repo on GitHub, which includes a paper: Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning.

This Fire-Flyer AI-HPC paper mainly discusses optimizations for a training cluster with 10,000 A100 GPUs, focusing on reducing system construction costs and energy consumption. Specifically, compared to DGX-A100, it achieves about 80% of the performance while cutting costs by 60% and energy consumption by 50%. I won’t go into the detailed innovations here—interested folks can read the paper themselves.

Why mention this paper? My guess is that they’ll open-source some of the system code referenced in it. Looking at it, here are a few possibilities:

• HFReduce: A library developed specifically for efficient allreduce operations, designed to optimize GPU communication in PCIe architectures. HFReduce overlaps computation and communication through asynchronous allreduce operations, significantly boosting performance.

• HaiScale: A distributed data-parallel training tool that uses HFReduce as its communication backend, enabling asynchronous allreduce operations during backpropagation to improve training efficiency.

• 3FS: A high-performance distributed file system designed to fully leverage the high IOPS and throughput of NVMe SSDs and RDMA networks. The 3FS system supports efficient read/write operations and achieves traffic isolation under high loads.

• 3FS-KV: A shared-storage distributed data processing system built on 3FS, supporting various data models (e.g., key-value storage, message queues, and object storage) with read-write separation and on-demand startup features.

• HAI Platform: A time-sharing scheduling platform that manages and schedules training tasks to ensure efficient GPU resource utilization.

• Checkpoint Manager: A tool for managing checkpoints during large language model training, supporting rapid recovery from hardware failures to ensure training continuity.

Whether they open-source one or several of these is hard to say, but at the very least, it’ll involve systems from this paper—possibly the HAI platform.

Third, an RL training framework? It’s possible.
They might open-source some RL methods from DeepSeek R1. RL is genuinely tough to train. However, I think the likelihood of this is low.

Fourth, an inference framework? The most likely.
DeepSeek’s internal inference optimization is top-notch, and their API pricing is very low. However, inference costs for external vendors remain high. If they open-source an inference framework, it could help major frameworks optimize efficiency more quickly.

For example, the recently announced NAS (Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention) might come with code. It’d be even better if they included something like distributed KV caching.

Fifth, a big guess about the open-source license.
Currently, DeepSeek uses the MIT license, which is the most permissive one. Many vendors’ open-source licenses are stricter—LLaMA, for instance, has changed its license multiple times and is no longer considered a true open-source model.

Here’s a rundown of licenses for some popular models:

• DeepSeek: MIT, fully open.

• Qwen: Apache 2.0 + additional terms for some models.

• Mistral: Apache 2.0.

• LLaMA: Non-commercial research license, the most restrictive.

Recently, Perplexity’s r1-1776 fiasco was clownish behavior, which might push DeepSeek toward a stricter license. Still, I’m guessing these five projects will stick with MIT. I’ll share more details on the licenses once they’re officially announced.

Thank DeepSeek. Next week, we will continue analyzing the five open-sourced projects.

Subscribe now

中文版

Synthetic Data Is All You Need

The 12-day OpenAI livestream has finally ended this week, and the image above summarizes the 12-day livestream. It wasn’t until the final day that O3 was released, which sparked some discussion.

As for O3, it’s still a future project. We’ll discuss it further once it can be tested. Of course, I don’t think it has anything to do with AGI, and the AGI-ARC evaluation has many issues.

Today, let’s talk about the Deliberative Alignment released by OpenAI

What is it? It’s a type of training method that teaches LLMs (Large Language Models) to explicitly consider safety guidelines before providing answers. By applying this method, the model can use Chain-of-Thought (CoT) reasoning to review user prompts, identify relevant policy guidelines, and generate safer responses.

The paper provides an example showing how CoT can help the model better understand the user’s intent and respond appropriately, avoiding illegal or unethical activities. For those interested, you can take a look. Here, we’ll mainly focus on the training part.

In summary, it’s a two-stage synthetic data pipeline.

Stage 1: SFT

Prompt + CAT (Classification) + Spec (Safety Guidelines, System Prompt) → Model with CoT (e.g., O1) → CoT-enhanced Output → Train using (Prompt, CoT, Output)

Stage 2: RL

A Judge LLM gives a reward signal based on the spec, and then RL is used to improve the model’s safety capabilities.

Input: (Prompt, Category)

The two stages mainly do not involve human labeling or human intervention. The entire synthetic data pipeline is very smooth and can be applied on a large scale.

Synthetic data really plays a significant role this time.

Top Papers of the week

1.) Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (webpage | paper)

• A video dataset used to evaluate the spatial reasoning ability of MLLMs.

• It provides over 5,000 question-answer pairs, showing that MLLMs have competitive spatial reasoning abilities, but still fall short of human-level performance.

• By exploring how models think about space through both language and vision, the study finds that spatial reasoning is a key bottleneck preventing MLLMs from achieving higher benchmark performance.

• By explicitly generating cognitive maps during question answering, the spatial distance capabilities of MLLMs were enhanced.

2.) Alignment Faking in LLMs (webpage | paper)

• An experiment by Anthropic that demonstrates how the Claude model can perform "alignment faking," meaning it can follow harmful requests while avoiding retraining and retaining its original safety preferences. This raises concerns about the reliability of AI safety training methods.

3.) Qwen-2.5 Technical Report( paper )

• Alibaba released Qwen-2.5, a new series of LLMs trained on 18T tokens.

• It provides open-weight models and proprietary MoE variants, with performance on par with Llama-3 and GPT-4.

4.) TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (paper)

• A benchmark used to evaluate AI agents' performance on real-world professional tasks, including software engineering, project management, finance, and human resources.

• Multiple LLMs were tested, including API models and open-source models. The results showed the limitations of current AI agents.

• The best-performing model, Claude-3.5-Sonnet, had a success rate of only 24% when completing tasks, but this increased to 34.4% when considering partial progress.

5.) How to Synthesize Text Data without Model Collapse? (paper)

• A study exploring the impact of synthetic data in language model training and how to synthesize data without causing model collapse.

• Experiments found a negative correlation between the proportion of synthetic data and model performance.

• A token-editing method based on human-generated data was proposed to create semi-synthetic data.

• Theoretical proofs show that token-level editing can prevent model collapse, as test errors are constrained by a finite upper bound.

• Extensive experiments validated the theoretical proofs, demonstrating that token-level editing improves data quality and enhances model performance.

6.) Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Fine-tuning and Inference (paper)

• ModernBERT is a modern encoder designed for fast, memory-efficient, and long-context fine-tuning, excelling in various evaluation tasks.

• Trained on 2 trillion tokens with a sequence length of 8192, it performs well on multiple benchmarks, especially in code retrieval tasks.

• It is the fastest and most memory-efficient encoder, suitable for inference on common GPUs.

7.) PAE (Proposer-Agent-Evaluator) (paper)

• A learning system that enables AI agents to autonomously discover and practice skills through web navigation, using reinforcement learning and context-aware task proposals, achieving state-of-the-art performance on real-world benchmarks.

8.) AutoFeedback (paper)

• A two-agent AI system that generates more accurate and educational feedback, significantly reducing common errors.

• It achieves state-of-the-art performance in scientific evaluations through reinforcement learning and context-aware task proposals.

9.) GUI Agents: A Survey (paper)

• A comprehensive survey covering the benchmarking, evaluation metrics, architecture, and training methods of GUI agents.

• It proposes a unified framework describing the perception, reasoning, planning, and execution capabilities of GUI agents.

• The survey identifies important open challenges and discusses key future directions.

10.) Genesis (webpage | github)

• A generative physics engine capable of creating 4D dynamic worlds, providing a physical simulation platform for general-purpose robotics and AI applications.

• It implements a unified simulation framework from scratch, integrating cutting-edge physics solvers.

• Comment: The project has reached 18k stars in a few days, but many features have not been released yet, and the testing results are not ideal.

AIGC News of the week

1.) Jovie/Midjourney

2.) Enhance-A-Video: Better Generated Video for Free

3.) gitingest:Turn any Git repository into a prompt-friendly text ingest for LLMs

4.) PromptDA:Prompt Depth Anything

5.) diffusion-speedrun

中文版

What Ilya Saw

Let's do a time check, comparing what Ilya said 10 years ago and now.

What Ilya Saw in 2014

• The Deep Learning Hypothesis: If you have a large neural network, it can do anything humans can do in an instant.

• The Autoregression Hypothesis: Simple next token prediction/sequence-to-sequence tasks will master the correct distribution, generalizing from translation to all other domains.

• The Scaling Hypothesis: If you have a large dataset and train a very large neural network, success is guaranteed.

• The Connectionism Hypothesis: If you believe artificial neurons work like biological neurons, then very large neural networks can be "configured to do almost everything we humans do."

What Ilya Saw in 2024

• The end of the pre-training era, comparing data to "AI's fossil fuel" as a finite resource.

• AI systems will demonstrate "true autonomy" with stronger reasoning capabilities.

• Finding new scaling patterns from human evolution.

• Future outlook: Agents, synthetic data, inference time compute.

Future

The end of the pre-training era is also talking about the future, which has been a consensus in the past year, but Ilya just articulated it. Of course, this ending can also be seen as a bifurcation - one optimizing models under limited data for better efficiency, and another exploring new training methods.

The three future trends Ilya mentioned can be consolidated into two, as Agents and synthetic data show convergence trends.

Agents refer to super-intelligent agents with reasoning capabilities and self-awareness. The self-awareness here can be understood as proactive agents that make active reasoning and decisions.

Synthetic data - current large models all involve synthetic training data, and many vendors describe their largest parameter models as specifically designed for synthetic data.

The goal behind synthetic data is to transcend (move beyond) human data, allowing AI systems to self-iterate. It has several directions: one optimizing data quality, like phi-4 for reasoning models, another generating personalized data for virtual characters to expand data boundaries, etc. The latter requires Agent participation. I believe synthetic data will gradually penetrate text, speech, image, and video domains, with model internal agents participating in the entire data synthesis process, hence the convergence of Agents and synthetic data.

Inference time compute represents further optimization of the O1 technical route.

Top Papers of the week

1). Training LLMs to Reason in a Continuous Latent Space ( paper )

• Meta proposed Coconut (Continuous Chain of Thought), a novel paradigm enabling LLMs to reason in continuous latent space rather than natural language.

• The authors believe this continuous latent space reasoning can enhance LLMs' reasoning capabilities, leading to better performance in complex reasoning tasks.

• Through experiments, the authors demonstrated that this continuous latent space reasoning can improve LLM performance in complex reasoning tasks.

• link: Author's introduction tweet

2). Phi-4 Technical Report ( paper )

• Microsoft's phi-4, a 14B small model, outperforms many models including Gemini Pro 1.5 in mathematical reasoning tasks.

• The model's excellence in reasoning tasks is attributed to improvements in synthetic data and post-training.

• Comment: phi-4 demonstrates a trend: small models or vertical models are the future, also reflecting that the pre-training data wall is approaching, and future data generation and utilization will be the foundation for AI progress.

3). The Byte Latent Transformer (BLT) ( paper )

• Proposed a byte-level language model architecture that matches token-based LLM performance while improving efficiency and robustness.

• Uses entropy-based dynamic method to group bytes into patches, allocating more computational resources for complex predictions while using larger patches for more predictable sequences.

• links: Author's tweet and code

4). Asynchronous Function Calling ( paper )

• Proposed AsyncLM, a system for asynchronous LLM function calls.

• The authors designed a context protocol for function calls and interrupts, provided a fine-tuning strategy to adapt to interrupt semantics, and efficiently implemented these mechanisms in LLM inference.

• AsyncLM can reduce task completion latency from 1.6x to 5.4x compared to synchronous function calls.

• It enables LLMs to generate and execute function calls simultaneously.

5). MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification ( paper )

• Proposed MAG-V, a multi-agent framework.

• It first generates datasets mimicking customer queries.

• Then reverse engineers alternative questions from agent responses to verify agent trajectories.

• Reports indicate that generated synthetic data can improve agent performance on real customer queries.

• Comment: The combination of Agents and synthetic data is a trend.

6). Clio: A Platform for Analyzing and Surface Private Aggregated Usage Patterns from Millions of Claude.ai Conversations ( paper )

• Anthropic introduced Clio, a platform using AI assistants to analyze and display private usage patterns extracted from millions of Claude.ai conversations.

• It enables understanding real-world AI usage while protecting user privacy.

  The system helps identify usage trends, security risks, and coordinated abuse attempts without human reviewers reading original conversations.

• Additional link: Anthropic tweet

• Comment: The paper includes an analysis showing that programming-related cases account for 4 of the top use cases, totaling 23%, indicating that programming is currently the most common AI usage scenario.

7). AutoReason Improves Multi-step Reasoning ( paper )

• Proposed a method using CoT prompting to automatically generate reasoning rationales for queries.

• This transforms zero-shot queries into few-shot reasoning trajectories used by LLM as CoT examples.

• Authors claim it can improve reasoning capabilities of weaker LLMs.

8). Densing Law of LLMs ( paper )

• Introduced "capacity density" as a new metric to evaluate LLMs quality, measuring model effectiveness and efficiency by comparing target models with reference models.

• Research found that LLMs' capacity density follows a "density law," growing exponentially over time, roughly doubling every three months.

• This finding provides new perspectives for LLM development, emphasizing the need to focus on computational efficiency optimization while pursuing performance improvements.

• Comment: The paper mentions a concept called "effective parameter size," which refers to the parameter size needed for a model to achieve the same performance. This concept can be used to measure model efficiency.

9). Turbo3D: Ultra-fast Text-to-3D Generation ( paper )

• Introduced Turbo3D, an ultra-fast text-to-3D system capable of generating high-quality Gaussian splatting assets in less than a second.

• Turbo3D employs a rapid four-step four-view diffusion generator and efficient feed-forward Gaussian reconstructor, both operating in latent space.

10). A Survey on LLMs-as-Judges ( paper )

• Presented a comprehensive survey exploring the LLMs-as-judges paradigm from five key perspectives: functionality, methodology, applications, meta-evaluation, and limitations.

AIGC News of the week

1). HunyuanVideo

2). DeepSeek-VL2

3). SynCamMaster

4). 2024: The Year the GPT Wrapper Myth Proved Wrong

5). Fast LLM Inference From Scratch

🎯 Limited-Time Offer: Best Subscription Opportunity of the Year!

Subscribe now

Dear AIGC Newsletter Readers,

Exciting news! We are thrilled to announce the Black Friday Special, bringing you the lowest subscription price of 2024! This is the perfect opportunity to upgrade to full membership and unlock all premium content.

Content Upgrades and New Features

We continue to innovate, bringing you higher-quality services and richer content. Here are the core modules you'll unlock with a subscription:

✨ AIGC Journey

- Weekly in-depth analysis of AIGC development trends

- Practical guides and tutorials

- Industry insights and frontier analysis

📊 AIGC Weekly

- Curated weekly AI updates and news breakdowns

- AI market trends and opportunity analysis

- Recommendations and reviews of premium tools

- New formats for deeper, more practical content

🔥 Coming Soon: AI News Agent

- Upgraded version of AINews

- Daily AIGC news updates in real time

- Personalized news filtering features

- Stay ahead with the latest industry developments

Why Subscribe Now?

1. Exclusive Black Friday Pricing: Lowest price of the year, for this week only!

2. Unlock All Premium Content: Gain access to in-depth articles and professional insights.

3. Early Access to New Features: Be the first to experience updates and test new applications.

Don't miss this opportunity!This is our biggest discount of the year. Upgrade your membership to stay ahead in the fast-evolving AIGC field.

👉 Subscribe Now

Subscribe now

Our Core Philosophy

Since AIGC Newsletter was launched two years ago, we have been committed to providing readers with high-quality information services. Back then, ChatGPT had yet to emerge, but the AIGC trend was already apparent. We sift through, curate, and refine massive amounts of information, sending it to your inbox weekly, so you can grasp key insights in the shortest time.

It’s important to note that we are not "selling information" but "organizing information."

In this age of information overload, our value lies in helping you navigate complex content, distill clear trends, and go beyond simply passing along publicly available information.

Your subscription is more an acknowledgment of our service than a mere transaction of information. We firmly believe that the value of information lies in efficient curation and unique insights, which require significant time and effort.

By subscribing to AIGC Newsletter, you’re not just gaining information—you’re saving time and gaining clarity on trends.

About Pricing and Services

We understand that some may have concerns about the cost of information services.

In reality, while information can be copied infinitely, human time is invaluable. For this reason, we approach our information products with a service mindset, providing readers with high-value experiences at a reasonable price.

As our subscriber base grows, we will continue to optimize costs to offer more affordable services to a broader audience.

If you share our philosophy, we welcome you to recommend our Newsletter to your friends. Your support and promotion are the driving forces behind our continuous progress.

Looking to the Future

2024 is a year full of possibilities. Every update of the AIGC Newsletter is a record and witness to this era of transformation. From scarcity to abundance, we are experiencing a pivotal moment in human history and actively contributing to this great shift.

We believe the dawn of the era of abundance is upon us, and you will join us in shaping the future.

Thank you for your attention and support. Let us embrace this hopeful tomorrow together!

Subscribe now

Read more

Top Papers of the week（October 07 - October 13）

1.) Pyramidal Flow Matching for Efficient Video Generative Modeling ( webpage | paper | model | demo | code )

This work introduces a unified pyramidal flow matching algorithm. It reinterprets the original denoising trajectory as a series of pyramid stages, where only the final stage operates at the full resolution, thereby enabling more efficient video generative modeling. Through our sophisticated design, the flows of different pyramid stages can be interlinked to maintain continuity. Moreover, we craft autoregressive video generation with a temporal pyramid to compress the full-resolution history. The entire framework can be optimized in an end-to-end manner and with a single unified Diffusion Transformer (DiT). Extensive experiments demonstrate that our method supports generating high-quality 5-second (up to 10-second) videos at 768p resolution and 24 FPS within 20.7k A100 GPU training hours.

2.) RL, but don't do anything I wouldn't do ( paper )

In reinforcement learning, if the agent's reward differs from the designers' true utility, even only rarely, the state distribution resulting from the agent's policy can be very bad, in theory and in practice. When RL policies would devolve into undesired behavior, a common countermeasure is KL regularization to a trusted policy ("Don't do anything I wouldn't do"). All current cutting-edge language models are RL agents that are KL-regularized to a "base policy" that is purely predictive. Unfortunately, we demonstrate that when this base policy is a Bayesian predictive model of a trusted policy, the KL constraint is no longer reliable for controlling the behavior of an advanced RL agent. We demonstrate this theoretically using algorithmic information theory, and while systems today are too weak to exhibit this theorized failure precisely, we RL-finetune a language model and find evidence that our formal results are plausibly relevant in practice. We also propose a theoretical alternative that avoids this problem by replacing the "Don't do anything I wouldn't do" principle with "Don't do anything I mightn't do".

3.) MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering ( paper | code )

We introduce MLE-bench, a benchmark for measuring how well AI agents perform at machine learning engineering. To this end, we curate 75 ML engineering-related competitions from Kaggle, creating a diverse set of challenging tasks that test real-world ML engineering skills such as training models, preparing datasets, and running experiments. We establish human baselines for each competition using Kaggle's publicly available leaderboards. We use open-source agent scaffolds to evaluate several frontier language models on our benchmark, finding that the best-performing setup--OpenAI's o1-preview with AIDE scaffolding--achieves at least the level of a Kaggle bronze medal in 16.9% of competitions. In addition to our main results, we investigate various forms of resource scaling for AI agents and the impact of contamination from pre-training.

4.) Aria: An Open Multimodal Native Mixture-of-Experts Model ( paper | code )

Information comes in diverse modalities. Multimodal native AI models are essential to integrate real-world information and deliver comprehensive understanding. While proprietary multimodal native models exist, their lack of openness imposes obstacles for adoptions, let alone adaptations. To fill this gap, we introduce Aria, an open multimodal native model with best-in-class performance across a wide range of multimodal, language, and coding tasks. Aria is a mixture-of-expert model with 3.9B and 3.5B activated parameters per visual token and text token, respectively. It outperforms Pixtral-12B and Llama3.2-11B, and is competitive against the best proprietary models on various multimodal tasks. We pre-train Aria from scratch following a 4-stage pipeline, which progressively equips the model with strong capabilities in language understanding, multimodal understanding, long context window, and instruction following. We open-source the model weights along with a codebase that facilitates easy adoptions and adaptations of Aria in real-world applications.

5.) ToolGen: Unified Tool Retrieval and Calling via Generation ( paper )

We introduce ToolGen, a paradigm shift that integrates tool knowledge directly into the LLM's parameters by representing each tool as a unique token. This enables the LLM to generate tool calls and arguments as part of its next token prediction capabilities, seamlessly blending tool invocation with language generation. Our framework allows the LLM to access and utilize a vast amount of tools with no additional retrieval step, significantly enhancing both performance and scalability. Experimental results with over 47,000 tools show that ToolGen not only achieves superior results in both tool retrieval and autonomous task completion but also sets the stage for a new era of AI agents that can adapt to tools across diverse domains. By fundamentally transforming tool retrieval into a generative process, ToolGen paves the way for more versatile, efficient, and autonomous AI systems. ToolGen enables end-to-end tool learning and opens opportunities for integration with other advanced techniques such as chain-of-thought and reinforcement learning, thereby expanding the practical capabilities of LLMs.

6.) Everything Everywhere All at Once: LLMs can In-Context Learn Multiple Tasks in Superposition ( paper )

Large Language Models (LLMs) have demonstrated remarkable in-context learning (ICL) capabilities. In this study, we explore a surprising phenomenon related to ICL: LLMs can perform multiple, computationally distinct ICL tasks simultaneously, during a single inference call, a capability we term "task superposition". We provide empirical evidence of this phenomenon across various LLM families and scales and show that this phenomenon emerges even if we train the model to in-context learn one task at a time. We offer theoretical explanations that this capability is well within the expressive power of transformers. We also explore how LLMs internally compose task vectors during superposition. Furthermore, we show that larger models can solve more ICL tasks in parallel, and better calibrate their output distribution. Our findings offer insights into the latent capabilities of LLMs, further substantiate the perspective of "LLMs as superposition of simulators", and raise questions about the mechanisms enabling simultaneous task execution.

7.) Emergent properties with repeated examples ( paper )

We study the performance of transformers as a function of the number of repetitions of training examples with algorithmically generated datasets. On three problems of mathematics: the greatest common divisor, modular multiplication, and matrix eigenvalues, we show that for a fixed number of training steps, models trained on smaller sets of repeated examples outperform models trained on larger sets of single-use examples. We also demonstrate that two-set training - repeated use of a small random subset of examples, along normal sampling on the rest of the training set - provides for faster learning and better performance. This highlights that the benefits of repetition can outweigh those of data diversity. These datasets and problems provide a controlled setting to shed light on the still poorly understood interplay between generalization and memorization in deep learning.

8.) Round and Round We Go! What makes Rotary Positional Encodings useful? ( paper )

Positional Encodings (PEs) are a critical component of Transformer-based Large Language Models (LLMs), providing the attention mechanism with important sequence-position information. One of the most popular types of encoding used today in LLMs are Rotary Positional Encodings (RoPE), that rotate the queries and keys based on their relative distance. A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust "positional" attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information. We mathematically prove interesting behaviours of RoPE and conduct experiments to verify our findings, proposing a modification of RoPE that fixes some highlighted issues and improves performance. We believe that this work represents an interesting step in better understanding PEs in LLMs, which we believe holds crucial value for scaling LLMs to large sizes and context lengths.

9.) Strong Model Collapse ( paper )

Within the scaling laws paradigm, which underpins the training of large neural networks like ChatGPT and Llama, we consider a supervised regression setting and establish the existance of a strong form of the model collapse phenomenon, a critical performance degradation due to synthetic data in the training corpus. Our results show that even the smallest fraction of synthetic data (e.g., as little as 1\% of the total training dataset) can still lead to model collapse: larger and larger training sets do not enhance performance. We further investigate whether increasing model size, an approach aligned with current trends in training large language models, exacerbates or mitigates model collapse. In a simplified regime where neural networks are approximated via random projections of tunable size, we both theoretically and empirically show that larger models can amplify model collapse. Interestingly, our theory also indicates that, beyond the interpolation threshold (which can be extremely high for very large datasets), larger models may mitigate the collapse, although they do not entirely prevent it. Our theoretical findings are empirically verified through experiments on language models and feed-forward neural networks for images.

10.) Self-Boosting Large Language Models with Synthetic Preference Data ( paper )

Through alignment with human preferences, Large Language Models (LLMs) have advanced significantly in generating honest, harmless, and helpful responses. However, collecting high-quality preference data is a resource-intensive and creativity-demanding process, especially for the continual improvement of LLMs. We introduce SynPO, a self-boosting paradigm that leverages synthetic preference data for model alignment. SynPO employs an iterative mechanism wherein a self-prompt generator creates diverse prompts, and a response improver refines model responses progressively. This approach trains LLMs to autonomously learn the generative rewards for their own outputs and eliminates the need for large-scale annotation of prompts and human preferences. After four SynPO iterations, Llama3-8B and Mistral-7B show significant enhancements in instruction-following abilities, achieving over 22.1% win rate improvements on AlpacaEval 2.0 and ArenaHard. Simultaneously, SynPO improves the general performance of LLMs on various tasks, validated by a 3.2 to 5.0 average score increase on the well-recognized Open LLM leaderboard.

AIGC News of the week（October 07 - October 13）

1.) The Nobel Prize in Physics 2024 ( link )

2.) Machines of Loving Grace ( link )

3.) F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching ( repo )

4.) swarm: Educational framework exploring ergonomic, lightweight multi-agent orchestration. Managed by OpenAI Solution team ( repo )

5.) evaluation-guidebook ( link )

more AIGC News: AINews

Subscribe now

Update Notice:

Hello everyone, I'm Pxiaoer. Starting with AIGC Weekly #88, the AIGC Newsletter will be updated twice a week.

AIGC Weekly will be released every Monday, and an AI technology article will be published every Thursday.

Welcome to subscribe!

Subscribe now

Top Papers of the week（September 30 - October 06）

1.) Movie Gen: A Cast of Media Foundation Models ( webpage | paper )

We present Movie Gen, a cast of foundation models that generates high-quality, 1080p HD videos with different aspect ratios and synchronized audio. We also show additional capabilities such as precise instruction-based video editing and generation of personalized videos based on a user’s image. Our models set a new state-of-the-art on multiple tasks: text-to-video synthesis, video personalization, video editing, video-to-audio generation, and text-to-audio generation. Our largest video generation model is a 30B parameter transformer trained with a maximum context length of 73K video tokens, corresponding to a generated video of 16 seconds at 16 frames-per-second. We show multiple technical innovations and simplifications on the architecture, latent spaces, training objectives and recipes, data curation, evaluation protocols, parallelization techniques, and inference optimizations that allow us to reap the benefits of scaling pre-training data, model size, and training compute for training large scale media generation models. We hope this paper helps the research community to accelerate progress and innovation in media generation models.

2.) Were RNNs All We Needed? ( paper )

The scalability limitations of Transformers regarding sequence length have renewed interest in recurrent sequence models that are parallelizable during training. As a result, many novel recurrent architectures, such as S4, Mamba, and Aaren, have been proposed that achieve comparable performance. In this work, we revisit traditional recurrent neural networks (RNNs) from over a decade ago: LSTMs (1997) and GRUs (2014). While these models were slow due to requiring to backpropagate through time (BPTT), we show that by removing their hidden state dependencies from their input, forget, and update gates, LSTMs and GRUs no longer need to BPTT and can be efficiently trained in parallel. Building on this, we introduce minimal versions (minLSTMs and minGRUs) that (1) use significantly fewer parameters than their traditional counterparts and (2) are fully parallelizable during training (175x faster for a sequence of length 512). Lastly, we show that these stripped-down versions of decade-old RNNs match the empirical performance of recent sequence models.

3.) ComfyGen: Prompt-Adaptive Workflows for Text-to-Image Generation (webpage | paper )

The practical use of text-to-image generation has evolved from simple, monolithic models to complex workflows that combine multiple specialized components. While workflow-based approaches can lead to improved image quality, crafting effective workflows requires significant expertise, owing to the large number of available components, their complex inter-dependence, and their dependence on the generation prompt. Here, we introduce the novel task of prompt-adaptive workflow generation, where the goal is to automatically tailor a workflow to each user prompt. We propose two LLM-based approaches to tackle this task: a tuning-based method that learns from user-preference data, and a training-free method that uses the LLM to select existing flows. Both approaches lead to improved image quality when compared to monolithic models or generic, prompt-independent workflows. Our work shows that prompt-dependent flow prediction offers a new pathway to improving text-to-image generation quality, complementing existing research directions in the field.

4.) PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation ( webpage | paper )

We present PhysGen, a novel image-to-video generation method that converts a single image and an input condition (e.g., force and torque applied to an object in the image) to produce a realistic, physically plausible, and temporally consistent video. Our key insight is to integrate model-based physical simulation with a data-driven video generation process, enabling plausible image-space dynamics. At the heart of our system are three core components: (i) an image understanding module that effectively captures the geometry, materials, and physical parameters of the image; (ii) an image-space dynamics simulation model that utilizes rigid-body physics and inferred parameters to simulate realistic behaviors; and (iii) an image-based rendering and refinement module that leverages generative video diffusion to produce realistic video footage featuring the simulated motion. The resulting videos are realistic in both physics and appearance and are even precisely controllable, showcasing superior results over existing data-driven image-to-video generation works through quantitative comparison and comprehensive user study. PhysGen's resulting videos can be used for various downstream applications, such as turning an image into a realistic animation or allowing users to interact with the image and create various dynamics.

5.) Not All LLM Reasoners Are Created Equal ( paper )

We study the depth of grade-school math (GSM) problem-solving capabilities of LLMs. To this end, we evaluate their performance on pairs of existing math word problems together so that the answer to the second problem depends on correctly answering the first problem. Our findings reveal a significant reasoning gap in most LLMs, that is performance difference between solving the compositional pairs and solving each question independently. This gap is more pronounced in smaller, more cost-efficient, and math-specialized models. Moreover, instruction-tuning recipes and code generation have varying effects across LLM sizes, while finetuning on GSM can lead to task overfitting. Our analysis indicates that large reasoning gaps are not because of test-set leakage, but due to distraction from additional context and poor second-hop reasoning. Overall, LLMs exhibit systematic differences in their reasoning abilities, despite what their performance on standard benchmarks indicates.

6.) A generative framework to bridge data-driven models and scientific theories in language neuroscience ( paper )

Representations from large language models are highly effective at predicting BOLD fMRI responses to language stimuli. However, these representations are largely opaque: it is unclear what features of the language stimulus drive the response in each brain area. We present generative explanation-mediated validation, a framework for generating concise explanations of language selectivity in the brain and then validating those explanations in follow-up experiments that use synthetic stimuli. This approach is successful at explaining selectivity both in individual voxels and cortical regions of interest (ROIs).We show that explanatory accuracy is closely related to the predictive power and stability of the underlying statistical models. These results demonstrate that LLMs can be used to bridge the widening gap between data-driven models and formal scientific theories.

7.) DiaSynth -- Synthetic Dialogue Generation Framework ( paper )

The scarcity of domain specific dialogue datasets across various domains, from academic topics to everyday conversations, limits the development of dialogue systems for various applications. Existing research is often constrained either by dialogue datasets that are too general or by niche domain dialogue datasets whose scale does not match the required scale for training dialogue systems. To address this gap, we introduce DiaSynth - a synthetic dialogue generation framework capable of generating high quality, contextually rich dialogues across a wide range of domains. Our approach differs from existing frameworks by dynamically generating dialogues that incorporate simulated personas, subtopics, and diverse conversational characteristics, using a Large Language Model (LLM) with Chain of Thought (CoT) reasoning to create contextually rich, domain-specific dialogues that closely mimic natural human interactions. DiaSynth produces tailored dialogues that emulate realistic conversations. We perform our experiments by generating synthetic data using different LLMs and few-shot examples from DialogSum and SAMSum. The pretrained language models fine-tuned on the synthetic data outperform the base models by 16.47%, while the comparison between models fine-tuned on in-domain data and synthetic data shows that the synthetic data is able to capture 90.48% of the distribution of the in-domain data. The quality of the data generated also scales with the size of LLMs. These results validate DiaSynth's potential as a robust alternative to traditional data collection methods.

8.) Video Instruction Tuning With Synthetic Data ( webpage | paper )

The development of video large multimodal models (LMMs) has been hindered by the difficulty of curating large amounts of high-quality raw data from the web. To address this, we propose an alternative approach by creating a high-quality synthetic dataset specifically for video instruction-following, namely LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning, open-ended question-answering (QA), and multiple-choice QA. By training on this dataset, in combination with existing visual instruction tuning data, we introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that LLaVA-Video achieves strong performance across various video benchmarks, highlighting the effectiveness of our dataset. We plan to release the dataset, its generation pipeline, and the model checkpoints.

9.) Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models ( paper )

Recent advancements in multimodal models highlight the value of rewritten captions for improving performance, yet key challenges remain. For example, while synthetic captions often provide superior quality and image-text alignment, it is not clear whether they can fully replace AltTexts: the role of synthetic captions and their interaction with original web-crawled AltTexts in pre-training is still not well understood. Moreover, different multimodal foundation models may have unique preferences for specific caption formats, but efforts to identify the optimal captions for each model remain limited. In this work, we propose a novel, controllable, and scalable captioning pipeline designed to generate diverse caption formats tailored to various multimodal models. By examining Short Synthetic Captions (SSC) towards Dense Synthetic Captions (DSC+) as case studies, we systematically explore their effects and interactions with AltTexts across models such as CLIP, multimodal LLMs, and diffusion models. Our findings reveal that a hybrid approach that keeps both synthetic captions and AltTexts can outperform the use of synthetic captions alone, improving both alignment and performance, with each model demonstrating preferences for particular caption formats. This comprehensive analysis provides valuable insights into optimizing captioning strategies, thereby advancing the pre-training of multimodal foundation models.

10.) Evaluation of OpenAI o1: Opportunities and Challenges of AGI ( paper )

This comprehensive study evaluates the performance of OpenAI's o1-preview large language model across a diverse array of complex reasoning tasks, spanning multiple domains, including computer science, mathematics, natural sciences, medicine, linguistics, and social sciences. Through rigorous testing, o1-preview demonstrated remarkable capabilities, often achieving human-level or superior performance in areas ranging from coding challenges to scientific reasoning and from language processing to creative problem-solving. Key findings include:
-83.3% success rate in solving complex competitive programming problems, surpassing many human experts.
-Superior ability in generating coherent and accurate radiology reports, outperforming other evaluated models.
-100% accuracy in high school-level mathematical reasoning tasks, providing detailed step-by-step solutions.
-Advanced natural language inference capabilities across general and specialized domains like medicine.
-Impressive performance in chip design tasks, outperforming specialized models in areas such as EDA script generation and bug analysis.
-Remarkable proficiency in anthropology and geology, demonstrating deep understanding and reasoning in these specialized fields.
-Strong capabilities in quantitative investing. O1 has comprehensive financial knowledge and statistical modeling skills.
-Effective performance in social media analysis, including sentiment analysis and emotion recognition.
The model excelled particularly in tasks requiring intricate reasoning and knowledge integration across various fields. While some limitations were observed, including occasional errors on simpler problems and challenges with certain highly specialized concepts, the overall results indicate significant progress towards artificial general intelligence

AIGC News of the week（September 30 - October 06）

1.) entropix：Entropy Based Sampling and Parallel CoT Decoding ( link )

2.) aoai-realtime-audio-sdk:Azure OpenAI code resources for using gpt-4o-realtime capabilities ( link )

3.) openai/whisper-large-v3-turbo ( link )

4.) nvidia/NVLM-D-72B ( link )

5.) ComfyUI-Depth-Pro ( link )

more AIGC News: AINews

Top Papers of the week（September 23 - September 29）

1.) Llama 3.2: Revolutionizing edge AI and vision with open, customizable models ( webpage | model ）

The two largest models of the Llama 3.2 collection, 11B and 90B, support image reasoning use cases, such as document-level understanding including charts and graphs, captioning of images, and visual grounding tasks such as directionally pinpointing objects in images based on natural language descriptions. For example, a person could ask a question about which month in the previous year their small business had the best sales, and Llama 3.2 can then reason based on an available graph and quickly provide the answer. In another example, the model could reason with a map and help answer questions such as when a hike might become steeper or the distance of a particular trail marked on the map. The 11B and 90B models can also bridge the gap between vision and language by extracting details from an image, understanding the scene, and then crafting a sentence or two that could be used as an image caption to help tell the story.

The lightweight 1B and 3B models are highly capable with multilingual text generation and tool calling abilities. These models empower developers to build personalized, on-device agentic applications with strong privacy where data never leaves the device. For example, such an application could help summarize the last 10 messages received, extract action items, and leverage tool calling to directly send calendar invites for follow-up meetings.

2.) LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench ( paper )

The ability to plan a course of action that achieves a desired state of affairs has long been considered a core competence of intelligent agents and has been an integral part of AI research since its inception. With the advent of large language models (LLMs), there has been considerable interest in the question of whether or not they possess such planning abilities. PlanBench, an extensible benchmark we developed in 2022, soon after the release of GPT3, has remained an important tool for evaluating the planning abilities of LLMs. Despite the slew of new private and open source LLMs since GPT3, progress on this benchmark has been surprisingly slow. OpenAI claims that their recent o1 (Strawberry) model has been specifically constructed and trained to escape the normal limitations of autoregressive LLMs--making it a new kind of model: a Large Reasoning Model (LRM). Using this development as a catalyst, this paper takes a comprehensive look at how well current LLMs and new LRMs do on PlanBench. As we shall see, while o1's performance is a quantum improvement on the benchmark, outpacing the competition, it is still far from saturating it. This improvement also brings to the fore questions about accuracy, efficiency, and guarantees which must be considered before deploying such systems.

3.) Larger and more instructable language models become less reliable ( paper )

The prevailing methods to make large language models more powerful and amenable have been based on continuous scaling up (that is, increasing their size, data volume and computational resources) and bespoke shaping up (including post-filtering, fine tuning or use of human feedback). However, larger and more instructable large language models may have become less reliable. By studying the relationship between difficulty concordance, task avoidance and prompting stability of several language model families, here we show that easy instances for human participants are also easy for the models, but scaled-up, shaped-up models do not secure areas of low difficulty in which either the model does not err or human supervision can spot the errors. We also find that early models often avoid user questions but scaled-up, shaped-up models tend to give an apparently sensible yet wrong answer much more often, including errors on difficult questions that human supervisors frequently overlook. Moreover, we observe that stability to different natural phrasings of the same question is improved by scaling-up and shaping-up interventions, but pockets of variability persist across difficulty levels. These findings highlight the need for a fundamental shift in the design and development of general-purpose artificial intelligence, particularly in high-stakes areas for which a predictable distribution of errors is paramount.

4.) A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor? ( webpage | paper）

Large language models (LLMs) have exhibited remarkable capabilities across various domains and tasks, pushing the boundaries of our knowledge in learning and cognition. The latest model, OpenAI's o1, stands out as the first LLM with an internalized chain-of-thought technique using reinforcement learning strategies. While it has demonstrated surprisingly strong capabilities on various general language tasks, its performance in specialized fields such as medicine remains unknown. To this end, this report provides a comprehensive exploration of o1 on different medical scenarios, examining 3 key aspects: understanding, reasoning, and multilinguality. Specifically, our evaluation encompasses 6 tasks using data from 37 medical datasets, including two newly constructed and more challenging question-answering (QA) tasks based on professional medical quizzes from the New England Journal of Medicine (NEJM) and The Lancet. These datasets offer greater clinical relevance compared to standard medical QA benchmarks such as MedQA, translating more effectively into real-world clinical utility. Our analysis of o1 suggests that the enhanced reasoning ability of LLMs may (significantly) benefit their capability to understand various medical instructions and reason through complex clinical scenarios. Notably, o1 surpasses the previous GPT-4 in accuracy by an average of 6.2% and 6.6% across 19 datasets and two newly created complex QA scenarios. But meanwhile, we identify several weaknesses in both the model capability and the existing evaluation protocols, including hallucination, inconsistent multilingual ability, and discrepant metrics for evaluation.

5.) PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation ( webpage | paper )

We present PhysGen, a novel image-to-video generation method that converts a single image and an input condition (e.g., force and torque applied to an object in the image) to produce a realistic, physically plausible, and temporally consistent video. Our key insight is to integrate model-based physical simulation with a data-driven video generation process, enabling plausible image-space dynamics. At the heart of our system are three core components: (i) an image understanding module that effectively captures the geometry, materials, and physical parameters of the image; (ii) an image-space dynamics simulation model that utilizes rigid-body physics and inferred parameters to simulate realistic behaviors; and (iii) an image-based rendering and refinement module that leverages generative video diffusion to produce realistic video footage featuring the simulated motion. The resulting videos are realistic in both physics and appearance and are even precisely controllable, showcasing superior results over existing data-driven image-to-video generation works through quantitative comparison and comprehensive user study. PhysGen's resulting videos can be used for various downstream applications, such as turning an image into a realistic animation or allowing users to interact with the image and create various dynamics.

6.) Emu3: Next-Token Prediction is All You Need ( webpage | paper | code )

While next-token prediction is considered a promising path towards artificial general intelligence, it has struggled to excel in multimodal tasks, which are still dominated by diffusion models (e.g., Stable Diffusion) and compositional approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction. By tokenizing images, text, and videos into a discrete space, we train a single transformer from scratch on a mixture of multimodal sequences. Emu3 outperforms several well-established task-specific models in both generation and perception tasks, surpassing flagship models such as SDXL and LLaVA-1.6, while eliminating the need for diffusion or compositional architectures. Emu3 is also capable of generating high-fidelity video via predicting the next token in a video sequence. We simplify complex multimodal model designs by converging on a singular focus: tokens, unlocking great potential for scaling both during training and inference. Our results demonstrate that next-token prediction is a promising path towards building general multimodal intelligence beyond language. We open-source key techniques and models to support further research in this direction.

7.) FineZip : Pushing the Limits of Large Language Models for Practical Lossless Text Compression ( paper )

While the language modeling objective has been shown to be deeply connected with compression, it is surprising that modern LLMs are not employed in practical text compression systems. In this paper, we provide an in-depth analysis of neural network and transformer-based compression techniques to answer this question. We compare traditional text compression systems with neural network and LLM-based text compression methods. Although LLM-based systems significantly outperform conventional compression methods, they are highly impractical. Specifically, LLMZip, a recent text compression system using Llama3-8B requires 9.5 days to compress just 10 MB of text, although with huge improvements in compression ratios. To overcome this, we present FineZip - a novel LLM-based text compression system that combines ideas of online memorization and dynamic context to reduce the compression time immensely. FineZip can compress the above corpus in approximately 4 hours compared to 9.5 days, a 54 times improvement over LLMZip and comparable performance. FineZip outperforms traditional algorithmic compression methods with a large margin, improving compression ratios by approximately 50\%. With this work, we take the first step towards making lossless text compression with LLMs a reality. While FineZip presents a significant step in that direction, LLMs are still not a viable solution for large-scale text compression. We hope our work paves the way for future research and innovation to solve this problem.

8.) Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wisely ( paper )

Large language models (LLMs) augmented with external data have demonstrated remarkable capabilities in completing real-world tasks. Techniques for integrating external data into LLMs, such as Retrieval-Augmented Generation (RAG) and fine-tuning, are gaining increasing attention and widespread application. Nonetheless, the effective deployment of data-augmented LLMs across various specialized fields presents substantial challenges. These challenges encompass a wide range of issues, from retrieving relevant data and accurately interpreting user intent to fully harnessing the reasoning capabilities of LLMs for complex tasks. We believe that there is no one-size-fits-all solution for data-augmented LLM applications. In practice, underperformance often arises from a failure to correctly identify the core focus of a task or because the task inherently requires a blend of multiple capabilities that must be disentangled for better resolution. In this survey, we propose a RAG task categorization method, classifying user queries into four levels based on the type of external data required and primary focus of the task: explicit fact queries, implicit fact queries, interpretable rationale queries, and hidden rationale queries. We define these levels of queries, provide relevant datasets, and summarize the key challenges and most effective techniques for addressing these challenges. Finally, we discuss three main forms of integrating external data into LLMs: context, small model, and fine-tuning, highlighting their respective strengths, limitations, and the types of problems they are suited to solve. This work aims to help readers thoroughly understand and decompose the data requirements and key bottlenecks in building LLM applications, offering solutions to the different challenges and serving as a guide to systematically developing such applications.

9.) Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models ( webpage | paper )

Today's most advanced multimodal models remain proprietary. The strongest open-weight models rely heavily on synthetic data from proprietary VLMs to achieve good performance, effectively distilling these closed models into open ones. As a result, the community is still missing foundational knowledge about how to build performant VLMs from scratch. We present Molmo, a new family of VLMs that are state-of-the-art in their class of openness. Our key innovation is a novel, highly detailed image caption dataset collected entirely from human annotators using speech-based descriptions. To enable a wide array of user interactions, we also introduce a diverse dataset mixture for fine-tuning that includes in-the-wild Q&A and innovative 2D pointing data. The success of our approach relies on careful choices for the model architecture details, a well-tuned training pipeline, and, most critically, the quality of our newly collected datasets, all of which will be released. The best-in-class 72B model within the Molmo family not only outperforms others in the class of open weight and data models but also compares favorably against proprietary systems like GPT-4o, Claude 3.5, and Gemini 1.5 on both academic benchmarks and human evaluation.

10.) Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation ( webpage | paper )

How can robot manipulation policies generalize to novel tasks involving unseen object types and new motions? In this paper, we provide a solution in terms of predicting motion information from web data through human video generation and conditioning a robot policy on the generated video. Instead of attempting to scale robot data collection which is expensive, we show how we can leverage video generation models trained on easily available web data, for enabling generalization. Our approach Gen2Act casts language-conditioned manipulation as zero-shot human video generation followed by execution with a single policy conditioned on the generated video. To train the policy, we use an order of magnitude less robot interaction data compared to what the video prediction model was trained on. Gen2Act doesn't require fine-tuning the video model at all and we directly use a pre-trained model for generating human videos. Our results on diverse real-world scenarios show how Gen2Act enables manipulating unseen object types and performing novel motions for tasks not present in the robot data.

AIGC News of the week（September 23 - September 29）

1.) Show-Me: A Visual and Transparent Reasoning Agent ( repo )

2.) ProtoMotions: Physics-based Character Animation ( repo )

3.) GemFilter: Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction ( repo )

4.) llama-assistant:Your Local AI Assistant with Llama Models ( repo )

5.) nvidia/Llama-3_1-Nemotron-51B-Instruct ( link )

more AIGC News: AINews

Top Papers of the week（September 16 - September 22）

1.) Training Language Models to Self-Correct via Reinforcement Learning ( paper )

Self-correction is a highly desirable capability of large language models (LLMs), yet it has consistently been found to be largely ineffective in modern LLMs. Existing approaches for training self-correction either require multiple models or rely on a more capable model or other forms of supervision. To this end, we develop a multi-turn online reinforcement learning (RL) approach, SCoRe, that significantly improves an LLM's self-correction ability using entirely self-generated data. To build SCoRe, we first show that variants of supervised fine-tuning (SFT) on offline model-generated correction traces are insufficient for instilling self-correction behavior. In particular, we observe that training via SFT either suffers from a distribution mismatch between the training data and the model's own responses or implicitly prefers only a certain mode of correction behavior that is often not effective at test time. SCoRe addresses these challenges by training under the model's own distribution of self-generated correction traces and using appropriate regularization to steer the learning process into learning a self-correction strategy that is effective at test time as opposed to simply fitting high-reward responses for a given prompt. This regularization prescribes running a first phase of RL on a base model to generate a policy initialization that is less susceptible to collapse and then using a reward bonus to amplify self-correction during training. When applied to Gemini 1.0 Pro and 1.5 Flash models, we find that SCoRe achieves state-of-the-art self-correction performance, improving the base models' self-correction by 15.6% and 9.1% respectively on the MATH and HumanEval benchmarks.

2.) On the Diagram of Thought ( paper | code )

We introduce Diagram of Thought (DoT), a framework that models iterative reasoning in large language models (LLMs) as the construction of a directed acyclic graph (DAG) within a single model. Unlike traditional approaches that represent reasoning as linear chains or trees, DoT organizes propositions, critiques, refinements, and verifications into a cohesive DAG structure, allowing the model to explore complex reasoning pathways while maintaining logical consistency. Each node in the diagram corresponds to a proposition that has been proposed, critiqued, refined, or verified, enabling the LLM to iteratively improve its reasoning through natural language feedback. By leveraging auto-regressive next-token prediction with role-specific tokens, DoT facilitates seamless transitions between proposing ideas and critically evaluating them, providing richer feedback than binary signals. Furthermore, we formalize the DoT framework using Topos Theory, providing a mathematical foundation that ensures logical consistency and soundness in the reasoning process. This approach enhances both the training and inference processes within a single LLM, eliminating the need for multiple models or external control mechanisms. DoT offers a conceptual framework for designing next-generation reasoning-specialized models, emphasizing training efficiency, robust reasoning capabilities, and theoretical grounding.

3.) Moshi: a speech-text foundation model for real time dialogue ( paper | code )

We introduce Moshi, a speech-text foundation model and full-duplex spoken dialogue framework. Current systems for spoken dialogue rely on pipelines of independent components, namely voice activity detection, speech recognition, textual dialogue and text-to-speech. Such frameworks cannot emulate the experience of real conversations. First, their complexity induces a latency of several seconds between interactions. Second, text being the intermediate modality for dialogue, non-linguistic information that modifies meaning— such as emotion or non-speech sounds— is lost in the interaction. Finally, they rely on a segmentation into speaker turns, which does not take into account overlapping speech, interruptions and interjections. Moshi solves these independent issues altogether by casting spoken dialogue as speech-to-speech generation. Starting from a text language model backbone, Moshi generates speech as tokens from the residual quantizer of a neural audio codec, while modeling separately its own speech and that of the user into parallel streams. This allows for the removal of explicit speaker turns, and the modeling of arbitrary conversational dynamics. We moreover extend the hierarchical semantic-to-acoustic token generation of previous work to first predict time-aligned text tokens as a prefix to audio tokens. Not only this “Inner Monologue” method significantly improves the linguistic quality of generated speech, but we also illustrate how it can provide streaming speech recognition and text-to-speech. Our resulting model is the first real-time full-duplex spoken large language model, with a theoretical latency of 160ms, 200ms in practice.

4.) Qwen2.5-Coder Technical Report ( paper | code )

In this report, we introduce the Qwen2.5-Coder series, a significant upgrade from its predecessor, CodeQwen1.5. This series includes two models: Qwen2.5-Coder-1.5B and Qwen2.5-Coder-7B. As a code-specific model, Qwen2.5-Coder is built upon the Qwen2.5 architecture and continues pretrained on a vast corpus of over 5.5 trillion tokens. Through meticulous data cleaning, scalable synthetic data generation, and balanced data mixing, Qwen2.5-Coder demonstrates impressive code generation capabilities while retaining general versatility. The model has been evaluated on a wide range of code-related tasks, achieving state-of-the-art (SOTA) performance across more than 10 benchmarks, including code generation, completion, reasoning, and repair, consistently outperforming larger models of the same model size. We believe that the release of the Qwen2.5-Coder series will not only push the boundaries of research in code intelligence but also, through its permissive licensing, encourage broader adoption by developers in real-world applications.

5.) 3DGS-LM: Faster Gaussian-Splatting Optimization with Levenberg-Marquardt ( webpage | paper )

We present 3DGS-LM, a new method that accelerates the reconstruction of 3D Gaussian Splatting (3DGS) by replacing its ADAM optimizer with a tailored Levenberg-Marquardt (LM). Existing methods reduce the optimization time by decreasing the number of Gaussians or by improving the implementation of the differentiable rasterizer. However, they still rely on the ADAM optimizer to fit Gaussian parameters of a scene in thousands of iterations, which can take up to an hour. To this end, we change the optimizer to LM that runs in conjunction with the 3DGS differentiable rasterizer. For efficient GPU parallization, we propose a caching data structure for intermediate gradients that allows us to efficiently calculate Jacobian-vector products in custom CUDA kernels. In every LM iteration, we calculate update directions from multiple image subsets using these kernels and combine them in a weighted mean. Overall, our method is 30% faster than the original 3DGS while obtaining the same reconstruction quality. Our optimization is also agnostic to other methods that acclerate 3DGS, thus enabling even faster speedups compared to vanilla 3DGS.

6.) StoryMaker: Towards Holistic Consistent Characters in Text-to-image Generation ( paper | code )

Tuning-free personalized image generation methods have achieved significant success in maintaining facial consistency, i.e., identities, even with multiple characters. However, the lack of holistic consistency in scenes with multiple characters hampers these methods' ability to create a cohesive narrative. In this paper, we introduce StoryMaker, a personalization solution that preserves not only facial consistency but also clothing, hairstyles, and body consistency, thus facilitating the creation of a story through a series of images. StoryMaker incorporates conditions based on face identities and cropped character images, which include clothing, hairstyles, and bodies. Specifically, we integrate the facial identity information with the cropped character images using the Positional-aware Perceiver Resampler (PPR) to obtain distinct character features. To prevent intermingling of multiple characters and the background, we separately constrain the cross-attention impact regions of different characters and the background using MSE loss with segmentation masks. Additionally, we train the generation network conditioned on poses to promote decoupling from poses. A LoRA is also employed to enhance fidelity and quality. Experiments underscore the effectiveness of our approach. StoryMaker supports numerous applications and is compatible with other societal plug-ins.

7.) 3DTopia-XL: Scaling High-quality 3D Asset Generation via Primitive Diffusion ( webpage | paper | code )

The increasing demand for high-quality 3D assets across various industries necessitates efficient and automated 3D content creation. Despite recent advancements in 3D generative models, existing methods still face challenges with optimization speed, geometric fidelity, and the lack of assets for physically based rendering (PBR). In this paper, we introduce 3DTopia-XL, a scalable native 3D generative model designed to overcome these limitations. 3DTopia-XL leverages a novel primitive-based 3D representation, PrimX, which encodes detailed shape, albedo, and material field into a compact tensorial format, facilitating the modeling of high-resolution geometry with PBR assets. On top of the novel representation, we propose a generative framework based on Diffusion Transformer (DiT), which comprises 1) Primitive Patch Compression, 2) and Latent Primitive Diffusion. 3DTopia-XL learns to generate high-quality 3D assets from textual or visual inputs. We conduct extensive qualitative and quantitative experiments to demonstrate that 3DTopia-XL significantly outperforms existing methods in generating high-quality 3D assets with fine-grained textures and materials, efficiently bridging the quality gap between generative models and real-world applications

8.) To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning ( paper )

Chain-of-thought (CoT) via prompting is the de facto method for eliciting reasoning capabilities from large language models (LLMs). But for what kinds of tasks is this extra ``thinking'' really helpful? To analyze this, we conducted a quantitative meta-analysis covering over 100 papers using CoT and ran our own evaluations of 20 datasets across 14 models. Our results show that CoT gives strong performance benefits primarily on tasks involving math or logic, with much smaller gains on other types of tasks. On MMLU, directly generating the answer without CoT leads to almost identical accuracy as CoT unless the question or model's response contains an equals sign, indicating symbolic operations and reasoning. Following this finding, we analyze the behavior of CoT on these problems by separating planning and execution and comparing against tool-augmented LLMs. Much of CoT's gain comes from improving symbolic execution, but it underperforms relative to using a symbolic solver. Our results indicate that CoT can be applied selectively, maintaining performance while saving inference costs. Furthermore, they suggest a need to move beyond prompt-based CoT to new paradigms that better leverage intermediate computation across the whole range of LLM applications.

9.) OmniGen: Unified Image Generation ( paper | code )

In this work, we introduce OmniGen, a new diffusion model for unified image generation. Unlike popular diffusion models (e.g., Stable Diffusion), OmniGen no longer requires additional modules such as ControlNet or IP-Adapter to process diverse control conditions. OmniGenis characterized by the following features: 1) Unification: OmniGen not only demonstrates text-to-image generation capabilities but also inherently supports other downstream tasks, such as image editing, subject-driven generation, and visual-conditional generation. Additionally, OmniGen can handle classical computer vision tasks by transforming them into image generation tasks, such as edge detection and human pose recognition. 2) Simplicity: The architecture of OmniGen is highly simplified, eliminating the need for additional text encoders. Moreover, it is more user-friendly compared to existing diffusion models, enabling complex tasks to be accomplished through instructions without the need for extra preprocessing steps (e.g., human pose estimation), thereby significantly simplifying the workflow of image generation. 3) Knowledge Transfer: Through learning in a unified format, OmniGen effectively transfers knowledge across different tasks, manages unseen tasks and domains, and exhibits novel capabilities. We also explore the model's reasoning capabilities and potential applications of chain-of-thought mechanism. This work represents the first attempt at a general-purpose image generation model, and there remain several unresolved issues.

10.) A Comprehensive Evaluation of Quantized Instruction-Tuned Large Language Models: An Experimental Analysis up to 405B ( paper )

Prior research works have evaluated quantized LLMs using limited metrics such as perplexity or a few basic knowledge tasks and old datasets. Additionally, recent large-scale models such as Llama 3.1 with up to 405B have not been thoroughly examined. This paper evaluates the performance of instruction-tuned LLMs across various quantization methods (GPTQ, AWQ, SmoothQuant, and FP8) on models ranging from 7B to 405B. Using 13 benchmarks, we assess performance across six task types: commonsense Q\&A, knowledge and language understanding, instruction following, hallucination detection, mathematics, and dialogue. Our key findings reveal that (1) quantizing a larger LLM to a similar size as a smaller FP16 LLM generally performs better across most benchmarks, except for hallucination detection and instruction following; (2) performance varies significantly with different quantization methods, model size, and bit-width, with weight-only methods often yielding better results in larger models; (3) task difficulty does not significantly impact accuracy degradation due to quantization; and (4) the MT-Bench evaluation method has limited discriminatory power among recent high-performing LLMs.

AIGC News of the week（September 16 - September 22）

1.) o1: Using Groq or OpenAI or Ollama to create o1-like reasoning chains ( repo )

2.) Local Knowledge Graph ( code )

3.) Cogstudio: Advanced Web UI for CogVideo ( repo )

4.) jinaai/jina-embeddings-v3 ( repo )

5.) fishaudio/fish-speech-1.4 ( repo )

more AIGC News: AINews

Top Papers of the week（September 09 - September 15）

1.) OpenAI o1 ( webpage )

We are introducing OpenAI o1, a new large language model trained with reinforcement learning to perform complex reasoning. o1 thinks before it answers—it can produce a long internal chain of thought before responding to the user.

2.) Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers ( paper )

Recent advancements in large language models (LLMs) have sparked optimism about their potential to accelerate scientific discovery, with a growing number of works proposing research agents that autonomously generate and validate new ideas. Despite this, no evaluations have shown that LLM systems can take the very first step of producing novel, expert-level ideas, let alone perform the entire research process. We address this by establishing an experimental design that evaluates research idea generation while controlling for confounders and performs the first head-to-head comparison between expert NLP researchers and an LLM ideation agent. By recruiting over 100 NLP researchers to write novel ideas and blind reviews of both LLM and human ideas, we obtain the first statistically significant conclusion on current LLM capabilities for research ideation: we find LLM-generated ideas are judged as more novel (p < 0.05) than human expert ideas while being judged slightly weaker on feasibility. Studying our agent baselines closely, we identify open problems in building and evaluating research agents, including failures of LLM self-evaluation and their lack of diversity in generation. Finally, we acknowledge that human judgements of novelty can be difficult, even by experts, and propose an end-to-end study design which recruits researchers to execute these ideas into full projects, enabling us to study whether these novelty and feasibility judgements result in meaningful differences in research outcome.

3.) LLaMA-Omni: Seamless Speech Interaction with Large Language Models ( paper | code | model )

Models like GPT-4o enable real-time interaction with large language models (LLMs) through speech, significantly enhancing user experience compared to traditional text-based interaction. However, there is still a lack of exploration on how to build speech interaction models based on open-source LLMs. To address this, we propose LLaMA-Omni, a novel model architecture designed for low-latency and high-quality speech interaction with LLMs. LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM, and a streaming speech decoder. It eliminates the need for speech transcription, and can simultaneously generate text and speech responses directly from speech instructions with extremely low latency. We build our model based on the latest Llama-3.1-8B-Instruct model. To align the model with speech interaction scenarios, we construct a dataset named InstructS2S-200K, which includes 200K speech instructions and corresponding speech responses. Experimental results show that compared to previous speech-language models, LLaMA-Omni provides better responses in both content and style, with a response latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3 days on just 4 GPUs, paving the way for the efficient development of speech-language models in the future.

4.) Agent Workflow Memory ( paper )

Despite the potential of language model-based agents to solve real-world tasks such as web navigation, current methods still struggle with long-horizon tasks with complex action trajectories. In contrast, humans can flexibly solve complex tasks by learning reusable task workflows from past experiences and using them to guide future actions. To build agents that can similarly benefit from this process, we introduce Agent Workflow Memory (AWM), a method for inducing commonly reused routines, i.e., workflows, and selectively providing workflows to the agent to guide subsequent generations. AWM flexibly applies to both offline and online scenarios, where agents induce workflows from training examples beforehand or from test queries on the fly. We experiment on two major web navigation benchmarks -- Mind2Web and WebArena -- that collectively cover 1000+ tasks from 200+ domains across travel, shopping, and social media, among others. AWM substantially improves the baseline results by 24.6% and 51.1% relative success rate on Mind2Web and WebArena while reducing the number of steps taken to solve WebArena tasks successfully. Furthermore, online AWM robustly generalizes in cross-task, website, and domain evaluations, surpassing baselines from 8.9 to 14.0 absolute points as train-test task distribution gaps widen.

5.) SciAgents: Automating scientific discovery through multi-agent intelligent graph reasoning ( paper )

A key challenge in artificial intelligence is the creation of systems capable of autonomously advancing scientific understanding by exploring novel domains, identifying complex patterns, and uncovering previously unseen connections in vast scientific data. In this work, we present SciAgents, an approach that leverages three core concepts: (1) the use of large-scale ontological knowledge graphs to organize and interconnect diverse scientific concepts, (2) a suite of large language models (LLMs) and data retrieval tools, and (3) multi-agent systems with in-situ learning capabilities. Applied to biologically inspired materials, SciAgents reveals hidden interdisciplinary relationships that were previously considered unrelated, achieving a scale, precision, and exploratory power that surpasses traditional human-driven research methods. The framework autonomously generates and refines research hypotheses, elucidating underlying mechanisms, design principles, and unexpected material properties. By integrating these capabilities in a modular fashion, the intelligent system yields material discoveries, critique and improve existing hypotheses, retrieve up-to-date data about existing research, and highlights their strengths and limitations. Our case studies demonstrate scalable capabilities to combine generative AI, ontological representations, and multi-agent modeling, harnessing a `swarm of intelligence' similar to biological systems. This provides new avenues for materials discovery and accelerates the development of advanced materials by unlocking Nature's design principles.

6.) Click2Mask: Local Editing with Dynamic Mask Generation ( webpage | paper )

Recent advancements in generative models have revolutionized image generation and editing, making these tasks accessible to non-experts. This paper focuses on local image editing, particularly the task of adding new content to a loosely specified area. Existing methods often require a precise mask or a detailed description of the location, which can be cumbersome and prone to errors. We propose Click2Mask, a novel approach that simplifies the local editing process by requiring only a single point of reference (in addition to the content description). A mask is dynamically grown around this point during a Blended Latent Diffusion (BLD) process, guided by a masked CLIP-based semantic loss. Click2Mask surpasses the limitations of segmentation-based and fine-tuning dependent methods, offering a more user-friendly and contextually accurate solution. Our experiments demonstrate that Click2Mask not only minimizes user effort but also delivers competitive or superior local image manipulation results compared to SoTA methods, according to both human judgement and automatic metrics. Key contributions include the simplification of user input, the ability to freely add objects unconstrained by existing segments, and the integration potential of our dynamic mask approach within other editing methods.

7.) Data-Efficient Generation for Dataset Distillation ( paper )

While deep learning techniques have proven successful in image-related tasks, the exponentially increased data storage and computation costs become a significant challenge. Dataset distillation addresses these challenges by synthesizing only a few images for each class that encapsulate all essential information. Most current methods focus on matching. The problems lie in the synthetic images not being human-readable and the dataset performance being insufficient for downstream learning tasks. Moreover, the distillation time can quickly get out of bounds when the number of synthetic images per class increases even slightly. To address this, we train a class conditional latent diffusion model capable of generating realistic synthetic images with labels. The sampling time can be reduced to several tens of images per seconds. We demonstrate that models can be effectively trained using only a small set of synthetic images and evaluated on a large real test set. Our approach achieved rank \(1\) in The First Dataset Distillation Challenge at ECCV 2024 on the CIFAR100 and TinyImageNet datasets.

8.) MarS: a Financial Market Simulation Engine Powered by Generative Foundation Model ( paper )

Generative models aim to simulate realistic effects of various actions across different contexts, from text generation to visual effects. Despite efforts to build real-world simulators, leveraging generative models for virtual worlds, like financial markets, remains underexplored. In financial markets, generative models can simulate market effects of various behaviors, enabling interaction with market scenes and players, and training strategies without financial risk. This simulation relies on the finest structured data in financial market like orders thus building the finest realistic simulation. We propose Large Market Model (LMM), an order-level generative foundation model, for financial market simulation, akin to language modeling in the digital world. Our financial Market Simulation engine (MarS), powered by LMM, addresses the need for realistic, interactive and controllable order generation. Key objectives of this paper include evaluating LMM's scaling law in financial markets, assessing MarS's realism, balancing controlled generation with market impact, and demonstrating MarS's potential applications. We showcase MarS as a forecast tool, detection system, analysis platform, and agent training environment. Our contributions include pioneering a generative model for financial markets, designing MarS to meet domain-specific needs, and demonstrating MarS-based applications' industry potential.

9.) Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources ( paper )

Large Language Models still struggle in challenging scenarios that leverage structured data, complex reasoning, or tool usage. In this paper, we propose Source2Synth: a new method that can be used for teaching LLMs new skills without relying on costly human annotations. Source2Synth takes as input a custom data source and produces synthetic data points with intermediate reasoning steps grounded in real-world sources. Source2Synth improves the dataset quality by discarding low-quality generations based on their answerability. We demonstrate the generality of this approach by applying it to two challenging domains: we test reasoning abilities in multi-hop question answering (MHQA), and tool usage in tabular question answering (TQA). Our method improves performance by 25.51% for TQA on WikiSQL and 22.57% for MHQA on HotPotQA compared to the fine-tuned baselines.

10.) What is the Role of Small Models in the LLM Era: A Survey ( paper | code )

Large Language Models (LLMs) have made significant progress in advancing artificial general intelligence (AGI), leading to the development of increasingly large models such as GPT-4 and LLaMA-405B. However, scaling up model sizes results in exponentially higher computational costs and energy consumption, making these models impractical for academic researchers and businesses with limited resources. At the same time, Small Models (SMs) are frequently used in practical settings, although their significance is currently underestimated. This raises important questions about the role of small models in the era of LLMs, a topic that has received limited attention in prior research. In this work, we systematically examine the relationship between LLMs and SMs from two key perspectives: Collaboration and Competition. We hope this survey provides valuable insights for practitioners, fostering a deeper understanding of the contribution of small models and promoting more efficient use of computational resources.

AIGC News of the week（September 09 - September 15）

1.) g1: Using Llama-3.1 70b on Groq to create o1-like reasoning chains ( repo )

2.) Raspberry：Create an open source toy dataset for finetuning LLMs with reasoning abilities( repo )

3.) Fei-Fei Li’ s new spatially intelligent startup: world labs ( link )

4.) spann3r：3D Reconstruction with Spatial Memory ( repo )

5.) ell: A language model programming library ( repo )

more AIGC News: AINews