STATE OF AI 2026: DeepSeek, Reasoning Models, and the Rise of Inference Scaling

📋 Overview

• Type: Podcast / Technical Roundtable / Strategic Forecast
• Main Topic: A retrospective analysis of the "DeepSeek Moment" of 2025, the technical shift impactful post-training techniques (RLVR), and predictions for the AI landscape in 2026.
• Speakers:
  • Lex Fridman (Host)
  • Sebastian Raschka (AI Researcher, Educator, Author of Build a LLM From Scratch)
  • Nathan Lambert (AI Researcher at AI2, RLHF Expert, Post-training Lead)

🎯 Core Purpose & Context

This conversation takes place in early 2026 (contextual timeframe), looking back at the explosive year of 2025. The primary goal is to dissect the technical breakthroughs that occurred—specifically the shift from massive pre-training to inference-time scaling and Reinforcement Learning with Verifiable Rewards (RLVR). The trio analyzes the geopolitical tension between US closed-source labs (OpenAI, Anthropic, Google) and Chinese open-weight dominance (DeepSeek, Qwen). They aim to provide a "state of the art" snapshot for researchers and engineers, demystifying how reasoning models work and where the economic value of AI will settle.

---

🧠 Key Concepts & Technical Definitions

The speakers provide high-level educational breakdowns of complex mechanisms.

• RLVR (Reinforcement Learning with Verifiable Rewards):

  • Definition: A training method where the model is rewarded based on the accuracy of a final answer (e.g., in math or code) rather than human preference.
  • Mechanism: It allows the model to generate thousands of internal "thoughts" or steps. If the final answer is correct, the entire chain is reinforced. This simulates "System 2" thinking.
  • Impact: This enabled the "DeepSeek R1" moment, proving that smaller models can achieve state-of-the-art results by thinking longer, rather than just being bigger.

• Inference-Time Scaling (The O1 Paradigm):

  • Concept: Instead of spending capital on training a massive model once (Pre-training scaling), you spend capital during the actual query (Inference scaling).
  • Trade-off: You trade latency (speed) for intelligence. The model generates hidden tokens ("thoughts") to self-correct before outputting the final answer.
  • Economic Shift: Shifts cost from the developer (training) to the user (inference time).

• Pre-training vs. Post-training:

  • Pre-training: "Soaking up knowledge." Predicting the next token on a massive corpus (Internet/Books). Huge compute, fixed cost.
  • Post-training: "Unlocking skills." Using RLHF or RLVR to teach the model how to use the knowledge it already has. The current frontier of innovation.

• Text Diffusion Models:

  • Alternative Architecture: Unlike Transformers (which write one word at a time, left-to-right), diffusion models start with noise and refine the entire text block simultaneously.
  • Promise: Potentially much faster for generating long code blocks or diffs, though currently less capable at reasoning.

---

🧭 Strategic Analysis & "Game Changers"

1. The Geopolitical Flip: China Owns "Open," US Owns "Closed"

A major theme is the inversion of historical tech norms. US companies (OpenAI, Google) have become secretive "closed shops" driven by profit and safety concerns. Meanwhile, Chinese labs (DeepSeek, Qwen/Alibaba, 01.AI) have aggressively pushed Open Weights.

• The "So What?": By flooding the market with state-of-the-art (SOTA) open models, China is capturing the "mindshare" of global developers. Western startups are building on Chinese infrastructure because it is free and performant.
• The Response: The "Atom Project" (mentioned by Nathan) is a policy push to force the US to fund open-weight models to regain relevance in the research community.

2. The Death of the "Vibe Check" (RLHF is Plateauing)

Traditional RLHF (Reinforcement Learning from Human Feedback) is hitting a ceiling. You cannot scale it indefinitely because human preferences are fuzzy and hard to grade.

• Game Changer: RLVR is the new engine. Because math and code have objective truth (the code runs or it doesn't), you can scale training data synthetically without humans. This is why coding and math abilities effectively skyrocketed in 2025.

3. The "Jagged" Intelligence Frontier

We are not approaching a uniform "God-like" AI. Instead, AI is becoming deeply jagged.

• Insight: AI is superhuman at writing individual functions (Python, React components) but terrible at distributed system architecture or novel scientific discovery.
• Implication: The "Superhuman Coder" isn't a replacement for engineers yet; it's a replacement for typing. The role of the human shifts from "writer" to "verifier" and "architect."

4. Convergence of Architecture

Despite the hype, Transformers are still the only game in town.

• From GPT-2 to GPT-5/DeepSeek, the architecture hasn't fundamentally changed. Innovations are algorithmic (RL strategies, data curation) rather than architectural.
• Hidden Connection: This suggests we might be nearing a local maximum on architecture, forcing labs to innovate on data quality and system integration (agents) instead.

---

🎙️ Notable Quotes & Insights

• On Coding: "Debugging is like a drink of water after walking through a desert... but you skip the desert part where you suffer [with AI]." — Lex/Sebastian on the loss of 'struggle' in learning.
• On Pre-training: "Pre-training is soaking up the knowledge. Post-training is the skill unlock." — Sebastian Raschka
• On Timelines: "Big Tech isn't going to spend $100 billion much faster than we get an automated AI researcher... The dream of one model to rule them all is kind of dying." — Nathan Lambert
• On 996 Culture: "The leapfrogging nature... is fueled by human expense. People are definitely burning out." — Nathan Lambert on the frantic pace of AI labs.

---

📊 Detailed Breakdown

The Landscape of 2025/2026

• [00:03:22] The DeepSeek Moment: The conversation pivots around January 2025, when the Chinese lab DeepSeek released DeepSeek R1. It offered SOTA performance at a fraction of the cost, shocking Western labs.
• [00:06:29] Model Wars:
  • Anthropic: Betting hard on "Claude Opus 4.5" and coding capabilities. They are seen as the "least chaotic" and most culturally coherent lab.
  • Google: Gemini 3 is technically excellent (specifically for long-context retrieval) but lacks the "meme hype" of Anthropic or OpenAI. Google has a structural advantage in chips (TPUs) and doesn't pay the "Nvidia tax."
  • OpenAI: Still the "incumbent," but chaotic. Their move to O1 (reasoning models) defined the inference-scaling trend.
• [00:10:29] The Chinese Strategy: Chinese firms (DeepSeek, MiniMax, Z.AI) are releasing models openly because Western enterprises won't buy their APIs due to security/export fears. Open weights are their only path to global influence.

User Experience & "Vibe Coding"

• [00:14:32] The Router Model: Users now use "Routers" (e.g., ChatGPT Pro) that dynamically switch between "Thinking models" (for hard math) and "Fast models" (for bash scripts).
• [00:22:46] Coding Tools:
  • Cursor vs. Claude Code: Sebastian prefers Codex/VS Code (more control), while Nathan prefers Claude Code (text-to-app, "vibe coding").
  • The Shift: Programming is moving from "writing syntax" to "managing English specifications."
  • Survey Data [01:31:03]: Senior developers use AI more than junior developers (50%+ of shipped code). Experts are better at verifying AI slop; juniors struggle to learn without doing the "grunt work."

Technical Deep Dive: How the Models Work in 2026

• [00:41:15] Architectural Tweaks:
  • MOE (Mixture of Experts): Used by DeepSeek and Mistral. Instead of one giant brain, you have many small experts. A router picks which expert answers a query. This saves compute (sparse activation).
  • Latent Attention & Group Query Attention: Tweaks to memory (KV Cache) to allow for massive context windows (1M+ tokens) without crashing GPU memory.
• [01:06:14] The Training Pipeline Defined:
  1. Pre-training: Trillions of tokens (Internet/Synthetic Data).
  2. Mid-training: High-quality, domain-specific data (long context, complex reasoning traces).
  3. Post-training:
     • SFT (Supervised Fine-Tuning): Teaching the format.
     • RLVR (Verifiable Rewards): The "Reasoning" phase. The model tries a math problem 10,000 times; the ones that get the right answer are reinforced.
• [01:46:33] The Contamination Debate: "Qwen" models performed suspiciously high on benchmarks. Discussion on whether they are memorizing test answers or actually learning logic. (Conclusion: It's messy, but RLVR does unlock latent capability).

The "Atom Project" & US Policy

• [03:53:52] The Atom Project: Nathan’s initiative. A realization that if the US doesn't publicly fund open-weight models, the global standard for AI research will become Chinese architecture.
• The Risk: Innovators in Brazil, India, or Europe might start building exclusively on DeepSeek/Qwen architecture, reducing US leverage in the tech stack.

Robotics & Physical AI

• [02:52:22] World Models: Applying LLM prediction to the physical world.
• [02:59:47] The Skepticism: The group is bearish on home robotics (Rosie the Robot) anytime soon due to the "failure is not an option" safety requirement in physical spaces.
• The Bull Case: Industrial Automation. Warehouses designed for robots (no humans allowed) will scale faster than robots designed for human homes.

Future Outlook: AGI & 2027

• [03:02:56] Defining AGI: The definition is shifting from "God AI" to "Economic Replacement." Can it replace a remote worker?
• [03:07:37] Timelines:
  • Superhuman Coder: Likely by 2027.
  • Automated AI Researcher: Post-2030.
• [03:22:44] The Plateau Risk: There is a real chance we plateau on "general" capability. We might get amazing coders and amazing biologists, but not a single "Omni-model."
• [03:41:36] Business Consolidation: Prediction that standalone LLM providers (that only sell tokens) will die or be bought. Only those with ecosystems (Apple, Google) or massive specialized workflows (Databricks, Salesforce) will survive.

---

🔑 Key Takeaways

1. Reasoning is the New Scale: The era of "bigger models are better" is transforming into "models that think longer are better." Inference-time compute (O1, DeepSeek R1) is the primary driver of value in 2026.
2. Open Source has shifted East: The US has largely abandoned open-weights for proprietary APIs, ceding the open-source ground to China, which is a massive long-term strategic risk for Western tech influence.
3. The "Senior Dev" Advantage: AI accelerates experts who can verify output. It potentially cripples juniors who bypass the "struggle" of learning. This creates a crisis for future talent pipelines.
4. Data Wall: We have run out of high-quality human text. The future depends on Synthetic Data and Verifiable Rewards (checking math/code) to create the next generation of training tokens.
5. Agency & Consolidation: We are moving toward "Agentic" workflows (AI using computers), but it's failing to work reliably on messy human interfaces. Expect huge consolidation in the AI startup market as "wrapper" companies get wiped out by foundational model capabilities.

❓ Unresolved Questions & Follow-up

• The "Data Wall" vs. Synthetic Quality: Can models truly self-improve indefinitely using synthetic data, or will they eventually undergo "model collapse" without fresh human input?
• Computer Use Reliability: Can agents actually reliable navigate a GUI (graphical user interface) without crashing? The current success rate is too low for deployment.
• US Policy Response: Will the "Atom Project" succeed? Will the US government fund a "NASA for Open Source AI" to counter DeepSeek?
• The "So What" for GDP: We have clear tech progress, but where is the GDP spike? The economic impact is still lagging behind the technical capability.