Every model inherits two numbers from its stack: how much capability it gets per FLOP, and how many FLOPs it gets per joule. Pretraining sets most of the first. Infrastructure and the chip set the second. I think energy and efficiency are first principles of intelligence, so I have moved my research focus from robot world models to these two numbers.

Multiplied, the two numbers give capability per joule:

The formula is bookkeeping. It is true for any model. The claim is about who can move each factor. Pretraining moves the first factor through architecture, data, optimizer and learning-rate schedule. Infrastructure moves the second through kernels, memory traffic and communication; the chip sets its ceiling. Capability needs a fixed proxy, such as loss on fixed data or a fixed evaluation score.

An LLM is a system, and infrastructure and pretraining are its core. Gains in this layer transfer across models. A new world-model architecture raises capability per FLOP for one class of tasks. A better optimizer, or a kernel that moves fewer bytes, can help many models that share the same operators and hardware. That transfer is why I call this layer more fundamental.

Algorithms fit the hardware

Aidan Smith, a co-founder of Flapping Airplanes, gives an example from the brain. He says that signals in the brain are slow next to a GPU, which works in microseconds. He adds that the brain spends about 20 times more energy on white-matter wiring to get even that speed. His conclusion: “everything fits to the hardware that it’s given” (05:52).

If algorithms fit the hardware, the cost of the hardware is part of what makes an algorithm good: joules, bytes moved, microseconds.

What small experiments can already do

Two recent measurements show that small experiments can study this layer. Neither is in joules yet.

One-row BF16 GEMMs in a decode step are bound by bytes read from memory. Triton’s do_bench clears the L2 cache by writing zeros. The zeros leave dirty lines that must go back to memory first, and this adds 12–20 µs to these GEMMs on one A100. For one layer shape, the flush changed the rank of 9 of 96 Triton INT8 kernel configurations against cuBLAS. In this layer, the measurement is part of the research.

In a GPT-2-small-sized model, a simple momentum model predicts that the equilibrium effective learning rate of Muon is 24% below that of AdamW. The one run pair that reached equilibrium gave 23%. One pair cannot isolate momentum, and a gap of similar size also appears away from equilibrium, for another reason.

Where the claim stops

The strongest objection is historical. The largest past gains in capability per FLOP came from architecture and data, such as the transformer. I accept it. I put architecture inside pretraining, and I do not claim that kernels matter more than architecture.

Data is a limit that the formula hides. When the number or quality of tokens binds, FLOP efficiency does not help. Even Aidan’s lab, which bets on data efficiency, works toward it through new GPU primitives.

So the claim needs three conditions: compute and energy limit capability, data does not, and the capability proxy is stable. Under them, gains in this layer reach many models that share the same operators and hardware.

每个模型都从它的技术栈继承两个数：每个 FLOP 换来多少能力，每焦耳能算多少个 FLOP。第一个数主要由预训练决定。第二个数由芯片和基础设施决定。我认为能量和效率是智能的第一性原理，所以把研究投入从机器人世界模型转到了这两个数上。

两个数相乘，就是每焦耳的能力：

这个式子只是记账，对任何模型都成立。真正的判断在于谁能改变哪个因子。预训练通过架构、数据、优化器和学习率调度改变第一个因子。基础设施通过 kernel、显存读写和通信改变第二个因子，芯片决定它的上限。能力要用一个固定的指标代替，例如固定数据上的 loss，或固定评测集上的分数。

LLM 是一个系统，基础设施和预训练是它的核心。这一层的改进能在模型之间迁移。新的世界模型架构，只提高一类任务的每 FLOP 能力。更好的优化器，或少搬运字节的 kernel，能帮到共用同一批算子和硬件的许多模型。所以我说这一层更本质。

算法适应硬件

Flapping Airplanes 的联合创始人 Aidan Smith 举过大脑的例子。他说，和以微秒计的 GPU 相比，大脑里的信号很慢；为了这个速度，大脑还要在白质连接上多花约 20 倍的能量。他的结论是：“everything fits to the hardware that it’s given”（05:52）。

既然算法适应硬件，硬件的代价就是衡量算法好坏的一部分：焦耳、搬运的字节、微秒。

小实验已经能做什么

最近的两项测量说明，这一层可以用小实验研究。两项都还没有以焦耳为单位。

decode 阶段的单行 BF16 GEMM，耗时由从显存读出的字节数决定。Triton 的 do_bench 用写零的方法清空 L2 缓存，留下的脏缓存行要先写回显存，在一张 A100 上让这类 GEMM 多出 12 到 20 微秒。在同一个层形状下，96 个 Triton INT8 kernel 配置里有 9 个因此改变了和 cuBLAS 的快慢排名。在这一层，测量本身就是研究的一部分。

在 GPT-2 small 规模的模型上，简单的动量模型预测：平衡状态下 Muon 的有效学习率比 AdamW 低 24%。唯一达到平衡的一对运行测得 23%。只有一对运行，分离不出动量的作用；而且离开平衡状态时，也会因为别的原因出现差不多大的差距。

这个判断的边界

最有力的反驳来自历史：过去每 FLOP 能力最大的提升来自架构和数据，例如 Transformer。我接受这一点。我把架构算在预训练里，也不认为 kernel 比架构更重要。

数据是这个式子藏起来的限制。token 的数量或质量成为瓶颈时，FLOP 效率帮不上忙。即使是押注数据效率的 Aidan 的实验室，走的路也是新的 GPU 原语。

所以这个判断要满足三个条件：能力受算力和能量限制，不受数据限制，能力指标足够稳定。满足这些条件时，这一层的一次改进，能同时帮到共用同一批算子和硬件的许多模型。