World models are having a moment. Here are some of my personal thoughts 🧵

1 / 5

The most compelling thing about WMs is counterfactual reasoning — the ability to mentally simulate different outcomes before ever interacting with the real world. That's where the real value lies.

2 / 5

There’s a commonly cited but somewhat misleading example: “A WM knows fire is dangerous, so the agent won’t touch it.” In reality, animals avoid fire because they’ve been burned before — reward shaped the actor. That’s instinct, not planning. A WM primarily models environment transitions ; it doesn’t necessarily produce reward. Crediting this to the WM conflates actor, dynamics model, and reward.

3 / 5

The reason the human brain is such a powerful WM is its ability to reason flexibly across language and vision. When playing chess, we mentally simulate different board positions after each move — images are the natural medium here. When planning our day, we list tasks, prioritize, and estimate time — that's entirely language-driven. Together, the two modalities cover the vast majority of reasoning scenarios. In this sense, LLMs are already a form of WM — CoT is essentially implicit planning in language space.

4 / 5

But a more powerful WM requires efficient visual representations. The problem today is that visual tokens are designed for pixel reconstruction, not decision-making. Generating a chessboard takes thousands of tokens, and models still make obvious errors like disappearing or duplicated pieces — yet encoding the decision-relevant information in that same image might only take a few dozen language tokens.

5 / 5

Putting it all together: the highest-priority goal for WMs right now is to find a compact visual representation, and then give models the ability to reason flexibly across modalities. Only when both are achieved will truly powerful WMs become possible.