The 'System 2' LLM: How Models Learn to Reason (o1, R1)

Introduction: Two Systems of Thinking

In cognitive science, Nobel laureate Daniel Kahneman described human thinking as two distinct systems:

• System 1: Fast, automatic, intuitive (e.g., recognizing faces, reading emotions)
• System 2: Slow, deliberate, analytical (e.g., solving math problems, planning)

Traditional LLMs operate almost entirely in System 1 mode: they generate responses instantly, token by token, with no deliberate planning or self-reflection. Ask GPT-4 a question, and it starts answering immediately—no visible “thinking time.”

But what if LLMs could engage System 2 thinking—spending time to reason, plan, critique, and refine before answering?

Enter reasoning-focused models like OpenAI’s o1 and DeepSeek’s R1: models trained to think before they speak.

System 1 vs. System 2 LLMs

System 1: Traditional LLMs

Characteristics:

• Instant response generation
• No visible intermediate reasoning
• Token-by-token streaming
• Fast but potentially shallow

System 2: Reasoning Models (o1, R1)

Characteristics:

• Deliberate reasoning phase
• Self-verification and correction
• Multi-step planning
• Slower but more accurate

The Reasoning Process: Internal Monologue

Reasoning models generate an internal thought process before producing the final answer. This is often called a scratchpad or chain-of-thought.

How System 2 Models Are Trained

Traditional LLMs are trained to predict the next token. Reasoning models add an extra step: learn to generate reasoning traces that lead to correct answers.

Training Pipeline

Step 1: Supervised Learning on Reasoning Traces

Train the model on examples with explicit reasoning:

Input: "What is the capital of the country where the Eiffel Tower is located?"

Reasoning trace (training data):
- The Eiffel Tower is in Paris
- Paris is in France
- The capital of France is Paris

Output: Paris

The model learns to generate intermediate reasoning steps.

Step 2: Reinforcement Learning with Outcome Rewards

Use RL to optimize for correct final answers, not just plausible-sounding reasoning:

Reward = +1 if final answer is correct, -1 if wrong

This encourages the model to:

• Try multiple approaches
• Self-correct mistakes
• Verify answers

Chain-of-Thought vs. Scratchpad Reasoning

Chain-of-Thought (CoT)

User sees the reasoning process:

Example:

Q: What is 15% of 80?

A: Let me calculate this step by step:
   - 10% of 80 = 8
   - 5% of 80 = 4
   - 15% = 10% + 5% = 8 + 4 = 12

Answer: 12

Scratchpad Reasoning (o1, R1)

Reasoning is hidden from the user:

Example:

Q: What is 15% of 80?

[Internal scratchpad - not shown to user]
- Let me think... 15% = 0.15
- 0.15 × 80 = ?
- 80 × 0.1 = 8
- 80 × 0.05 = 4
- 8 + 4 = 12
- Verify: 12/80 = 0.15 ✓

A: 12

Branching Reasoning: Exploring Multiple Paths

Advanced reasoning models can explore multiple reasoning paths and select the best one:

This is similar to Monte Carlo Tree Search in game-playing AI (AlphaGo), but for reasoning.

Self-Correction: The Critique Loop

One of the most powerful features of System 2 models is self-correction:

Example: Self-Correction in Action

Question: If 3x + 6 = 45, what is x?

[Internal Reasoning]
Initial attempt:
- 3x = 45 - 6
- 3x = 39
- x = 13

Verification:
- Check: 3(13) + 6 = 39 + 6 = 45 ✓
- This is correct!

Final Answer: x = 13

Compare to a case where the model catches its own mistake:

Question: If 3x + 6 = 45, what is x?

[Internal Reasoning]
Initial attempt:
- 3x = 45 + 6  [MISTAKE]
- 3x = 51
- x = 17

Verification:
- Check: 3(17) + 6 = 51 + 6 = 57
- Wait, that's 57, not 45!
- I made an error.

Corrected attempt:
- 3x = 45 - 6  [FIXED]
- 3x = 39
- x = 13

Re-verification:
- Check: 3(13) + 6 = 39 + 6 = 45 ✓
- Correct!

Final Answer: x = 13

Real-World Example: OpenAI o1 vs. GPT-4

Problem: Complex Logic Puzzle

Question: “In a room, there are three light switches. Each switch controls one of three light bulbs in another room. You can flip the switches as many times as you want, but you can only enter the other room once. How do you determine which switch controls which bulb?”

GPT-4 (System 1) Response

You can determine which switch controls which bulb by:
1. Flipping switches in different combinations
2. Entering the room to see which lights are on
3. Matching the patterns

[Incomplete/incorrect reasoning]

o1 (System 2) Response

[Internal Reasoning - simplified]
- Direct observation won't work since I can only enter once
- Need to use multiple signals...
- Idea: Switches affect bulbs, but what if I use heat?

Step-by-step plan:
1. Turn on switch 1, leave it on for 10 minutes
2. Turn off switch 1, turn on switch 2
3. Leave switch 3 off
4. Enter the room

Results interpretation:
- Bulb that is ON → controlled by switch 2
- Bulb that is OFF but WARM → controlled by switch 1
- Bulb that is OFF and COOL → controlled by switch 3

This works because the first bulb was on long enough to heat up!

Final Answer: [Complete solution using heat as a signal]

The reasoning model explores the problem space more thoroughly and finds creative solutions.

Architecture: How Is This Implemented?

While exact details are proprietary, we can infer the architecture:

Special Tokens for Reasoning Control

Models likely use special tokens to delineate reasoning:

<think>
Let me break this problem down step by step.
First, I need to understand what is being asked...
[... extensive reasoning ...]
So the answer is 42.
</think>

<answer>
The answer is 42.
</answer>

During inference:

• Everything between <think> and </think> is hidden from the user
• Only content in <answer> tags is shown

Training for Self-Correction: Process Supervision

Instead of rewarding only the final answer, reward intermediate reasoning steps:

This is called Process-Supervised Reinforcement Learning (from OpenAI’s research).

Compute-Time Scaling: More Thinking = Better Answers

A key insight: inference-time computation can improve accuracy:

Traditional scaling: Bigger models = better performance New paradigm: More thinking time = better performance

This allows dynamic quality/cost trade-offs at inference time!

Comparison: Traditional vs. Reasoning Models

Limitations and Challenges

1. Computational Cost

More reasoning = more tokens = higher cost:

Traditional model: 50 tokens → $0.001
Reasoning model: 500 reasoning + 50 output = 550 tokens → $0.011

2. Latency

Reasoning takes time:

GPT-4: ~2 seconds
o1: ~30 seconds for complex problems

3. Hallucination in Reasoning

The model might generate plausible-sounding but incorrect reasoning:

4. Opacity

Even if reasoning is shown, it may not reflect the model’s “true” process—it’s just more generated text.

Use Cases: When to Use Reasoning Models

Excellent For:

• Complex math problems: Multi-step calculations
• Coding challenges: Algorithm design, debugging
• Logic puzzles: Requires deep reasoning
• Scientific problem-solving: Hypothesis generation and testing
• Strategic planning: Multi-step decision making

Not Ideal For:

• Simple questions: “What is the capital of France?”
• Creative writing: May over-analyze
• Speed-critical applications: Too slow
• Cost-sensitive applications: 10× more expensive

The Future: Hybrid Models

Future models may dynamically choose reasoning depth:

Conclusion: The Dawn of Thinking Machines

Reasoning models like o1 and R1 represent a paradigm shift in AI:

From: Predict the next token as fast as possible To: Think carefully before responding

This mirrors human cognition more closely—we don’t always answer immediately; sometimes we need to stop, think, plan, and verify.

The implications are profound:

• Accuracy: Better performance on complex tasks
• Reliability: Self-correction reduces errors
• Transparency: Reasoning traces provide interpretability
• Flexibility: Tune reasoning depth based on task difficulty

We’re moving from reactive AI (System 1) to deliberative AI (System 2). The next frontier is teaching models not just what to think, but how to think.

Key Takeaways

• System 1 vs. System 2: Fast intuition vs. slow deliberation
• Scratchpad Reasoning: Hidden internal thought process
• Self-Correction: Models can verify and fix their own mistakes
• Process Supervision: Reward intermediate reasoning steps
• Compute-Time Scaling: More thinking time = better accuracy
• Trade-offs: Accuracy and reliability vs. cost and latency

The future of LLMs is not just bigger models—it’s smarter reasoning.

Further Reading

• OpenAI o1 System Card
• DeepSeek R1: Incentivizing Reasoning via RL
• Chain-of-Thought Prompting Elicits Reasoning in LLMs
• Let’s Verify Step by Step (Process Supervision)
• System 1 and System 2 Thinking (Kahneman)