1. Panorama

2. Base Model

2.1 Introduction

The pretraining use massive training text to help LLM acquire the following abilities:

1
2
3
4
5
6

language patterns
knowledge
syntax
semantic representations
code patterns
reasoning patterns

You give LLM a string of token and it will output other tokens. This is called ==next-token prediction==. After pretraining, the LLM learns basic ability. And we call LLM in this phase ==Base Model==.

The training goal is:

max logP_θ(x_t|x_< t)

2.2 Problem

The problem of base model is that it can continue but it can’t follow the instruction:

1
2

User:
Please explain TCP Three-way Handshake.

The base model may continue:

1
2

User:
Please explain TCP Four-way Handshake.

or just generate contents not qualified.

3. Level 0: Instruction Following

3.1 Introduction

We want LLM learns how to follow the instruction. So we can use SFT (Supervised Fine-Tuning) to do this.

Supervised mans that we have a answer to supervise the LLM. The training data structure is simple:

1

(prompt, answer)

e.g. 

1
2
3
4
5
6
7
8

Prompt:
Explain the TCP three-way handshake.

Answer:
The TCP three-way handshake includes:
1. The client sends a SYN packet
2. The server replies with SYN-ACK
3. The client sends an ACK

We hope to:

max P_θ(Answer|Prompt)

SFT in essence is supervised learning / behavior cloning.

We have:

1
2
3
4
5

input (Explain the TCP three-way handshake.)
  ↓
expert
  ↓
correct behavior (The TCP three-way handshake includes:...)

In essence we make the LLM imitate the expert:

1
2

Expert trajectory:
Q → A

1
2

Expert trajectory:
Q → A

3.2 Problems

3.2.1 Q1: LLM Only Learn the Answer (CoT SFT)

By normal SFT, LLM learns how to answer question according to the data it learned.

e.g. There’s a math problem:

1

2x + 3 = 7

the training data is:

1
2

2x = 4
x = 2

The LLM didn’t try other ways. It just imitate the expert trajectory.

We use ==Chain-of-Thought== to solve this problem: Not only train the answer, but also train the reasoning process.

e.g. 

1
2
3
4
5
6
7
8
9

Question:
Roger has 5 apples and buys 3 more.
How many apples does he have?

Answer:
He starts with 5.
He buys 3.
5 + 3 = 8.
Therefore he has 8 apples.

So the SFT data is now:

1
2
3
4
5

Question
   ↓
Reasoning
   ↓
Answer

instead of:

1
2
3

Question
   ↓
Answer

Now the training data is like this:

1
2
3
4
5
6
7
8
9
10
11
12
13

Q:
If x + 3 = 10, what is x?

A:
We have:

x + 3 = 10

Subtract 3 from both sides:

x = 7

Therefore x = 7.

P(this reasoning trajectory|Q)

When encountering such questions, LLM can follow this reasoning pattern to solve them.

3.2.2 Q2: LLM Don’t Know the Optimized Reasoning Path (Preference Learning)

The trajectory is provided by Human. The LLM only imitates the trajectory, it never knows that if the trajectory is best.

e.g. 

1
2

User:
What is TCP？

LLM might generate 2 answers:

1
2
3
4
5
6
7

Answer 1:

TCP is a protocol.
It exists in computer networks.
TCP is very important.
TCP can provide reliable communication.
TCP has many features.

1
2
3
4
5

Answer 2:

TCP is a connection-oriented, reliable transport layer protocol.
It utilizes mechanisms such as sequence numbers, ACKs, retransmissions, and congestion control,
Ensure reliable transmission of data.

Which answer is better?

Obviously most people will choose latter, so we can collect:

1
2
3
4
5
6
7
8

Question
   │
   ├── Answer A
   │
   └── Answer B

Human:
B > A

This data is called ==Preference Data==.

Why we need ==Preference Learning==?

e.g. problem:

1

Explain what is mutex?

LLM:

1
2
3

Answer A
Answer B
Answer C

All of the 3 answers might be correct, but :

1
2
3

A: Too wordy
B: Accurate and clear
C: Technical errors

The tradition SFT is hard to express “B is better than A”.

4. Level 1: Preference Alignment

4.1 RLHF (Reinforcement Learning from Human Feedback)

==RLHF== (Reinforcement Learning from Human Feedback) makes LLM output answers more approach to human preference.

1
2
3
4
5
6
7
8
9
10

Human Preference
        │
        ▼
  Reward Model
        │
        ▼
   RL Training
        │
        ▼
   LLM Policy

It can be divided into 3 steps:

4.1.1 Train a Reward Model

Collect performance data like:

1
2
3
4
5
6
7

Question
   │
   ├── Answer A
   └── Answer B

Human:
B > A

And train a Reward Model. The goal of the model is to assess “How good the answer is”

e.g. 

1

RewardModel(question, answer)

output:

1
2
3
4

A → 0.2
B → 0.9

0.9 > 0.2

Then the model learns B is more preference-aligned.

4.1.2 LLM Generates Answer

Input question to LLM, and it will output a answer. Then send Question + Answer to Reward Model, the Reward Model will judge the answer’s quality.

1
2
3
4
5
6
7

Question + Answer
       │
       ▼
Reward Model
       │
       ▼
Reward = 0.8

4.1.3 Improve Reward by RL

Now we hope that the future answer will get higher reward:

1
2
3
4
5
6
7
8
9
10
11
12

Generate a good answer
↓
Reward
↓
Increase the probability of generating this answer


Generate a bad answer
↓
Punish
↓
Reduce the probability of generating this answer

4.2 PPO (Proximal Policy Optimization)

PPO is a classic RLHF algorithm. After getting reward(s) by reward model, PPO is used to update LLM’s weight.

Its core thought is :

• Need to make LLM more inclined to generate high-reward responses, while ensuring that a single update does not deviate too far from the original model.

…

4.3 DPO (Direct Preference Optimization)

4.3.1 Difference

Tradition RLHF:

1
2

1. train a reward model
2. optimize LLM by RL algorithm like PPO

DPO:

1

Skip the explicit Reward Model and PPO, train model with "Human Preference Data" directly.

The data DPO can see is the same as RLHF’s:

1
2
3

Prompt
Chosen Answer
Rejected Answer

The tradition RLHF:

1
2
3
4
5
6
7
8
9

Preference
    ↓
Reward Model
    ↓
Reward
    ↓
PPO
    ↓
LLM

The DPO:

1
2
3
4
5
6
7

Preference
     │
     ▼
    DPO
     │
     ▼
    LLM

4.3.2 How does DPO know which answer is better?

Assume:

1
2
3
4

Prompt = x

Chosen   = y_w
Rejected = y_l

we hope:

1
2

π(y_w | x) ↑
π(y_l | x) ↓

but if we do it directly, the LLM might deviate too far

5. Level 2: Reasoning Training

5.1 Why RL?

The earlier post-training process of ChatGPT-style model is:

1
2
3
4
5
6
7
8
9

Pretraining
     ↓
    SFT
     ↓
Preference Learning
     ↓
RLHF / DPO
     ↓
Instruction-following + Alignment

The LLM’s goal is to answer precisely, more like human. But it cannot resolve a problem it never trained.

Tradition LLM:

1
2
3

Question
   ↓
Answer

Reasoning LLM:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Question
   ↓
Think
   ↓
Think
   ↓
Try
   ↓
Check
   ↓
Correct
   ↓
Think
   ↓
Answer

The key is not “introduce CoT” but “regard reasoning itself as the optimization objective”.

Why Reasoning Training needs RL?

Tradition SFT:

1
2
3
4
5

Question
   ↓
Human-written reasoning
   ↓
SFT

SFT will tell LLM: You need to imitate this reasoning path.

But in some complex problems:

1
2
3
4
5
6
7
8

Question
   │
   ├── Reasoning path A → Correct
   ├── Reasoning path B → Correct
   ├── Reasoning path C → Wrong
   ├── Reasoning path D → Wrong
   ├── Reasoning path E → Correct
   └── ...

Human cannot write every correct path, so we need LLM itself to inference the path and judge the result.

We have two RL ways to improve the reasoning ability:

• RLHF (Reinforcement Learning from Human Feedback)
• RLVR (Reinforcement Learning with Verifiable Reward)

5.2 RLHF

e.g.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

Q: Prove a certain mathematical conclusion.

Model A:

Step 1: ……
Step 2: ……
Step 3: ……
Answer: Correct

Model B：

Step 1: ……
Step 2: ……
Step 3: ……
Answer: Correct

Then we have a preference data:

1

A > B

Then we can train a reward model.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

Human Preference Data
          │
          ▼
    Reward Model
          │
          ▼
  Reasoning Reward
          │
          ▼
    RL Algorithm
          │
    ┌─────┴─────┐
    │           │
   PPO         GRPO
    │           │
    └─────┬─────┘
          ▼
    Reasoning LLM

Reward may includes:

1
2
3
4
5
6
7
8

Reward = 
Correctness + 
Reasoning Quality + 
Clarity + 
Usefulness + 
Format + 
Safety + 
..

5.3 RLVR

RLVR (Reinforcement Learning with Verifiable Rewards) means that use “Verifiable Rewards” for reinforcement learning.

RLVR are suitable for Reasoning Tasks because many reasoning tasks’ final result is verifiable:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

Mathematics:
Is the answer correct?

Code:
Has the test been passed?

Theorem proof:
Is the proof valid?

Chess:
Did you win in the end?

Logic question:
Does the answer satisfy the constraints?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

         Problem
            │
            ▼
        Reasoning LLM
            │
    ┌───────┴────────┐
    │                │
Reasoning         Answer
    │                │
    └───────┬────────┘
            ▼
        Verifier
            │
     ┌──────┴──────┐
     │             │
  Correct        Wrong
     │             │
  Reward +1      Reward 0

LLM can try different pathes:

1
2
3
4

Path A → Wrong → 0
Path B → Wrong → 0
Path C → Correct → 1
Path D → Correct → 1

One problem:

The RLVR only verifies the final the result. The intermediate reasoning process may be incorrect.

1
2
3
4
5
6
7
8

Reasoning:
❌
❌
❌
❌

Final Answer：
✓

Then the verifier will give it reward:

1

Reward = 1

Outcome Verification ≠ Process Verification !

5.4 Combination of RLHF and RLVR

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

           Reasoning LLM
                │
                ▼
        Generate Reasoning
                │
     ┌──────────┴──────────┐
     │                     │
     ▼                     ▼
 Verifier              Reward Model
     │                     │
     ▼                     ▼
Objective Reward      Human Preference
     │                     │
     └──────────┬──────────┘
                ▼
           Combined Reward
                │
                ▼
          RL Algorithm
                │
                ▼
           Update LLM

e.g.

Reward = Reward_verifier + λReward_human

5.5 PPO & GRPO

RLHF and RLVR are two algorithms for calculating rewards.

PPO and GRPO are algorithms for updating LLM.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

        Reasoning Training
               │
      ┌────────┴────────┐
      │                 │
    RLHF               RLVR
      │                 │
Reward Model        Verifier
      │                 │
      └────────┬────────┘
               │
           Reward
               │
  ┌────────────┴────────────┐
  │                         │
 PPO                       GRPO
  │                         │
  └────────────┬────────────┘
               │
               ▼
         Update LLM

6. Agentic Fine-Tuning

There are many popular training methods in Agent area, including:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

             Agent Model Post-Training
                      │
     ┌────────────────┼─────────────────┐
     │                │                 │
     ▼                ▼                 ▼
① Imitation       ② Preference       ③ RL
  Learning          Optimization       │
     │                │                │
     │                ├── DPO          ├── Offline RL
     │                └── Step-DPO     └── Online Agentic RL
     │                                     │
     │                              ┌──────┼──────┐
     │                              │      │      │
     │                             RLHF   RLVR  Process RL
     │                              │      │      │
     │                              └──┬───┴──────┘
     │                                 │
     │                          PPO / GRPO / PMD
     │
     ├── Trajectory SFT
     ├── Critical-Step SFT
     ├── Synthetic-Trajectory SFT
     └── Distillation

6.1 Imitation Learning (SFT)

Imitation Learning is basically SFT.

ℒ_SFT = −∑_tlog P_θ(tokent ∣ token < t)

6.1.1 Trajectory SFT

Core Process:

1
2
3
4
5
6
7

Expert Agent Trajectory
        ↓
Teacher Forcing
        ↓
Cross Entropy
        ↓
LLM

Trajectory:

1
2
3
4
5
6
7

task
→ thought
→ action
→ observation
→ action
→ ...
→ final answer

• SWE-Gym (ICML 2025), Rejection Sampling / Self-Improvement
• SWE-Lego (arXiv:2601.01426, 2026)

6.1.2 Critical-Step SFT

A trajectory includes many steps, but not all steps are necessary. There’s one way only to SFT the critical steps in trajectories.

• ATLaS — (Findings ACL 2025), only train critical steps

6.1.3 Synthetic Trajectory

One problem of SFT is that high-quality training data is few, so there’s a research direction is to compose Synthetic Trajectory.

• SWE-smith, Synthetic Trajectory

6.1.4 Distillation

Distillation is to extract ==reasoning trajectories== form different teachers and distill into a student model.

1
2
3
4
5
6
7
8
9
10
11
12
13

Teacher A
  │
  ├── Tool-based reasoning
  │
Teacher B
  │
  └── Text reasoning
       │
       ▼
     Distill
       │
       ▼
Student Agent

• Agentic-R1 (EMNLP 2025)

6.2 Preference Learning (mainly DPO)

DPO is prevail in LLM post-training area, but in Agent area it’s not as popular as SFT and Online RL.

SFT makes LLM realize what is right answer. Preference Learning just tells LLM: A is better than B.

1
2
3
4

Task

Trajectory A -> good
Trajectory B -> bad

Train:

1
2

πθ(A|x) ↑
πθ(B|x) ↓

The most popular Preference Learning method in Agent area is DPO (Direct Preference Optimization).

The basic DPO process:

1
2
3
4
5
6
7

Prompt
 ├── Chosen
 └── Rejected
       ↓
      DPO
       ↓
      LLM

It doesn’t need traditional reward model…

The traditional DPO compare different trajectories. Nowadays the tendency is to research which certain step makes trajectory better.

1
2
3
4
5
6
7

Trajectory-level
       ↓
Step-level
       ↓
Action-group level
       ↓
Critical-step preference

• HPL (ICLR 2026): Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents
• Online DPO (ICLR 2025)
• SDPO (ACL 2025)

6.3 Online Agentic RL

Reinforcement Learning categories can be divided into 2 kinds: Online and Offline.

The tendency is to use Online one.

Online RL requires Agent to generate real trajectories, and calculate reward and update policy.

• RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning (ICML 2025)
• WebAgent-R1 (EMNLP 2025)
• DeepResearch (EMNLP 2025)

6.3.1 Reward Design

The reward is a criterion to judge how good a trajectory is.

There are many ways to calculate reward:

1
2
3
4
5
6
7

            Reward
              │
  ┌───────────┼────────────┐
  │           │            │
Human      Verifier    Process Reward
  │           │            │
 RLHF        RLVR       PRM/PR

RLVR (Reinforcement Learning Verifiable Reward) is an important type, because many engineering problems are verifiable.

• SWE-RL (NeurIPS 2025)
• WebAgent-R1 (EMNLP 2025)

==Process Reward== is to solve a problem:

1
2
3
4
5
6
7
8
9

Step 1
Step 2
Step 3
...
Step 50
 ↓
Success
 ↓
Reward = 1

The final answer is correct, but LLM doesn’t know if step 3 is important.

1
2
3
4
5

Outcome Reward
       ↓
Process Reward
       ↓
Step-level Credit Assignment

• Rewarding Progress (ICLR 2025)
• PURE (NeurIPS 2025)
• ReasonFlux-PRM (NeurIPS 2025)

6.3.2 RL Optimization Algorithm

After calculating the reward of a trajectory, we need to update the weights of LLM.

1
2
3
4
5
6
7
8
9
10
11
12

         Reward
           │
           ▼
     Policy Optimization
           │
 ┌─────────┼─────────┐
 │         │         │
PPO       GRPO      PMD
 │         │         │
 └─────────┼─────────┘
           ▼
      Update LLM

6.3.2.1 GRPO

Basic GRPO process:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

Question x
   │
   ├── rollout 1 → R1
   ├── rollout 2 → R2
   ├── rollout 3 → R3
   ├── ...
   └── rollout G → RG
                 │
                 ▼
      Group Relative Advantage
                 │
                 ▼
                GRPO
                 │
                 ▼
                LLM

The GRPO doesn’t need a value/critic model which is necessary for PPO.

6.3.2.2 PPO

• RLEF (ICML 2025 Spotlight)
• Open-Reasoner-Zero (NeurIPS 2025)
• SPPO (ACL 2026)
• Memory-R1 (ACL 2026)

6.3.2.3 PMD

PMD (Policy Mirror Descent) is another technology roadmap used by KIMI. But is not as popular as GRPO/PPO.

6.4 Offline Agentic RL

• OREO (ACL 2025): Offline Reinforcement Learning for LLM Multi-step Reasoning
• Reward-Weighted Fine-Tuning (NeurIPS 2025): Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization

todo list

too lazy to edit :)

Paper Summary of "SWE-smith: Scaling Data for Software Engineering Agents".

This work describes the panorama of Agent area. It is suggested to review this article frequently.

This article describes the panorama of Agent area. It is suggested to review this article frequently.

This article records notes on encoding and format in Computer Science.

Paper Summary of "SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution".

Notes on Mathematics

Paper Summary of "Sample-Efficient Learning from Agent Experience".