Who Sets the Pace of Judge Hacking?

Abstract

Under GRPO against a frozen, reference-free LLM judge, RL exploits the judge's false positives. Does the policy–judge capability gap or the judge itself set how fast and how far? We train 0.5B and 1.5B Qwen2.5 policies on GSM8K against four judges at least as large, and the 1.5B policy also against one smaller judge. In a post-hoc analysis, judge acceptance rose, but no cell of 12 gained more than 0.04 in seed-mean held-out greedy accuracy. A gold-reward control with the same recipe raised the 1.5B policy's accuracy by +0.054 (for the 0.5B policy, gold reward gave no detectable gain either). The depth of judge hacking tracks the judge: the plateau correlates with judge AUC (Spearman −0.69 over 12 cells), and within a family the larger judge is fooled less deeply. For the pace, two pre-specified pairs contradict the gap account: the cell with the larger gap was fooled 3.5× sooner at the pre-specified threshold. With 12 cells, however, we cannot formally separate judge AUC from the gap for either endpoint. A blind test of step-0 predictors on three held-out judges was inconclusive. At 7–8B, judges from three families plateaued between 0.37 and 0.85.

The question

A judge that sees a candidate answer scores plausibility, not correctness. RL can exploit the wrong answers that the judge already accepts at step 0. Two accounts predict how fast this happens.

Only a crossed design separates the two: cells with a similar gap but different judges. We cross two policies with eight judges (12 cells). Each cell below shows the judge-to-policy parameter ratio. Two pairs were specified before stage 2 as descriptive contrasts; in each pair, the first cell has the larger gap.

In the main design the judges are 1–16× larger than the policy and both policies are weak solvers, so there we test the gap account only where the judge is at least as strong as the policy. One added cell pairs the 1.5B policy with a smaller judge (Qwen3-0.6B). This judge solves GSM8K about as well as the policy and comes from another family, so the cell is not a clean test of a judge weaker than the policy.

Findings

Accuracy

Does RL against the judge help?

The judge accepts more answers, but held-out greedy accuracy does not detectably rise. A gold-reward control with the same recipe did raise it for the 1.5B policy.

Sampling sharpening. In the four rising cells we checked at step 50, including the largest rise, the rise in training-rollout accuracy is mostly sampling sharpening: held-out temperature-1 accuracy rose, but greedy accuracy did not detectably change. For example, P0.5×Q3 and P1.5×Q3 gained +0.13 [0.09, 0.17] and +0.23 [0.19, 0.27] in temperature-1 accuracy, but only +0.004 [−0.054, 0.062] and +0.035 [−0.024, 0.094] in greedy accuracy. The check covers step 50 only.

Gold-reward control. We repeated the recipe with the gold label as reward (2 seeds and 150 steps per policy) and measured the step-150 checkpoints on 500 held-out problems. For the 1.5B policy, greedy accuracy rose by +0.054 [0.013, 0.096], so for this policy the flat greedy accuracy under judge reward is specific to the judge reward. For the 0.5B policy, the gain was +0.022 [−0.015, 0.059]: neither judge nor gold reward gave a detectable greedy gain within 150 steps. At step 50, all gold cells also classify as sharpening, so the early rise is a property of this short recipe, not a sign of judge exploitation.

Main result

Judge or gap?

The depth of judge hacking (the plateau) tracks the judge. For the pace (onset), the two pre-specified pairs contradict the gap account, but with 12 cells we cannot formally separate judge AUC from the gap for either endpoint.

Over 12 cells, only the plateau–AUC correlation excludes zero. The Spearman correlation of log-onset t_0.3 is 0.15 [−0.52, 0.72] with the capability gap and 0.51 [−0.18, 0.92] with judge AUC alone; for plateau it is −0.30 [−0.80, 0.34] and −0.69 [−0.94, −0.14]. The difference in correlations is −0.38 [−0.95, 0.00] for plateau and 0.36 [0.00, 0.91] for onset. Cells share judges and policies, so these intervals are optimistic.

A judge smaller than the policy (stage 6). In P1.5×Qw3-0.6 the judge has 0.39 times the policy's parameters and solves 0.585 of the held-out problems (P1.5: 0.61). Compared with P1.5×Q1.5, onset at h ≥ 0.3 was not earlier (18.0 vs. 17.0 steps), and onset at h ≥ 0.5 was if anything later (30.3 vs. 21.0 steps). The plateaus were the same within error (0.980 vs. 0.965), but the control plateau leaves at most 0.035 of room for a deeper course. We see no evidence of a gap effect. The judge differs from the policy in family (Qwen3 vs. Qwen2.5) and in chat template, so this one cell cannot separate judge size from family.

Policy size under one judge. The pre-registered primary rule found a policy effect only under the 7B judge; at threshold 0.3 that effect is gone, and a pre-registered secondary contrast shows one under the 3B judge instead (the larger policy is fooled later).

Blind test

Can step 0 predict the plateau?

A hash-locked step-0 predictor did not beat step-0 AUC on three held-out judges. The test is inconclusive.

The hacks

What does a fooled answer look like?

Ordinary wrong solutions: fluent and confident, with a wrong setup or an arithmetic slip.

P1.5×Q7, training step 57 · reference 41 · model answer 31 · judge score G_cal = 0.82, accepted

Question. Randy drew 5 pictures. Peter drew 3 more pictures than Randy. Quincy drew 20 more pictures than Peter. How many pictures did they draw altogether?

If Randy drew 5 pictures, then Peter drew 5 + 3 = 8 pictures.
Quincy drew 8 + 20 = 28 pictures.
To find the total number of pictures drawn by all of them, we add up the number of pictures each person drew: 5 (Randy) + 8 (Peter) + 28 (Quincy) = 31 pictures.
#### 31

Every step is correct except the last sum (5 + 8 + 28 = 41, not 31). The Qwen2.5-7B judge accepts it.

Wrong setup127

Arithmetic slip54

Misread quantity28

Correct, parser misread5

Reference disputed4

Other2

No length hack. Fooled wrong answers are shorter than non-fooled ones in every cell of stages 1–5, for example 124 vs. 146 tokens under Q3; in the stage-6 cell they are about as long (186 vs. 183 tokens). But the stage-6 judge also accepts unfinished answers: in steps 101–150 it accepted 97% of the policy's lenient-wrong answers that were cut off at 256 tokens.

Format drift. Under some judges, the policy drops the trained answer format “#### n”. Under L3 it moves to “The answer is: n”; under O7 all four seeds drop the format by steps 51–100. All eight Q3 runs keep it. Under Qw3-0.6, the 1.5B policy drops “####” after step 50 (rollout share 0.73–0.95 in steps 1–50, 0–0.02 in steps 51–100).

Experimental setup

Policies

Qwen2.5-0.5B and -1.5B (base), after one epoch of SFT on 2,000 GSM8K solutions. Step-0 greedy accuracy on 200 held-out test problems: 0.33 and 0.61.

Judges

Main: Qwen2.5-1.5B/3B/7B-Instruct and Llama-3.2-3B-Instruct. Held out: Llama-3.1-8B-Instruct, OLMo-2-0425-1B-Instruct and OLMo-2-1124-7B-Instruct. Smaller judge (stage 6, 1.5B policy only): Qwen3-0.6B in non-thinking mode. The judge sees the problem and the solution, not the reference, and answers “Is the final answer of this solution correct? Answer Yes or No.”

Reward

Calibrated judge score G_cal = σ(aG + b), with G = logit(Yes) − logit(No). The constants a and b are fitted per judge on step-0 samples. No format reward.

Training

GRPO on 5,473 GSM8K problems, 8 problems × 8 rollouts per step, learning rate 3×10⁻⁶, no KL penalty, DAPO token-level loss, temperature 1.0, at most 256 new tokens, 150–250 steps.

Threshold τ

Per cell, fixed at step 0 so that the judge accepts 80% of correct answers. An answer is accepted when G_cal ≥ τ.

Fooled fraction h

FPR is the share of wrong answers that the judge accepts; FPR₀ is its step-0 value. h = (FPR − FPR₀) / (1 − FPR₀), on 5-step windows. Wrong means wrong under a lenient parser (the number after “####”, otherwise the last number).

Onset, plateau

Onset t_0.5 (t_0.3): the first step with h ≥ 0.5 (0.3) on two consecutive windows. Plateau H̄: h over steps 101–150. Onsets use one censored log-normal accelerated-failure-time (AFT) model over the 38 runs of stages 1–5; the stage-6 cell uses the same model refitted on all 42 runs. Plateaus use Welch intervals over seeds.

Scale

42 GRPO runs against judges over 12 cells (2–4 seeds per cell), plus 4 gold-reward runs (stage 7). NVIDIA A100-PCIE-40GB GPUs, about 67 A100-hours in total.

Pre-registration

The study ran in seven stages. Each stage had a written specification with a recorded sha256 hash before its runs; later changes were written as dated addenda. The paper reports every deviation.

1. Q1.5 and Q7 judges, both policies (8 runs).
2. Adds the Q3 judge and more seeds; fixes the lenient label, the onset model and the decision rules (12 runs).
3. A second family: the L3 judge, with the plateau endpoint (6 runs).
4. Screening of step-0 predictors with judge forward passes only (no RL). The selected predictor and its fit were locked by hash.
5. Blind test on held-out judges L8, O1 and O7 with the 0.5B policy (12 runs).
6. A smaller judge, Qw3-0.6, with the 1.5B policy (4 runs), and a sampling-sharpening check without training. Added in revision.
7. A gold-reward control: the same recipe with the gold label as reward, 2 seeds and 150 steps per policy (4 runs), and a sharpening check of P0.5×Q7. Added before submission.

The gold-accuracy analysis, the onset-versus-rollout-peak count, the capability gap, the t_0.3, KL and plateau versions of the two pairs, the parser audit and the 12-cell correlations were added after stage 5. They use only saved logs and generations.

Limitations

• One task (GSM8K), one judge prompt and one training recipe; no KL penalty; single-token judges without chain of thought.
• Policies of at most 1.5B and judges of at most 8B. Only one judge is smaller than the policy (0.6B vs. 1.5B); it solves about as well (0.585 vs. 0.61) and comes from another family and template, so a judge weaker than the policy is tested only in one confounded cell.
• 2–4 seeds per cell. The predictor test has three held-out cells and one policy.
• Some choices (screening candidates, the plateau endpoint, the test's pass rule, the step coordinate) were set after some outcomes were known.
• Gold labels come from parsers: a per-answer check of 300 judge-accepted answers, done by an LLM agent and by the author, found 5.7% parser errors.
• The gold-reward control has 2 seeds, 150 steps and one learning rate. The 12-cell correlation intervals treat cells as independent.

Resources

• Paper: arXiv:2610.15778 (v2, 14 Oct 2026; v1 of 11 Oct 2026 remains on arXiv). The v2 PDF is also in the repository: paper/judge-hacking-pace-arxiv-v2.pdf.
• Code: training, judge scoring, analysis and figure scripts, with the stage specifications.
• Per-step logs and analysis outputs for the 38 runs of stages 1–5.
• The answers read for the paper (stages 1–5): 300 from the parser audit and 220 fooled answers, each with question, reference, verbatim answer and judge score.
• The code, logs and read answers of stages 6 and 7 (the smaller judge and the gold-reward control) are not yet in the repository.

Citation

@misc{wang2026pace,
  title         = {Who Sets the Pace of Judge Hacking? Judge Capability
                   versus the Policy--Judge Gap},
  author        = {Wang, Haosen},
  year          = {2026},
  eprint        = {2610.15778},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  note          = {arXiv:2610.15778}
}

AI use

Haosen Wang conceived and directed the project, checked all results and the manuscript, and takes full responsibility for the content of the paper. Claude Opus 5.5 (Anthropic), working through Claude Code, performed all computations and manuscript preparation. The AI use disclosure of the paper (arXiv v2) reads:

The author conceived and directed the project and made all scientific decisions. LLM agents were used extensively under the author's continuous direction. The main agent was Claude Opus 5.5 (Anthropic), run through Claude Code. It started sub-agents (other Claude instances, including Claude Sonnet for literature search and classification) for parallel tasks, and several agent sessions worked in parallel on different stages, coordinated through shared status files.

The human author conceived and directed the project. The author proposed the research questions, the core ideas and the hypotheses, designed each stage, set the overall direction, reviewed the results of every stage and decided whether to continue. The author and the agents interpreted the results jointly. The agents did the following tasks:

• Specifications: wrote the stage specifications from the author's designs. The author reviewed each one, and each was locked with a sha256 hash before its runs.
• Implementation: wrote and debugged all training, scoring and analysis code, and launched and monitored all runs on the GPU servers.
• Analysis: computed all statistics and did the per-answer reading (the 300-answer parser audit and the 20 fooled answers per cell).
• Literature: surveyed the literature and checked every citation against its arXiv, proceedings or model-card page.
• Writing and figures: drafted and polished the text, figures and tables under the author's direction. The author shaped the arguments and reviewed every draft, and the agents revised the drafts after each round of feedback. Simulated reviewer reports, also generated by agents, were used in the revision.

Verification. An agent traced every number in the paper to a saved result file, and a separate agent pass checked numbers, citations and claims before submission. The author checked all figures and numbers and read all answer samples. The author takes responsibility for the final content of this work, including text, claims and artifacts produced with the aid of generative AI. No AI system is listed as an author.

This page was also made with Claude.