LLM-as-judge uses a language model — or, for tasks with visual output, a vision-language model — to evaluate another model’s output, in place of a human rater or a hand-written rule-based checker. This scales evaluation to settings where human annotation is too slow or expensive to apply to every example, and where the target task (open-ended text, a multi-step agent trajectory) is too unstructured for a simple programmatic check to grade reliably.
The approach only shifts the evaluation problem rather than solving it outright: the judge model’s own reliability, biases, and blind spots become part of whatever it is used to measure or train against. A judge with a systematic bias — for instance, a tendency to rate a plausible-looking but ultimately unsuccessful attempt as successful — will pass that bias into any downstream use, whether that is a leaderboard, a filtered training dataset, or a reward signal for reinforcement learning.
Related papers
- Sun, Cheng, Wang, et al., “OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models”, arXiv:2607.28609 (2026) — benchmarks vision-language-model judges on computer-use agent trajectories against human-verified ground truth, finding a systematic leniency bias.