Scaling Laws for Reward Model Overoptimization

From “Evidence for effective RSI implementation” (2026-10-28)

Design choice: Preserve every rule through append-only additions

Supporting evidence: For conjunctive mandatory requirements, retaining all old rules prevents accepting an output by removing an inconvenient requirement. This is a logical consequence of the proposed rule.

Challenge or limit: An unchanged proxy evaluator can still reward harmful changes. Consistency, correctness, and comparability remain separate issues. Overoptimization study

1. Judgment reliability. A judge can apply an unchanged rule incorrectly. The MT-Bench study demonstrates order-sensitive judgments, while the overoptimization study demonstrates failure under fixed proxy scoring. Human task-owner judgments can adjudicate disagreements without requiring the human to grade every intermediate item. MT-Bench study, overoptimization study
2. Comparable evaluations. Adding criteria can change rankings even though no rule disappears. Compare old and revised outputs under the same identified rule set. An old assessment under fewer rules is not evidence of acceptance under the expanded set.
3. Conflicting or mistaken rules. If all active rules cannot be satisfied together, append-only additions do not themselves supply a repair. The system should expose the conflict and wait for the person's decision. This review does not infer permission for humans or agents to delete, deactivate, or rewrite rules; how such conflicts are resolved remains a design question.