HarnessPro: Recursive Harness Self-Improvement with Pareto Optimization for Performance-Cost Trade-offs

From “What HarnessPro contributes to RSI design” (2026-10-28)

This review covers paper v2, revised 20 October 2026 and marked under review, and repository commit 6fd2d4d. Methods, results, test-authoring instructions, example diagnoses and plans, and relevant runtime appendices were read. Tables 2–3 and Figure 2 were also checked on rendered PDF pages. Repository inspection was static; no model calls, installations, containers, or benchmark runs were performed. Results below are the authors' reports. The source manifest records exact files, hashes and reading scope.

How the mechanism works

The object being improved is a harness: instructions, tools, skills, context handling, hooks, and verification policies surrounding a foundation model. The outer improvement procedure specifies four agent roles; the paper does not demonstrate that this outer procedure itself evolves.

1. Split tasks into training, validation and held-out testing. Select ten training tasks using difficulty and diversity estimated from initial execution traces. Keep that training set fixed.
2. Generate executable tests for training and validation tasks from public requirements and the initial harness's execution. Freeze these tests before improving harnesses. Official tests and solutions are unavailable during optimization.
3. Run each retained harness three times on each training task. Diagnose failures and unnecessary expense, attaching trace paths and event ranges to findings.
4. Propose three plans per parent, emphasizing cost, balance, or performance. Implement each in a separate workspace and preserve its version. The planner can inspect peer implementations and diagnoses; validation and test tasks are hidden from it.
5. Evaluate parents and new candidates once per validation task. Reject candidates below the initial harness's validation pass rate and remove candidates dominated on both pass rate and cost. From the remaining candidates, select up to three representatives and repeat.

Selection minimizes a normalized squared distance from the best observed performance and cost. Performance/cost weights are 1:2, 1:1 and 2:1 for lite, medium and heavy. These are preferences among observed candidates, not a proof of finding the global Pareto frontier. The three selections can coincide. Evolution ends after three epochs for Terminal-Bench 2 and SWE-Bench Pro and four for ProgramBench. Paper §§3–5.1, Algorithm 1, Appendix A

One diagnosis illustrates the mechanism better than the architecture alone. For a Caffe/CIFAR-10 task, all three initial runs stopped before completing training. The diagnosis separates observed out-of-memory build failures from a stalled download and a possible later OpenCV dependency problem. The last is explicitly a predicted risk, not an observed failure. For a successful mailing-list task, diagnoses instead identify broad configuration dumps and repeated verification that added expense. Both successful and unsuccessful results can therefore inform execution improvements. These are representative, lightly edited examples, not independent evidence that every diagnosis was correct. Appendix C.1

What the quantitative evidence establishes

These Table 2 results use GPT-5.5 with Codex. Cost is mean estimated token cost per test task at the paper's fixed prices.

Benchmark; held-out tasks	RHO result; cost	HarnessPro-heavy result; cost	Difference from RHO
Terminal-Bench 2; 59	76.3%; $1.09	80.8% ± 1.0; $0.72 ± 0.05	+4.5 percentage points; 33.9% cheaper
SWE-Bench Pro; 100	80.0%; $2.68	80.7% ± 0.6; $1.97 ± 0.02	+0.7 points; 26.5% cheaper
ProgramBench; 100	68.8%; $3.59	69.6% ± 1.03; $2.95 ± 0.13	+0.8 points; 17.8% cheaper

Terminal-Bench 2 and SWE-Bench Pro require all official tests for a task to pass. ProgramBench reports the macro-average of within-task test pass fractions; its percentage is not the percentage of completely solved tasks. HarnessPro's reported deviations are sample standard deviations over three test runs for the first two benchmarks and two for ProgramBench. They do not establish uncertainty over independently repeated improvement searches. Baseline rows do not give comparable deviations. Paper §5.1 and Table 2

The abstract's +2.0-point and 26.1% figures are unweighted averages across these three comparisons. They are not a pooled probability of complete task success, and deployment cost excludes the earlier search for the harness.

The recursive baseline matters. RHO+Iteration uses the same number of epochs, although equal epochs do not establish equal total search expenditure. Against that baseline, heavy improves performance on all three benchmarks, but SWE-Bench Pro costs $1.97 versus $1.90, about 3.7% more. Therefore the strongest claim is an improved set of performance/cost choices, rather than universal dominance over every comparator. Table 2

The evaluator evidence is especially relevant:

Preliminary check pooled across two training sets	Direct LLM judge	Generated executable tests
Correctly detects an officially failing artifact	5 of 12	10 of 12
Incorrectly rejects an officially passing artifact	0 of 8	1 of 8

The 41.7% → 83.3% failure-recall result is encouraging, but it contains only twenty cases. Generated tests still accept two failing artifacts and reject one passing artifact. This is evidence for a better proxy in this check, not a solved evaluation problem. Figure 2

In Table 3, removing test synthesis reduces SWE-Bench Pro medium performance from 79.0% to 69.0%; removing the planner reduces it to 73.0%. Simply adding budget hooks to RHO reduces cost from $2.68 to $1.75 but reduces performance from 80.0% to 75.0%. A mechanism that helps in one complete harness may harm another. The ablations provide useful component evidence, with limited reported variation and search-budget accounting. Table 3

What the released code lets us inspect

The release contains nine GPT-5.5 harnesses, evaluation runners, task-ID manifests, runtime adapters, stores, accounting code and bundled evaluators. It does not contain a runnable implementation of the complete analyzer/planner/editor/test-synthesis/selection evolution loop in Algorithm 1. There are inherited optimization and diagnosis interfaces in protocols.py, but the supplied orchestrators implement solving; interfaces are not an implemented optimizer. The release also excludes the evolved GLM-5.2 harness snapshots, benchmark data and required services/images. Evaluating the released artifacts is more reproducible than recreating their discovery. Release scope, orchestrators, interfaces

Three concrete implementation details are useful:

• Preserve recoverable evidence. The Terminal-Bench medium hook saves long raw results and supplies a bounded head/tail view with a file path. This reduces recurring context without making the summary the only surviving evidence. Harness snapshots are identified from file-content hashes; trajectory stores refuse to overwrite an existing trajectory ID. These are inspectable versioning mechanisms, not proof that all earlier behavioral rules remain active. Hook, harness store, trajectory store
• Distinguish guidance from enforced behavior. Terminal-Bench medium's routing text describes standard stopping near 38 calls, while its executable policy sets 44. The hook denies further calls when its counter exceeds the selected limit. A runtime decision therefore needs the actual selected policy version; reading prose alone is insufficient. The paper prints this inherited-text discrepancy too, so it is not evidence of a paper/code mismatch. Rubric, policy
• Observe the overhead too. The Terminal-Bench runner persists profile-classification trajectories and aggregates usage across stored trajectories, with breakdowns by kind and stage. This supports accounting for routing as execution work. It does not provide the missing total cost of harness evolution. ProgramBench uses fixed profiles rather than the same task classifier, so task-adaptive routing is not a demonstrated requirement for every benchmark. Runner, accounting, ProgramBench policy