Can LLMs Engineer Their Own Agent Harness? ByteDance Seed's HarnessDev Says Only 34 of 64 Changes Generalize

HarnessDev: LLMs Build Runnable Agent Harnesses; Dead Code, Portability, And Cost Limit Real Use » Saipien

Can LLMs engineer their own harness? HarnessDev finds only 34 of 64 changes moved feedback and held‑out scores together One creator LLM added 17, 111 net lines of code to its harnesses. Another added just 1, 006 lines and still led a terminal benchmark. That gap (more code does not equal more robustness) is the […]

Can LLMs Engineer Their Own Agent Harness? ByteDance Seed's HarnessDev Says Only 34 of 64 Changes Generalize

ByteDance Seed, SUTD, Georgia Tech, M-A-P, and TokenWave.AI introduce HarnessDev, a benchmark that scores the runnable harness a model builds rather than the answer it returns. Starting from a seed that scores 0, 6 creator LLMs construct harnesses across 5 benchmarks and 2,207 tasks, then evolve them from execution feedback. Self-built harnesses match human references on writing and ML experimentation but trail on code and search, and only 34 of…