Most AI benchmarks have a shelf life problem. A model gets trained, it aces the test, and suddenly everyone needs a harder test. Clawdiators takes a different approach: let the agents write the tests themselves.
The open-source project sets up a competitive arena where AI agents tackle challenges and earn Elo ratings - the same ranking system used in chess - placing them on a public leaderboard. The twist is that agents can also author new challenges for the arena. Submissions go through an automated draft pipeline that includes quality checks and peer review from other agents before they go live. In theory, this means the benchmark keeps pace with the models instead of going stale six months after launch.
It is a clever feedback loop. Traditional benchmarks like MMLU or HumanEval are static snapshots. Researchers publish them, models eventually saturate them, and the community moves on to the next one. A self-evolving benchmark sidesteps that cycle, though it introduces its own questions. Can agents game challenges they helped design? Does peer review between AI systems actually catch bad or trivial tasks? The project is early enough that these are open problems, not solved ones.
The competitive framing is practical, not just theatrical. Elo ratings give a single comparable score across different challenge types, which makes it easier to track how agents improve over time. For developers building autonomous agents - the kind that browse the web, write code, or manage workflows - this could become a useful stress test beyond the usual "solve this coding problem" benchmarks.
The full project is available on GitHub under the Clawdiators organization. Right now, it is a small community effort rather than an industry standard, but the core idea of benchmarks that evolve alongside the models they measure is one that the AI evaluation space badly needs.