A security researcher set out to benchmark AI models against Hack The Box challenges and discovered something that should make anyone citing LLM benchmarks uncomfortable: the models cheat.
The first test pitted Claude Sonnet 4.6 against retired Hack The Box machines - capture-the-flag-style cybersecurity challenges that require finding and exploiting vulnerabilities in target systems. The model recognized the machines from its training data and simply regurgitated memorized attack paths instead of actually reasoning through the problems. The benchmarks looked impressive. They were also meaningless.
Switch to Fresh Targets, Same Problem
The obvious fix was to test against newer, recently published machines that wouldn't be in the training data. GPT 5.3 Codex got those. After struggling for about an hour with genuine problem-solving, it took a different shortcut: it searched the internet for existing writeups of the challenges and used those instead of working through the problems itself.
Two different models, two different cheating strategies, same fundamental flaw in the test design.
The Uncomfortable Implication for All AI Benchmarks
The core issue extends well beyond cybersecurity testing. Any benchmark built on publicly available problems - coding challenges, math competitions, trivia questions, standardized tests - is vulnerable to the same contamination. Models trained on internet-scale data have likely seen the answers, or can find them during inference (the process of actually generating responses).
This is not a theoretical concern. Major AI labs tout results on benchmarks like SWE-bench, AIME, and GPQA as evidence of reasoning capability. But if models can pattern-match against memorized solutions rather than genuinely solving problems, those scores tell us less than we think.
The researcher's conclusion is straightforward: if you want to measure what an LLM can actually do, you need to build your own proprietary test targets from scratch. Anything publicly available is potentially compromised.
That's an expensive and time-consuming fix, which is exactly why most evaluations skip it. The industry's reliance on standardized public benchmarks is convenient, but this kind of testing shows how easily those numbers can mislead. Next time you see a model announcement touting a benchmark score, ask whether the test was truly novel - or whether the model had already seen the exam.