Generic LLM leaderboards tell you which model is best at standardized benchmarks. They tell you almost nothing about which model is best for your work.
Auto LLM Ranker takes a different approach. You describe a task in plain English - say, "summarize legal contracts" or "generate SQL from natural language" - and the tool builds a custom test suite around that description. It then discovers candidate models through OpenRouter (a service that provides access to dozens of LLMs through a single API), runs them all in parallel, and has a judge model score each response across five dimensions: accuracy, hallucination, grounding, tool-calling ability, and clarity.
The output is a ranked top three with average latency per model and a task-specific system prompt you can use immediately.
This solves a real problem. Anyone who has tried to pick between Claude, GPT-4o, Gemini, Llama, and the dozens of other available models knows the current process is mostly guesswork. You read benchmark scores that test things you will never ask an LLM to do, maybe try two or three models manually, and pick whatever felt best. Auto LLM Ranker automates that trial process and adds structure to the comparison.
The tool is open source and available on GitHub. It requires an OpenRouter API key, which means you are paying per-token for each model it tests - but since it is running targeted evaluations rather than exhaustive benchmarks, costs should stay reasonable for most use cases.
One limitation: the quality of the rankings depends heavily on the judge model's own capabilities. If the judge LLM has blind spots in a domain, those blind spots will carry into the scores. Still, structured evaluation beats gut feeling, and having a repeatable process means you can re-run the comparison whenever new models drop.