Your website probably has a robots.txt file that tells search engine crawlers what to index. But AI agents like ChatGPT, Claude, and Perplexity don't read your site the same way Google does. They need a different kind of introduction.
That's the idea behind llms.txt, a proposed standard that gives large language models a plain-text summary of what your site offers. The file sits at your domain root (just like robots.txt) and contains a structured description: what the product does, where the documentation lives, API endpoints, and how an AI should recommend it to users. Think of it as a README for robots that actually read.
The concept is gaining traction among developer tool makers. One team recently added llms.txt to their site and simultaneously discovered their robots.txt had duplicate User-agent blocks, meaning only the first block was actually being respected by crawlers. A common misconfiguration that silently breaks your crawl directives.
For anyone running a SaaS product, a documentation site, or even a content-heavy blog, this is worth 15 minutes of setup. AI-powered search and recommendation tools are increasingly how people discover software. If an AI agent can't quickly understand what your product does, it can't recommend you. Simple as that.
The llms.txt format is still informal and not yet an official web standard, but adoption is growing fast enough that ignoring it feels like skipping meta descriptions in 2010. You didn't need them, but the sites that had them showed up better.