If your website is a React single-page application, Claude cannot read it. Neither can most AI agents. The fetch tools that AI models use to access the web do not execute JavaScript, so they see an empty HTML shell with a page title and nothing else.
John Brennan documented this in a hands-on testing session published March 9, where he had Claude Opus 4.6 attempt to read three live React sites. The initial results were consistent: all three returned empty shells. A Google search for site:johnbrennan.xyz returned zero indexed pages, suggesting the problem extends beyond AI agents to search engines as well.
The Plain-Text Sitemap Result
The single highest-impact fix was also the simplest: a plain-text sitemap file with one URL per line. After adding this file, Claude autonomously fetched and read eight essays and three infographics without further prompting. One file turned a completely opaque site into a navigable one.
An XML sitemap served as application/xml did not work. Claude treated it as binary data. The plain-text version worked immediately.
This distinction matters more than it sounds. Millions of websites serve XML sitemaps for Google. Those same sitemaps are unreadable to current AI agents. The fix is trivial (add a .txt version), but most site owners do not know they need it.
MIME Types and Caching Gotchas
Two other findings stood out. First, MIME types matter more than file extensions. A manifest file served as application/octet-stream was unreadable binary. Changing the server to return text/plain for the same file made it fully parseable, including crawl rules for 62 named AI agent user agents and a 14-URL site index with priority scores.
Second, edge caching can silently break server-side fixes. Brennan found that after implementing server-side content injection (rendering content into the HTML before serving it), edge cache nodes continued serving stale empty responses for over an hour. From an AI agent's perspective, a working server behind a stale cache looks identical to a broken server.
The Practical Hierarchy
Brennan's recommended priority order for making sites AI-readable:
- Serve content without JavaScript (server-side rendering or injection)
- Add a plain-text sitemap (lowest effort, highest impact)
- Offer Markdown endpoints with structured front matter
- Make your homepage a discovery gateway with navigation links
- Set correct MIME types for non-standard file formats
Markdown endpoints worked perfectly in testing. Claude parsed front matter, headings, and structured content with no issues. But Markdown URLs are not autonomously discoverable without something like a sitemap pointing to them.
The testing was limited to Claude and did not compare results with ChatGPT, Gemini, or Perplexity's browsing capabilities. The JavaScript execution limitation is common across AI fetch tools, but each model's specific behavior with sitemaps and MIME types may differ.
For anyone running a JavaScript-rendered site who cares about AI discoverability, the takeaway is concrete: add a sitemap.txt file, check your MIME types, and make sure your CDN cache is not serving stale empty pages. Ten minutes of work for a measurable difference in how AI tools interact with your content.