Do You Need an llms.txt File? What Google Actually Says
Key Takeaways
- llms.txt is a proposed markdown file that tells AI models which pages on your site to read.
- Google has said plainly that llms.txt is not needed to appear in its AI Search features.
- A Chrome Lighthouse audit checks for the file, but Google confirms that check is not a search signal.
- Independent data shows almost no AI crawlers actually request the file in practice.
- It still has one real use case: developer documentation for AI coding assistants.
- For most Perth businesses, your effort is better spent on clear, citable content.
What Is llms.txt
If you have spent any time in SEO forums lately, you have probably seen people asking whether they need an llms.txt file. It has become one of the most debated topics in AI search, so it is worth understanding before you spend time on it.
In simple terms, llms.txt is a markdown file placed at the root of your website. The idea, proposed by Jeremy Howard of Answer.AI back in September 2024, is to give large language models a clean, curated list of the pages and content you most want them to read. Think of it as a friendly map for AI, sitting alongside files like robots.txt and your XML sitemap.
The theory is appealing. AI tools strain to parse messy HTML, so a tidy markdown summary should make your content easier to ingest and, in theory, easier to cite. The reality, as you will see, is more complicated.
| Detail | Summary |
|---|---|
| What it is | A plain markdown file that lists the pages you want AI models to read |
| Proposed by | Jeremy Howard, co-founder of Answer.AI, in September 2024 |
| Where it lives | At the root of your domain, similar to robots.txt |
| Official status | A community proposal, not an adopted standard or a Google requirement |
What Google Actually Says About llms.txt
Here is where the myth meets the facts. Google has addressed llms.txt directly, and the message is consistent.
Google’s own guidance on optimising for generative AI features includes a mythbusting note stating that machine readable files like llms.txt are not needed to appear in AI Search. In other words, you do not need an llms.txt file to show up in AI Overviews or AI Mode.
Google Search Advocate John Mueller went further. He explained that llms.txt is not done for search and described it as a temporary crutch, perhaps to save some tokens for AI coding tools reading developer documentation. He also pointed out a deeper trust problem: an llms.txt file lets a site owner claim what their content is about, and that claim may not match the actual page. Search engines prefer to read the real thing.
Earlier, in July 2025, Google’s Gary Illyes confirmed Google does not support llms.txt and is not planning to, and as of 2026 that position has not changed. So if your goal is ranking in Google or being pulled into its AI answers, the file does nothing. If you want a grounding in how AI visibility really works, our guide on generative engine optimisation for WordPress sites is a better starting point.
| Source | Position on llms.txt |
|---|---|
| Google guidance | Not needed to appear in generative AI Search features |
| John Mueller | Not done for search; a token saving crutch for AI coding tools |
| Gary Illyes | Google does not support it and has no plans to |
| OpenAI | Recommends robots.txt for crawler control, not llms.txt |
But Google Built a Checker for It, So Why Ignore It
This is the part that confuses people, and it is a fair objection. If Google really does not care about llms.txt, why did Google add a check for the file to Chrome’s Lighthouse audit tool in 2026? At first glance that looks like Google quietly endorsing it.
The answer is that Google is a big company with many products, and they do not all speak with one voice. Lighthouse is a developer tool that audits technical best practices for sites and apps. A check living there is a nod to the developer documentation use case, not a search ranking signal.
Mueller addressed this contradiction head on, repeating that llms.txt is not done for search. So the Lighthouse audit and the Search guidance are not actually in conflict once you separate the two audiences. The takeaway stands: a green tick in Lighthouse does not mean Google Search, or its AI features, will treat your llms.txt file as a ranking input.
Why Almost Nothing Reads It Yet
The strongest argument against rushing to add an llms.txt file is not opinion. It is data.
In a mid 2026 study, Ahrefs analysed around 137,000 sites and found that 97 percent of llms.txt files were never requested at all. Of roughly 38,000 domains with a valid file, only about 1,100 received any traffic to it, and most of those requests came from non AI bots like SEO audit tools rather than from AI assistants. The major crawlers from Anthropic, Google and Perplexity effectively ignore it.
That is the crux of the myth. A file no one reads cannot improve your AI visibility, no matter how neatly it is written. Adoption among publishers exists, with names like Anthropic, Stripe and Vercel using one, but those are mostly developer focused platforms with a specific reason to.
When an llms.txt File Genuinely Helps
To be fair, llms.txt is not useless. It has one solid, real world use case.
If your audience is developers, and your site hosts extensive technical or API documentation, an llms.txt file can help AI coding assistants such as Cursor, Windsurf and Claude Code find and parse your docs efficiently. These IDE based agents and connected tools do fetch the file, and it can save them time and tokens. That is exactly the narrow purpose Mueller described.
Picture a Perth based SaaS or fintech with a public API and a documentation portal that developers query through AI coding tools. For a business like that, a tidy llms.txt pointing to the key reference pages is worth a look. The file itself is just markdown, structured something like this:
# Acme API
> Developer documentation for the Acme payments API.
## Docs
- [Quickstart](https://acme.com.au/docs/quickstart): set up in five minutes
- [Authentication](https://acme.com.au/docs/auth): API keys and tokens
- [Endpoints](https://acme.com.au/docs/endpoints): full referenceFor the vast majority of Australian businesses, a local trades company, a clinic, a law firm or an ecommerce store in WA, that use case simply does not apply. You are not serving API documentation to coding agents, so the file gives you nothing in return for the effort.
| Helps | Does not help |
|---|---|
| Developer docs read by AI coding tools | Ranking better in Google search |
| Large API references and technical libraries | Appearing in Google AI Overviews or AI Mode |
| Sites whose users work in Cursor or Claude Code | A local Perth service or ecommerce website |
Our Take at Dlinkers: Should You Add One
Across the AI visibility audits we run for Australian clients, adding an llms.txt file has never been the thing that moved citations. The brands that get pulled into AI answers win because their content is clear, well structured and genuinely trustworthy, not because of a file at the root of the domain. If you only have so many hours, that is where they should go.
There is a reasonable case on the other side, and it is worth stating fairly. Supporters argue that an llms.txt file is low cost insurance: it takes little time to create, it does no harm, and if the standard gains real traction you are an early mover rather than a latecomer. Standards in AI search have shifted fast before, so betting against one entirely carries some risk.
Our view is that the cost is not quite zero. A file you publish and forget can drift out of date, quietly claiming things about your site that are no longer true, which is the exact trust problem Mueller flagged. So unless you are in the developer documentation camp, we would skip it for now, keep an eye on how the major AI platforms behave, and revisit if any of them start genuinely reading it. Reasonable practitioners can disagree here, and that is fine.
What to Do Instead for AI Visibility
If llms.txt is not the lever, what is? The good news is that the work that earns AI citations is the same work that has always earned trust. Focus your energy here instead.
- Write clear, self contained answers. Lead each section with a direct response so AI systems can lift and cite it cleanly.
- Use structured data and clean headings. FAQ schema and question based headings make your content easy to parse without any special file.
- Demonstrate real expertise. First hand experience, named authors and local proof signal trust to both Google and AI models.
- Keep your robots.txt and sitemap correct. These are the files crawlers genuinely use, so make sure AI crawlers are allowed where you want them.
For practical detail, our piece on writing content that gets cited by ChatGPT walks through the structure that works, and our overview of how Perth brands win in AI search ties it into a full strategy.
Frequently Asked Questions
Does an llms.txt file help my Google rankings?
No. Google has stated that llms.txt is not needed for AI Search, and Google does not support the file. It has no effect on your rankings or on whether you appear in AI Overviews.
Why does Chrome Lighthouse check for llms.txt then?
Lighthouse is a developer tool, and that check reflects the developer documentation use case, not search. Google has confirmed the Lighthouse audit is not a ranking signal, so passing it does not improve your visibility.
Who should actually use an llms.txt file?
Mainly software and developer tool companies with large documentation libraries that AI coding assistants read. If your audience is not developers, you can comfortably skip it for now.
The Bottom Line
The honest answer to do you need an llms.txt file is almost certainly no. Google does not use it, the major AI crawlers rarely touch it, the Lighthouse check is a developer nicety rather than a ranking signal, and the one genuine use case applies to developer documentation rather than to everyday business websites.
Trends like this come and go quickly in AI search. The smart move for Australian businesses is to ignore the hype, watch how the standard evolves, and keep investing in clear, trustworthy, well structured content. That is what wins citations today, with or without an llms.txt file.
Confused about what actually moves the needle in AI search? Dlinkers helps Perth and Australian businesses get found and cited across Google, ChatGPT and AI Overviews with content built to be trusted. Book a free AI search visibility review today and get a clear plan.
