What is llms.txt? Does it help AI search?
llms.txt is a proposed text file that lists a website's key pages in Markdown for AI tools. As of November 2026, Google says Search ignores it, and an Ahrefs study found 97% of these files got no requests in May 2026.
# The Social Theory > Nathan Swift is an AEO and local SEO specialist who builds AI-ready websites that help small businesses get found on Google and in AI answers. ## Start here - [AI Search Optimization for Small Business](https://thesocialtheory.com/): how to get named in ChatGPT, Google and AI answers, a free check for your site and real results - [AI Search Guides for Small Business](https://thesocialtheory.com/blog/): every guide, grouped by the question it answers - [Answer Engine Optimization Services for Small Businesses](https://thesocialtheory.com/answer-engine-optimization-services/): what Nathan does for small businesses, what AEO usually costs, real results and how to start ## AI search guides ...
The first lines of our own llms.txt, rewritten in November 2026.
What is llms.txt?
llms.txt is a Markdown file at a website's root that lists its most useful pages for AI tools. Jeremy Howard proposed it in September 2024.
1# The Social Theory 2> Nathan Swift is an AEO and local SEO specialist who builds AI-ready websites... 3## AI search guides - [What Is AEO?](https://thesocialtheory.com/blog/...): how AEO works - [How to Rank in ChatGPT](https://...): the searches ChatGPT runs 4## Optional - [YouTube](https://...): videos
- 1A heading with the site's name
The proposal calls it "the only required section."
- 2A one-line summary
A quote block that says what the site is, so the rest makes sense.
- 3Sections of links
Each H2 lists pages with a short note on what each one covers.
- 4An "Optional" section
Links an AI tool can skip when it needs a shorter version.
The format, from the llmstxt.org proposal, shown with our own file.
“We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content.”
Jeremy Howard, who proposed llms.txt, published September 3, 2024: The /llms.txt file
| llms.txt | robots.txt | |
|---|---|---|
| What it does | Lists your pages, with notes, for AI tools to read | Tells crawlers which pages they may fetch |
| Who reads it | Google says Search ignores it. OpenAI, Anthropic and Perplexity don't say their crawlers read it | Google, OpenAI, Anthropic and Perplexity all say their crawlers follow it |
| Blocks anything? | No. It controls nothing and blocks nothing | Yes, for every crawler that follows it |
Robots.txt rules come from each company's crawler page. "Controls nothing and blocks nothing" is Ahrefs' description of llms.txt.
What this means for you
llms.txt is a table of contents written for AI tools. It sits at yoursite.com/llms.txt, next to robots.txt, but it does a different job: it suggests what to read, and it can't hide or open up any page.
The proposal says the file is used "on demand, when an agent needs information about a topic while assisting a user." So it helps only when an AI tool goes looking for it.
Does llms.txt help you show up in AI search?
No, llms.txt doesn't help you show up in Google Search or its AI answers. A 137,000-site study also found AI search bots almost never request the file.
“You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.”
Google Search Central, documentation, read November 17, 2026: Optimizing your website for generative AI features
of llms.txt files got zero requests in May 2026.
Ahrefs checked 137,210 sites. About 38,000 had a valid llms.txt.“If your goal is showing up in ChatGPT, Perplexity, or AI Overviews, an llms.txt file is largely decoration.”
Louise Linehan, Ahrefs, June 15, 2026: We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read
What this means for you
Two different checks point the same way. Google says its Search and AI answers ignore the file. Ahrefs looked at what bots request and found almost nothing asks for it.
Google also says keeping one "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." So a file you already have is fine. Building one won't move your rankings.
One limit: Ahrefs' sites skew technical, and it calls every number a ceiling, because a request doesn't prove anything read the file.
Who uses llms.txt files today?
Mostly non-AI bots and tools fetch llms.txt files, and AI agents are the biggest AI group. In Ahrefs' May 2026 study, AI search bots made 1.1% of requests.
| Who requested llms.txt files | Share of requests | Detail |
|---|---|---|
| Bots and tools that aren't AI | 77% | Ahrefs: "77% of the bots fetching llms.txt aren't AI tools at all" |
| AI agents | 10.5% | Includes Claude Code, Anthropic's coding agent |
| AI training crawlers | 5.3% | GPTBot 4.51%, ClaudeBot 0.8% |
| People | 4% | Visits from people, not bots |
| AI assistants | 2.5% | |
| AI search bots | 1.1% | OAI-SearchBot, ChatGPT's search crawler, 0.74% |
Share of all llms.txt requests in May 2026, from the 3% of files that got any. Ahrefs, June 15, 2026.
What this means for you
llms.txt was built for AI agents that read documentation, like coding assistants. That is who comes closest to using it. In Ahrefs' words, "Slackbot alone fetched llms.txt files more often than PerplexityBot did."
OpenAI and Perplexity both link an llms.txt from their own developer docs, as an index for coding tools. Their crawler pages control access with robots.txt and say nothing about reading the llms.txt on your site (see the OpenAI and Perplexity crawler pages).
Should a small business have an llms.txt file?
A small business doesn't need an llms.txt file to show up in Google or AI answers. If your website platform makes one for free, keep it accurate.
Your platform makes one for free
Keep it. Check a few times a year that it describes your business the way you do today.
Costs you nothing, can't hurt your Google rankings.You'd have to build it by hand
Skip it for now. Put that hour into your pages and your Google Business Profile.
That is where people and AI tools find you.What happened to ours
Our llms.txt still described the business we used to run.
- "helps small businesses build their brand and learn to use AI"
- Linked "Learn" and "AI News", menu labels we had already retired
- Sent "Contact" to /#contact/, a typo that skips the contact form
One line that matches the rest of the site, and every guide listed by its current title.
- "Nathan Swift is an AEO and local SEO specialist who builds AI-ready websites..."
- Grouped the way the guides page is grouped
- Older case studies moved to the Optional section
What this means for you
A file nobody reads goes stale without anyone noticing. Ours did. If you keep one, it should say the same thing your website, your Google Business Profile and your structured data say.
Some audit tools still score it. The free AEO Site Checker gave our site 5 of its 100 points for having an llms.txt. Treat those points as tidiness, not visibility.
How do you create an llms.txt file?
To create an llms.txt file, list your key pages in a short Markdown file. Start with your business name and a one-line summary, then upload it to yoursite.com/llms.txt.
# Your Business Name > One sentence: what you do, for whom, and where. ## Main pages - [Services](https://yoursite.com/services/): what you offer and what it costs - [About](https://yoursite.com/about/): who you are, licenses and service area - [Contact](https://yoursite.com/contact/): phone, hours and how to book ## Guides - [Your most useful guide](https://yoursite.com/your-guide/): the question it answers ## Optional - [Blog](https://yoursite.com/blog/): older posts and news
AWrite it in Markdown
Start with your business name as a heading, then one sentence in a quote block. List your key pages under short headings, each with a few words on what it covers.
BUpload it to your site's root
It has to open at yoursite.com/llms.txt as plain text. Some website platforms make one for you (Ahrefs names Wix). Otherwise, ask yours where to upload it.
CKeep it true
Update it when you add, rename or retire a page. A stale file tells AI tools the wrong thing about you.
What helps you show up in AI search instead?
Pages that AI tools can crawl, index and quote help you show up in AI search. Lead with direct answers, add real numbers and sources, and run a free check.
- Google can crawl and index the page: robots.txt allows it and it loads without a login
- The page opens with a direct answer to the question, in 25 to 50 words
- Headings are the questions your customers ask
- Real numbers, named sources and first-hand detail
- Your name, phone and hours match on your site, your Google Business Profile and your structured data
What this means for you
These are what Google's guide and the GEO study point to. Our AI Overview optimization guide lists the tricks Google says to skip, llms.txt among them. For ChatGPT, how to rank in ChatGPT covers what its search reads.
The GEO study found that adding quotes, statistics and sources made pages show up more in AI answers. That is time better spent than a text file.
Can someone set up my website for AI search?
Yes, an AEO expert can set up your website for AI search. I build answer-first pages on Halo, quoted for your business, or you can run Halo yourself.
What this means for you
Halo is the AI website you run by text that I build on. Every Halo site ships with sitemaps, structured data and an AEO readiness score, so the technical checks on this page are handled. You text Halo the questions your customers ask, and it builds the pages.
Get it built by an AEO expert
I'm an AEO and local SEO specialist. I find the questions your customers ask and build answer-first pages for them on Halo.
Quoted for your business.Work with meDo it yourself on Halo
Halo Pro is $99 a month for two websites on your own domains, and Halo Start is free for one page.
Checked on huzihalo.com, November 16, 2026. Prices can change.See Halo pricingWhat is llms.txt: every answer in one place
What is llms.txt?
llms.txt is a Markdown file at a website's root that lists its most useful pages for AI tools. Jeremy Howard proposed it in September 2024.
Does llms.txt help you show up in AI search?
No, llms.txt doesn't help you show up in Google Search or its AI answers. A 137,000-site study also found AI search bots almost never request the file.
Who uses llms.txt files today?
Mostly non-AI bots and tools fetch llms.txt files, and AI agents are the biggest AI group. In Ahrefs' May 2026 study, AI search bots made 1.1% of requests.
Should a small business have an llms.txt file?
A small business doesn't need an llms.txt file to show up in Google or AI answers. If your website platform makes one for free, keep it accurate.
How do you create an llms.txt file?
To create an llms.txt file, list your key pages in a short Markdown file. Start with your business name and a one-line summary, then upload it to yoursite.com/llms.txt.
What helps you show up in AI search instead?
Pages that AI tools can crawl, index and quote help you show up in AI search. Lead with direct answers, add real numbers and sources, and run a free check.
Can someone set up my website for AI search?
Yes, an AEO expert can set up your website for AI search. I build answer-first pages on Halo, quoted for your business, or you can run Halo yourself.
Start with a free AI visibility check or read what AEO is next.
Where these facts come from
- The /llms.txt file (the proposal) llmstxt.org, read on November 17, 2026
- Optimizing your website for generative AI features on Google Search developers.google.com, read on November 17, 2026
- Search Central documentation updates (llms.txt note, June 15) developers.google.com, read on November 17, 2026
- We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read ahrefs.com, read on November 17, 2026
- Overview of OpenAI crawlers developers.openai.com, read on November 17, 2026
- Perplexity crawlers docs.perplexity.ai, read on November 17, 2026
- Does Anthropic crawl data from the web, and how can site owners block the crawler? support.claude.com, read on November 17, 2026
- AEO Site Checker audit of thesocialtheory.com, 93 of 100 aeositechecker.com, read on November 17, 2026
- Halo huzihalo.com, read on November 17, 2026
Skip the file. Fix the pages.
Have an AEO expert build pages AI tools quote, or see how AI tools read your site today.
Want an AEO expert to set it up?
I'm an AEO and local SEO specialist. Get in touch and tell me which questions you want AI tools to answer with your business. Or read what Halo costs and my Huzi Halo review first.