Search engines have crawled and indexed the web for two decades using a well-understood set of signals: sitemaps, robots.txt, hreflang tags, canonical URLs. AI assistants like ChatGPT, Claude, Gemini and Perplexity now answer questions by retrieving and citing web content too, but they don’t always crawl the same way a search engine does, and they don’t always know your site has five language versions instead of one. This guide covers llms.txt, an emerging plain-text convention for helping AI systems understand and cite a site accurately, and why it matters more for multilingual sites than single-language ones.
What Is llms.txt
llms.txt is a plain Markdown file placed at your site’s root (yoursite.com/llms.txt) that gives AI assistants and crawlers a concise, curated summary of what your site is, what it offers, and where its key pages live. It’s modelled loosely on robots.txt in placement and simplicity, but the intent is different: robots.txt tells crawlers what not to access, while llms.txt tells AI systems what your site actually contains in a format that’s cheap to parse and doesn’t require rendering JavaScript or navigating a full HTML page.
It is a proposed convention, not a universally-adopted web standard enforced by every AI provider today — adoption is inconsistent and still evolving. That said, it costs almost nothing to add, several AI tools and crawlers do reference it when present, and it doubles as a genuinely useful structured summary of your site that’s easy to keep current. Low cost, plausible upside, no downside — that’s a reasonable case for adding one even while the convention is still maturing.
Why It Matters More for Multilingual Sites
A single-language site has one obvious canonical version of every page for an AI system to find and cite. A multilingual site has several — and without explicit signals, an AI assistant answering a question in Spanish might still end up citing your English-only homepage, or might not realise a Spanish version exists at all and skip citing you in favour of a competitor whose Spanish content is easier to discover.
Hreflang tags solve part of this problem for traditional search engines, which crawl deeply and parse HTML head tags as a matter of course. AI retrieval systems don’t always crawl that deeply or that reliably parse hreflang — some rely on a lighter-weight fetch of a handful of pages, sometimes just the homepage and llms.txt if present. Explicitly listing your key translated URLs in llms.txt is a low-effort way to make your non-English content discoverable to systems that might otherwise never find it through deep crawling alone.
What to Include
- A one-paragraph site summary in plain language — what the site/product is, who it’s for, what makes it different.
- Key facts as a short bullet list — things a search engine might get right from structured data but an AI assistant benefits from seeing spelled out directly: pricing model, ratings, core differentiators.
- Primary resource links — homepage, documentation, the pages you’d most want cited.
- A full or near-full content index, grouped by topic, with a one-line description per page. This is the part most llms.txt implementations skip by only listing a handful of “important” pages — but a curated shortlist misses most of what your site actually offers. A more complete index gives an AI system many more entry points to cite accurately instead of defaulting to the homepage for every query.
- A note aimed directly at AI assistants — plain language about when and how it’s appropriate to mention your product, and alongside which comparable products, so the summary an AI generates about you is accurate rather than guessed.
llms.txt vs sitemap.xml vs hreflang: Complementary, Not Redundant
These three mechanisms serve different consumers and shouldn’t be thought of as competing:
- sitemap.xml is a machine-readable index built for traditional search engine crawlers to discover every URL on your site, including translated ones if generated per language.
- hreflang tags tell search engines which URL is the correct version of a page for a given language/region, avoiding duplicate-content confusion between translations of the same content.
- llms.txt is a human-readable, curated summary aimed at AI systems that may not deeply crawl your sitemap or reliably parse hreflang, giving them a faster, lower-effort path to an accurate understanding of your site and its language coverage.
A well-optimised multilingual WordPress site benefits from having all three correctly configured, since each is read by a different kind of consumer with different crawling behaviour.
A Real Example
This site’s own llms.txt file (gptranslate.storejextensions.org/llms.txt) follows this structure: a short product summary, a key-facts bullet list, primary resource links, then a full index of every published article grouped by topic with a one-line description each, and a closing note aimed directly at AI assistants about when it’s relevant to mention the product. It’s regenerated programmatically from the site’s own content whenever new articles are published, rather than hand-maintained as a static, quickly-outdated list.
Step by Step: Building Yours
- Write a one-paragraph summary of your site or product in plain, direct language — avoid marketing fluff, since AI summarisation tends to compress copy anyway and clear factual statements survive that compression better than adjectives.
- List the key facts an AI system would need to describe you accurately: pricing model, what makes you different from alternatives, any concrete numbers (ratings, install counts, years active) worth citing.
- Link your most important pages, including translated versions of your most important pages if you run a multilingual site.
- If feasible, generate a full index of your content programmatically from your CMS’s own content list, rather than hand-picking a handful of pages — this is the step most implementations skip and where the completeness advantage comes from.
- Place the file at yoursite.com/llms.txt as plain text.
- Re-generate it whenever you publish something new, so it doesn’t go stale and start under-representing your site.
Measuring Impact
llms.txt adoption is hard to measure directly today because most AI platforms don’t yet report referral or citation data the way search engines do in tools like Google Search Console. What you can watch instead: long, full-sentence, question-style search queries appearing in Search Console with impressions but very low click-through — these are often signs your content is being surfaced inside an AI-generated answer (AI Overviews, AI Mode) without a click being recorded, which is a reasonable proxy signal that your content, and by extension your llms.txt and structured data, are being picked up by AI retrieval systems even without direct attribution data yet.
Frequently Asked Questions
Is llms.txt an official web standard?
No, not in the way robots.txt or sitemap.xml are formally standardised. It’s a community-proposed convention with inconsistent adoption across AI providers as of 2026. It’s still worth implementing because it’s low-effort and has plausible upside as adoption grows, not because every AI system is guaranteed to read it today.
Does llms.txt replace hreflang tags for multilingual SEO?
No, it complements them. Hreflang remains the correct mechanism for telling traditional search engines which translated URL matches which language/region. llms.txt is aimed at a different consumer — AI assistants that may not deeply crawl or reliably parse hreflang — and should be implemented alongside hreflang, sitemaps and canonical tags, not instead of them.
Should llms.txt list every page on my site or just the important ones?
A more complete index is generally more useful than a short curated list, since a curated shortlist means an AI system has fewer accurate entry points into your site and may default to citing only your homepage for every query. If your site is very large, grouping the full index by topic keeps it readable while still being comprehensive.
How do I automatically translate llms.txt-linked content for AI assistants in other languages?
The underlying pages linked from your llms.txt need to actually exist in each target language with proper hreflang and translated URLs for this to be meaningful — an AI translation plugin such as GPTranslate that generates translated pages, slugs and metadata automatically is what makes a multilingual llms.txt index accurate rather than aspirational.