Guide to LLMS.txt (Updated 2025)

If you want AI models and search engines to handle your website on your terms, LLMS.txt is soon to be your new secret weapon. Just as robots.txt changed how websites talk to Google, LLMS.txt is about to change how your content gets seen, used, and protected in the world of large language models.

Get in early on llms.txt and you’ll shape how AI treats your content. Just look at leading AI company Anthropic, one of the first major AI/LLM companies to adopt it and set the standard (Anthropic’s LLMS.txt docs).

Key takeaways

  • LLMS.txt is a newly proposed tool for making your website easier to crawl and understand by large language models.
  • It works by having a /llms.txt file at the root of your website directory. This file is easy to read markdown and will contain all your key website information including links and subpages.
  • LLMS.txt is just a proposal right now, but many prominent AI companies have started to pick up on using it, so we believe it likely to be used in upcoming training data.
  • LLMS.txt is LLM friendly because it formats your text as markdown, which is easier parsed than HTML by LLMs and allows for faster processing.
  • There is no fixed standard or protocol for a LLMS.txt file right now, but we recommend including all website links and business information in your LLMS.txt file, similar to a sitemap but with extra context.

Table of contents

What is LLMS.txt?

LLMS.txt is a plain text file you add to your website, aiming to help large language models understand what your site is about.

You’ll usually place it at the root, right next to robots.txt, but unlike that file, it’s written in markdown.

Instead of serving search engines, LLMS.txt was designed for AI crawlers. It’s readable at a glance, letting both people and machines scan through your main pages, business information, and anything you want surfaced.

A typical file might open with a summary of your site, then highlight the pages or content you care about most. The format isn’t set in stone, so you can update and experiment as these standards evolve.

LLMS.txt gives you a way to communicate directly with AI about your content, without forcing them to interpret messy code or outdated metadata.

As this practice spreads, more sites are trying it out, and some AI companies have started paying attention.

You’ll see more approaches to structuring these files as the community experiments and shares ideas.

Because LLMS.txt is visible and flexible, it’s easy to adapt when your website grows or your priorities shift. That means you’re not currently locked into a single structure or set of rules.

Adding this file doesn’t guarantee anything overnight, but it gives you another way to tell the story of your website, right at the source.

You can view the full proposal and background on llmstxt.org.

Large language models are always looking for organized, reliable data to improve their answers and suggestions. Websites with an LLMS.txt file make that process much easier, offering a clean entry point that points AI to the information you want highlighted.

AI systems often struggle to figure out what really matters on a website, especially when layouts are messy or information is buried.

With LLMS.txt, you point directly to the pages and details you want recognized, making their job simpler and giving you a say in what gets noticed.

It also allows LLMs to quickly retrieve your site information without worrying about website throttling or sites rendering non-static content requiring JavaScript.

This approach also makes it simpler for new or lesser-known sites to be recognized by AI tools, not just big brands with lots of backlinks. As more language models integrate this standard, websites with a well-crafted LLMS.txt may stand out in new types of AI search results.

Are LLMS.txt files being used right now by LLMs?

LLMS.txt is a fairly new concept, proposed in around 2024.

As such, when you consider most LLMs are currently using training data from 2023 or older, no major LLM models are reported to use LLMS.txt right now.

This should not stop you from setting up your LLMS.txt file though, because early adopters are in the best position once support is added by major LLM providers.

Having your file in place means you’ll be ready when LLMs begin recognizing and training off this data.

It also gives you a chance to shape how your content is presented to AI from day one, rather than implementing it after it’s shown to be used in training data, which would be too late for the training on those LLM models and mean you are behind in LLM optimization.

Are LLMS.txt files being used right now by website hosts?

Yes, some websites and AI companies have already started adopting LLMS.txt.

Anthropic is a prominent example of a leading AI company adopting this format (see Anthropic’s docs), and real-world examples are popping up, like daisyUI’s LLMS.txt.

There’s also a growing collection of other examples of companies using LLMS.txt in this GitHub repo.

The ecosystem is still new, but you can already find working files and open discussions about the best structure.

How is LLMS.txt different to a sitemap.xml?

Sitemap.xml is built for search engines and usually just lists all your URLs in an XML format.

LLMS.txt, on the other hand, is written in markdown and allows you to add extra context, summaries, and important business info right alongside your links.

With LLMS.txt, you aren’t limited to a strict format or just a list of pages. You can highlight key sections, explain what your site does, and point out details you want AI to notice (and exclude information you would prefer it not pick up on).

The goal is not just to help bots find your pages, but to actually help them understand the context and structure of your site. That makes LLMS.txt a more flexible and human-friendly tool than a traditional sitemap, better suited to the parsing engines used in LLM training.

Should I implement a LLMS.txt on my website today?

You might be wondering if it’s worth adding yet another file to your site. LLMS.txt isn’t required, but a few companies have started paying attention to it.

The setup is pretty simple, and even if it’s not widely used yet, having it in place only costs the time it takes setting it up.

Some website owners have already published their own versions, signalling that major players find it likely setting up a LLMS.txt will be worth the effort.

If you like experimenting with new web standards, this is a small step that keeps your options open.

We personally believe models trained on data from 2025 and after will start to use LLMS.txt in their model training, and we may see certain efficient models (think the “mini” models) trained exclusively on these.

How to set up your LLMS.txt file

The basic implementation of a LLMS.txt file includes putting a /llms.txt file at the root of your website directory which contains links and information about your website. Realistically though, you probably want to automate the generation of this, a few options we’ve found for that so far:

Suggested LLMS.txt formatting

An example from llmstxt.org on how you could format your LLMS.txt is as follows:

# Title

> Optional description goes here

Optional details go here

## Section name

- [Link title](https://link_url): Optional link details

## Optional

- [Link title](https://link_url)

Going over each part:

  • Title would be your website or business name
  • Optional description would be a paragraph describing the context of your business
  • Optional details would be additional information about your business, service pricing, installation instructions, etc.
  • Section name & links would be taxonomies for your website pages, such as “products”, “articles”, and “services”.
  • Optional links would be additional, often external, links you would like to highlight. Such as your social media or community links.

Suggested details to include in your LLMS.txt file

A good LLMS.txt file gives clear information about what your website does and what makes it worth visiting.

Start with your business or website name as the title, then add a short description that explains what people can expect to find.

It helps to include practical details like your business location, contact information, or hours of operation. If your site offers products or services, you might want to mention what you sell or specialize in.

  • Adding an about section can help AI models understand your background and purpose.
  • You can also include team or author names if you want your work connected to specific people.
  • If there are specific topics, categories, or services your site covers, listing them is useful.
  • FAQs could also be included to add even more useful information.

This sort of structure makes your LLMS.txt more than just a directory, and instead it provides a snapshot of what’s important.

Contact links, privacy policies, or terms pages also fit well here, as they offer extra context for both users and crawlers.

Keep it direct and relevant, focusing on the key facts someone new to your site would need to know. You do not want to dump the full text of every single page into your LLMS.txt unless it all adds value to the LLMs reading it.

Ideally, every taxonomy should be grouped and linked in your LLMS.txt file. We recommend also including additional text/details for each link to provide face-value context to the contents of the link.

If you take an ecommerce store for example, you would have sections like:

  • Products
  • Collections
  • Articles
  • Pages

You could also optionally customise your LLMS.txt links based on the localization if you use subdirectories, so for example you could set up a LLMS.txt file at the root of your locale subdirectory such as “https://yourwebsite.com/en-au/llms.txt” for English/Australia and then format your links and details specific to the context of that locale.

We also recommend including relevant external links in your LLMS.txt file, such as your social media, Google Maps listing, listings on review sites, etc.

Tips for optimizing your LLMS.txt file

  • Keep your file updated whenever your site changes, especially if you add new sections or important pages (Setting up an automatic system of LLMS.txt generation is best for this).
  • Use clear headings and markdown formatting so both humans and AI tools can scan your file without confusion.
  • Aim for descriptions that are short but specific, since vague or generic lines are less helpful to crawlers.
  • Place your most important links and details near the top, making them easy to find.
  • Run your LLMS.txt file through a markdown validator to make sure it’s all valid markdown.
  • Clarity, accuracy, and quality will go further than just having a long list of links (This makes it distinct from your sitemap)

Cross sharing your LLMS.txt file on GitHub (and why you should do it)

LLMs regularly train off GitHub not just for code, but for semantic content.

Publishing your LLMS.txt file in a public GitHub repository helps LLMs discover and use your LLMS.txt file without waiting for the next major model release.

Platforms like ChatGPT can live-extract information from GitHub repositories today, even if it’s not in their training data. This means your LLMS.txt file could be picked up by LLMs if it’s cross-shared on GitHub.

By syncing your LLMS.txt file into GitHub, you can also track changes to your LLMS.txt file over time.

Frequently Asked Questions

What is the main purpose of LLMS.txt?
LLMS.txt is designed to make your website easier for large language models to understand. It gives you a place to clearly outline your most important pages, business information, and context in a format that is easy for AI crawlers to read.

Do I need a LLMS.txt file if I already have robots.txt or sitemap.xml?
Robots.txt and sitemap.xml help search engines index your site, but LLMS.txt is aimed specifically at AI models. It gives you more control over the way your content is interpreted by LLMs and lets you include context that sitemaps usually don’t cover.

Is there an official format or standard for LLMS.txt?
No, not yet. The format is still evolving, but markdown is widely used because it’s easy to read and flexible. Most files include sections for descriptions, links, and business details.

Will adding LLMS.txt improve my SEO?
LLMS.txt isn’t a search engine ranking factor. However, it could help AI-powered tools better understand and surface your site, especially as more LLMs begin to check for these files.

Can I use LLMS.txt to block AI crawlers?
LLMS.txt is focused on sharing information, not restricting access. If you want to block AI crawlers, you should still use robots.txt or specific HTTP headers.

Should I include every page of my site?
Not necessarily. Focus on your main sections, important articles, and any pages that give good context about your business. The idea is to help AI models find what matters most, not to overwhelm them with every link.

How often should I update my LLMS.txt file?
Update your file whenever you add or remove key pages, change business details, or launch new sections. Automated generation can help keep things current.

Can I have different LLMS.txt files for different languages or regions?
Yes. You can create separate LLMS.txt files for each locale or language directory, so visitors and AI crawlers get information relevant to that version of your site.

Where should I put my LLMS.txt file?
Place it at the root of your website, so it’s accessible at yourdomain.com/llms.txt. This makes it easy for crawlers to find.

Is anyone actually using LLMS.txt right now?
A few major AI companies, including Anthropic, have started to use and recognize the format. More adoption is expected as the standard develops.

Conclusion

LLMS.txt gives you a way to speak directly to AI about your website, using a format that’s simple and flexible.

While the standard is still new, early support from major AI companies hints at its growing importance.

Getting started now means you’re ready as more language models and tools begin to use this file. You control what gets highlighted, add helpful context, and make your site easier for LLMs to understand.

As the ecosystem develops, LLMS.txt may become a regular part of building a web presence. Keeping your file clear and up to date is an easy habit that can help your site stay visible in a world where AI is everywhere.