What is llms.txt? (And why your website probably needs one)

If you've been running a website for a while, you've probably come across something called robots.txt. It's the little file that sits at the root of your domain and tells search engines what they're allowed to crawl. You probably didn't write it yourself (your developer or CMS handles it), but you know it exists.

llms.txt is the same idea. Same place on your website, same basic logic. But instead of talking to Google's crawler, it talks to AI systems: ChatGPT, Claude, Perplexity, and the rest.

Here's what it does, why it matters for UK businesses, and how to get one set up without any technical headaches.

Why did anyone need a new file?

When an AI assistant gets asked a question ("what's a good florist near me?" or "how do I write a commercial lease?"), it needs to pull information from somewhere. Often that means your website.

But AI systems have a constraint you might not have thought about: they can't read your whole site in one go. A large language model like GPT-4 has a "context window" (think of it as a working memory limit). It can only hold so much at once, so it has to make quick decisions about which bits of which sites to trust and quote.

Without an llms.txt file, it's making educated guesses. With one, you've given it a direct briefing.

What does it actually look like?

The file lives at yourdomain.co.uk/llms.txt and it's just plain text. Here's a realistic example:

# llms.txt for Acme Ltd
# https://llmstxt.org specification

> Acme Ltd is a Limited Company operating under UK law.

## Permissions

# OpenAI: trains ChatGPT and GPT-4o models
User-agent: GPTBot
Allow: /

# Anthropic: trains Claude AI assistants
User-agent: ClaudeBot
Allow: /

# Google: Gemini AI training (no impact on Search rankings)
User-agent: Google-Extended
Allow: /

# ByteDance: TikTok and ByteDance AI products
User-agent: Bytespider
Disallow: /

# DeepSeek: open-source AI models
User-agent: DeepSeekBot
Disallow: /

## Legal

All content is subject to UK copyright law (CDPA 1988).
This site is governed by the laws of England and Wales.

## Contact

For AI licensing enquiries: legal@acme.co.uk

## Last Updated

2025-05-18

Simple, right? That's intentional. Jeremy Howard (who proposed the standard in September 2024) kept it minimal on purpose. Simple standards get adopted. Complex ones don't.

The file tells AI systems four key things: who you are, which crawlers can use your content, what legal framework applies, and who to contact for licensing. That's it.

Who's actually reading it?

More AI companies than you might expect. Anthropic (Claude), Perplexity, and others have indicated they'll respect llms.txt permissions. The community-maintained ai-robots-txt project now tracks 23 major AI crawlers from 14 companies, including OpenAI, Google, Meta, Apple, Amazon, ByteDance, Mistral, Cohere, and DeepSeek.

Is it legally binding? No. Think of it less like a contract and more like a clear sign on your door. Most people who see it will respect it. The ones that don't? Well, that's a different conversation involving a solicitor.

But having a clear, timestamped statement of your permissions matters under UK copyright law. The Copyright, Designs and Patents Act 1988 protects your content from unauthorised use, and "AI training without permission" is exactly the kind of thing that's currently being tested in courts.

How llms.txt compares to robots.txt

When robots.txt was proposed in 1994, it wasn't legally enforceable either. It took a few years to become the universal standard it is today. By 2000, virtually every serious web crawler respected it.

llms.txt is in roughly the same position now. Early momentum, no competing proposals, and a growing number of AI companies acknowledging it. The businesses setting one up in 2025 are getting ahead of the curve, and they'll have a dated record to show for it.

What it doesn't do

Worth being upfront about the limitations:

• It doesn't have legal force on its own. It's a signal, not a contract.
• It can't guarantee compliance. An AI company that wants to ignore it can.
• It doesn't replace robots.txt. Both files serve different purposes and you need both.

What it does is create a clear, public, machine-readable record of your position. In an era where AI companies are under increasing scrutiny over how they use content, that signal matters more than people realise.

How do you get one?

That's the part we made straightforward. geotool.co.uk generates your llms.txt file in about 60 seconds. You pick which AI crawlers to allow or block (there are 23 to choose from, grouped by company), add your UK copyright details, and download the file.

Then you (or your developer) upload it to the root of your website. Done. No account needed, no cost, no AI API calls. The whole thing runs in your browser.