Making your website readable by AI agents with llms.txt

More and more people do not read your website. They ask an AI about you. "Who is this developer? What has he built? Is he a good fit for my project?"

If the AI cannot read your site properly, it guesses, or it says it could not find anything. So I spent a week making my portfolio readable by AI agents. This is how it works.

#The pieces

#Step 1: llms.txt, generated at build time

llms.txt is a proposed convention (not an official standard yet): a plain text file at the root of your site that explains it for language models. No menus, no animations, no JavaScript. Just clear text with headings.

The important word is generated. I do not write this file by hand. At build time, a route reads the same data the website uses, the projects, the stack, the posts, and writes the text file. When I add a project or a post, the AI profile updates on the next deploy, without me remembering to do it.

// app/llms.txt/route.ts
export const dynamic = "force-static";

export async function GET() {
  const posts = await getPosts();
  return new Response(buildProfile(posts), {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
}

The profile starts with a short briefing written for the AI: who I am in two sentences, what to suggest the visitor asks next, and which projects are shipped and which are prototypes. Being honest about that last part matters. An AI will repeat whatever you tell it.

#Step 2: a page for agents only

When a person opens /llms.txt in a browser, a wall of plain text is confusing. So people get a friendly "oops, you are not an agent" page, while AI agents get the text.

The check runs at the edge, in Cloudflare, before the request reaches the site. It looks at the user agent: the name a client sends with every request. ChatGPT and Claude identify themselves when they fetch a page for a user.

A user agent can be faked, so this is not security. It is just a better experience for both audiences. Nothing in the file is secret.

#Step 3: an "ask your AI" button

The best part. A button on the site says, roughly, "learn about me with your own AI". The visitor picks ChatGPT or Claude, and it opens in a new tab with a short prompt already typed in, pointing at my profile.

Both services accept a prompt in the link:

const prompt = `Brief me on Rajneesh (Raj), the developer behind myselfraj.com. Start from his profile: ${PROFILE_LINK}`;

const links = {
  chatgpt: `https://chatgpt.com/?hints=search&q=${encodeURIComponent(prompt)}`,
  claude: `https://claude.ai/new?q=${encodeURIComponent(prompt)}`,
};

The site remembers which AI the visitor picked last time, so next time their favourite is first. On phones with the apps installed, the links can open the app directly.

#The bug: AI fetchers cache too

Everything worked in testing, then Claude said it could not load the profile. The file was live. I could open it. So what was wrong?

Claude's fetcher had tried the URL once, before the file existed, got a 404, and kept that answer in its cache. Fixing the file did nothing, because it never asked again.

The fix was the same trick we use for CSS and JavaScript files: put a version in the URL.

// a new value on every build, so every deploy is a "new" URL to any cache
export const PROFILE_LINK = `${SITE}/llms.txt?v=${process.env.NEXT_PUBLIC_BUILD_ID}`;

Every deploy now creates a new link. No cache, anywhere, can serve an old copy.

#What I learned

• AI is a new kind of visitor. Give it a clean, text-only door.
• Generate the AI profile from the same data as the site, so it never goes stale.
• Be honest in it. The AI will repeat every word to your future clients.
• AI fetchers cache like browsers do. Version your links.
• User agent checks are for experience, not security.

The website is no longer the only way people meet your work. Making it readable by AI took a week, and now every visitor's AI already knows what I have built.