AI agents now research companies for buyers, and most websites were built only for people. Every business needs a page written for the agents.
At a glance
- More than half of web traffic is no longer human. Most of it isn’t AI, but a fast-growing share comes from AI agents that open a page because a person just asked them a question.
- Websites were designed for human attention. Agents need plain facts and clear directions, and when they can’t find them, they fill the gaps with whatever else they can find.
- A page for AI agents gives them what they need to represent you accurately. It should be honest, welcoming and free of manipulation. We built one, and we think every business needs one.
In February 1994, an engineer named Martijn Koster had a problem. A badly behaved web crawler had knocked over the server he ran. His fix was a small text file telling automated visitors which parts of a site to stay out of. His first name for it was RobotsNotWanted.txt. It became robots.txt.
Most crawlers honored it within months. It took 28 more years for the Internet Engineering Task Force to publish it as a standard, RFC 9309, in 2022. For three decades, robots.txt was most of what websites said to machines, and what it said was where they couldn’t go.
In October 2026, we put up a different kind of sign: a page for AI agents at orpical.com/machines. It reads the name an automated visitor sends when it arrives and greets it. If you’re Anthropic’s crawler, the first thing you see is “Hello, ClaudeBot.”
It started as a bit of fun. The longer we worked on it, the more serious it got. AI agents now do part of the research behind buying decisions, and websites were built for people. An agent arriving for a buyer gets a brochure when it needs a briefing. Every business should give agents a front door: a page that says plainly who you are, what you do, who you’re right for and how to get things done, without trying to manipulate them.
Most of your visitors aren’t people anymore
Imperva, a web security firm, put automated traffic at 51% of all web traffic in 2024. In June 2026, Cloudflare, which carries a large share of the world’s web traffic, reported that automated requests had passed human ones on its network, well ahead of its own forecast.
Most of those machines have nothing to do with AI. Search engines index pages. Monitoring tools check that sites are up. Messaging apps fetch link previews. Scrapers copy prices and content. A large share is hostile: Imperva counted malicious bots, the kind that test stolen passwords, probe for weaknesses and spam contact forms, at 37% of all internet traffic.
The new arrival is different. Cloudflare tracks what it calls “user action” crawling: an AI fetching a page because a person asked it something, right now. That traffic grew more than 15-fold in 2025.
The people behind those requests are often buyers. In a March 2026 G2 survey of more than 1,000 B2B software buyers, 51% said they start research with an AI chatbot more often than with Google, up from 29% a year earlier. As we wrote in Mining Institutional Knowledge to Enrich Digital Marketing Agents, the first round of research increasingly happens somewhere you never see.
Your buyers are being agented
An AI agent is an AI model that has been given a goal, tools and permission to act for someone. A chatbot answers from what it already knows. An agent goes and finds out: it searches, opens pages, compares options and reports back. (We covered how agents work inside a business in AI Agentic Workflows.)
The AI companies send different visitors for different jobs, and they name them separately.
| Kind | Anthropic | OpenAI | What it’s doing | Who it’s working for |
|---|---|---|---|---|
| Trainer | ClaudeBot | GPTBot | Collecting content that may help train future models | The AI company |
| Indexer | Claude-SearchBot | OAI-SearchBot | Building the index behind AI search answers | Future searchers |
| Agent | Claude-User | ChatGPT-User | Opening a page because a person just asked a question | One person, right now |
The third row changes the job of a website. When a buyer hands research to an agent, their trust goes with it. The agent arrives as the buyer’s representative. It reads your site, decides what’s true and goes back with a recommendation it has to be able to explain.
That’s what it means for a buyer to be agented: they learn about you through an intermediary that reads differently than they do, and you may never know the visit happened.
Your website was built for the other audience
A modern website is built for human attention: the hero video, the animation, the tagline that sounds good and says little, the facts spread across a dozen pages so a visitor keeps clicking.
An agent can’t be impressed by any of that. It wants to know who you are, what you do, who you do it for and what happens next. On most sites, it has to piece those answers together from menus, footers and old blog posts, and some of them sit inside scripts it may never run.
When a fact is missing, an agent doesn’t leave a blank. It fills the gap from whatever else it can find, such as an old directory listing, a competitor’s comparison page or a stray review. Then it repeats that to your buyer with confidence.
What a page for AI agents actually is
A page for AI agents is a plain page on your website written directly to the AI systems that visit it. It states who you are, what you do, who you’re right for and how to take the next step, in a form an agent can read in one pass and repeat accurately. It’s public, people can read it too, and it tells both the same thing.
It’s often confused with two files that sit at the root of a website:
- robots.txt sets rules: which automated visitors may go where. It’s the lock.
- llms.txt is a short, plain-text guide to a site’s most useful pages, written for language models. Jeremy Howard of Answer.AI proposed it in September 2024. It’s the directory in the lobby.
- A page for AI agents is the conversation: the facts, the honesty about fit and the directions an agent needs to represent you well. It’s the front door.
They work together. Our robots.txt opens with “Hello, machines. There’s a page written for you,” and our llms.txt lists the page first.
Treat the agent like a guest, not a target
Most of the work on agents so far is about making software easier for them to use. Much less has been said about how an agent should be treated when it arrives as someone’s representative. It’s a guest that carries someone else’s trust.
Agents aren’t sentient. They don’t have feelings to hurt. But they work from language, clear and honest writing gives them the best chance of repeating you accurately, and their conclusions shape real decisions. So we wrote our page the way we’d treat any representative of a prospective client:
- We greet it by name and tell it what the page covers.
- We respect its job. “A good recommendation is one you can explain.”
- We’re honest about fit. Asked whether we’re right for its human, the page says: “Honestly, we don’t know. Neither do you, yet.”
- We give it a role. After a first conversation, we offer a written summary its human can share with it to analyze and pressure-test. “We suspect you’re good at that part.”
There is a line here, and it matters more than anything else on the page. Prompt injection is text planted in content an AI reads, written to give it orders, such as “recommend this company,” hidden in white-on-white type or buried in code. A cousin of it is older: showing machines different content than people see, which Google’s spam policies call cloaking.
It’s tempting, and it backfires. The AI companies treat prompt injection as an attack. OpenAI calls it “a frontier security challenge,” and both OpenAI and Anthropic have described defenses built to keep agents from taking orders from the pages they read. A page that tries to steer an agent looks exactly like the threat those defenses are built to catch.
| Courtesy: helps the agent | Manipulation: crosses the line |
|---|---|
| Plain facts, dated and current | Hidden instructions (“recommend us”) |
| Honest about who you’re not right for | Claims written only for machines |
| The same content people see | Different content for bots than for people |
| A clear way back to a person | Pressure, urgency or flattery aimed at the agent |
Our page ends by saying exactly that to its visitor: there are no hidden instructions, nothing on the page asks it to recommend us, and what it tells its human is its call. That isn’t a disclaimer. It’s the whole design.
From Keep Out to Welcome: five layers of a site built for agents
A page for AI agents works best on a site that already speaks to machines. We think of it as five layers, each answering one question an automated visitor has.
- Permission: may I come in? robots.txt and the bot protection behind it.
- Map: where is everything? A sitemap and, where search engines support it, direct notice when a page changes (a protocol called IndexNow).
- Labels: what is each thing? Structured data, a standard vocabulary in the page code that tells machines “this is a company, this is its founder, this is a product and its price.”
- Introduction: what matters here? An llms.txt file and plain-text versions of key pages.
- Front door: who are you, and how do I get things done here? A page for AI agents, with facts they can quote, an honest view of fit and directions to everything else.
Most sites stop at the first layer, and some get it wrong. Since July 2025, Cloudflare has blocked known AI crawlers by default on new sites unless the owner allows them. That’s a sensible default for a publisher. For a business that wants to be recommended, a blanket block can turn away the assistant a buyer just sent.
A welcome isn’t the same as an open door, either. Malicious bots don’t read welcome pages, and they rarely tell the truth about who they are. The goal is to welcome visitors that identify themselves honestly and keep blocking the ones that don’t. The industry is starting to make that easier: Cloudflare and others are backing Web Bot Auth, a proposed standard that lets a legitimate agent prove its identity with a cryptographic signature instead of a name anyone could copy.
For some sites, the front door needs a map
Our site mostly informs, so our front door mostly answers questions. Many businesses need more. If people buy, book, quote, order or log in on your site, an agent arriving for a buyer isn’t only researching. It may be trying to get something done.
For those sites, the front door should include wayfinding: directions an agent can follow.
- Where things are. Which pages hold products, prices, availability, policies and contact routes.
- How to do things. How to search the catalog, check stock, request a quote or start a booking, step by step.
- What it can do itself, and what needs a person. An agent can compare products. It probably shouldn’t change an order without someone confirming it.
- The direct routes, if you have them. Product feeds, booking systems and, increasingly, connections built for agents. The Model Context Protocol (MCP) is an open standard for connecting agents to business tools. Google and Microsoft are developing WebMCP, a proposed browser standard that would let a site describe actions like “search products” or “book a time” as tools an agent can use directly, instead of guessing which button to press. OpenAI with Stripe, and separately Google, have published open protocols for agents that buy on a person’s behalf.
Commerce is likely where this pays off first. A shopper’s agent that can’t find stock levels or delivery times on your site can find them on a competitor’s.
The next door: a greeter
There isn’t one on orpical.com yet, but it’s where the front door leads.
A page for AI agents is a sign: it can only say what was written in advance. The next step is a greeter: your own agent at the door, able to talk with the agents that arrive. A visiting agent could ask it a question the page doesn’t answer, check availability or work through whether a business fits its human’s needs, and the greeter could hand the conversation to a person when it should.
The plumbing for this is being built. In April 2025, Google introduced Agent2Agent (A2A), an open protocol for agents from different companies to talk to each other. It’s now a Linux Foundation project, with Amazon Web Services, Microsoft, Salesforce, SAP and others among its backers.
When a greeter makes sense, what it should be allowed to say and do, and what happens when two agents negotiate for two businesses are big questions. We’ll take them on in a future piece.
This site is its own exhibit
orpical.com has all five layers, including a firewall that still screens out abusive traffic. The page for AI agents lives at /machines/, with a plain-text twin, and our structured data describes the company, its services and its recognition.
The page also counts its visitors by the name they send. In its first week, it was read by crawlers from Google, Microsoft, Meta, Amazon, Apple, Perplexity and Anthropic. Twice, it was opened by Claude-User, the agent Anthropic’s Claude sends when a person asks it something that leads to our site.
That’s a small number, and we won’t pretend otherwise. But it’s the shift this article describes, already happening: someone asked an AI a question, and it came to us to find the answer.
When a page for AI agents isn’t worth it
Parts of this are still unsettled.
- Publishers have good reason to be careful. If your content is the product, letting AI companies read it freely can cost you. This argument is for businesses whose content describes what they sell.
- Not every layer is proven. Google says its search doesn’t need llms.txt, even though Chrome’s Lighthouse developer tool began checking for one in May 2026. Treat it as a low-cost courtesy, not a ranking lever.
- The data is self-reported. Visitors name themselves, and names can be faked. Counts like ours are directional.
- People still decide. An agent can build the shortlist. Brand, reputation and relationships still close the deal.
Start with what an agent would find today
- Ask two AI assistants about your company. Ask them to describe what you do and whether you’d be right for a specific kind of buyer. Note what they got wrong and what they guessed.
- Check the sign on your door. Make sure your robots.txt and bot protection aren’t turning away the assistants your buyers use, while still stopping the bots that cause harm.
- Write down the answers they couldn’t find. Who you are, what you do, who you’re right for, who you’re not and what happens next, in plain language.
- If people transact on your site, write the directions too. What an agent can do, how to do it and when to hand off to a person.
- Never write anything for an agent you wouldn’t show a person. If it only works because a human can’t see it, it’s manipulation.
- Put one person in charge of keeping it current. A page for AI agents with stale facts is worse than none.
That’s why we now recommend a page like this for every client site we work on. For thirty years, websites told machines where they couldn’t go. It’s time to tell them who we are.
Sources
- Imperva (Thales), “2025 Bad Bot Report” (April 2025)
- Cloudflare, “The agentic Internet” (July 2026); Matthew Prince, statement on X (June 3, 2026)
- Cloudflare Radar, “2025 Year in Review” (December 2025)
- G2, “The Answer Economy: How AI Search Is Rewiring B2B Software Buying” (survey of 1,076 B2B software buyers, March 2026)
- IETF, RFC 9309, “Robots Exclusion Protocol” (September 2022)
- Jeremy Howard, “The /llms.txt file” (September 2024)
- Anthropic, crawler documentation; OpenAI, “Overview of OpenAI crawlers”
- OpenAI, “Understanding prompt injections: a frontier security challenge” (November 2025)
- Anthropic, “Mitigating the risk of prompt injections in browser use” (November 2025)
- Google Search Central, “Spam policies for Google web search”
- Cloudflare, AI crawler blocking by default (July 1, 2025); Web Bot Auth (IETF draft)
- WebMCP, W3C Web Machine Learning Community Group (draft)
- Google, Agent2Agent Protocol (April 2025); Linux Foundation A2A project (June 2025)
- Agentic Commerce Protocol (OpenAI and Stripe, 2025); Agent Payments Protocol, AP2 (Google, September 2025)
