My robots.txt says AI crawlers are allowed so why might my site still be invisible to them?

A robots.txt with allow rules only controls what a crawler is permitted to fetch, not what it can find. If your sitemap never lists a page, or a CDN like Cloudflare is serving its own managed robots.txt in front of your real one, an agent can be fully allowed and still never discover the page exists. Allow rules and discoverability are two separate problems, and fixing one does not fix the other.

How can server log analysis surface AI agent visits that Google Analytics completely misses?

Every AI crawler request appears in a website's raw server logs as an HTTP line with a distinct user agent, such as GPTBot, ClaudeBot or PerplexityBot, whether or not a browser ever ran JavaScript. Searching a log export for those names shows exactly which agents reached your site and when, which is the only reliable way to confirm access, since GA4 never sees a server-to-server request. Most hosts and CDNs will pull a log slice on request.

I added an llms.txt file, how do I know if any AI system has actually fetched it?

Check your raw server logs for a request to /llms.txt from a named AI user agent; if the line is never there, nothing has fetched it yet. llms.txt is a voluntary convention proposed in 2024, not a standard every AI system reads, so an unfetched file doesn't mean it's broken, only that nothing you use currently looks for it. Neither Search Console nor GA4 will tell you either way.

If AI crawlers take my content and send no traffic back, why let them in at all?

It depends which crawler. Training crawlers like GPTBot and CCBot gather content and send nothing back, while retrieval crawlers like OAI-SearchBot and Perplexity-User are the ones that can actually cite and link to your business inside an AI answer. Blocking everything indiscriminately protects you from the first group but closes the door on the second too, which is real referral value you'd otherwise never see in GA4 anyway.

Is llms.txt actually required, or is it optional busywork?

It's optional. No major AI system currently requires it, and it won't fix a sitemap or robots.txt that isn't doing its job. It's worth the low effort as a forward-looking signal, but a working robots.txt and a complete sitemap matter far more right now.

Does llms.txt need to be submitted anywhere the way a sitemap goes into Search Console?

No. Nothing currently consumes llms.txt the way Search Console consumes a submitted sitemap; it simply sits at yoursite.com/llms.txt for any AI system built to look for it. There's no submission step and no confirmation screen telling you it's been read.

Beyond robots.txt and a sitemap, what actually makes my pages readable to an AI agent?

Structured data, usually schema markup written as JSON-LD. A sitemap and robots.txt help an agent find and fetch a page; schema labels the facts on it, such as your business name, address, opening hours and phone number, so the agent can lift them without guessing from the layout. LocalBusiness schema is the most useful type for a local business, and an AI discovery file like llms.txt does not replace it. Getting found and being readable are two separate jobs, and schema handles the second one.

How do I tell a legitimate AI crawler from a scraper pretending to be one?

Check the request against the crawler's published IP ranges, not just the user agent, which anyone can copy. OpenAI, Anthropic, Perplexity and Google all publish the IP addresses their bots use, and a genuine GPTBot or ClaudeBot line in your server log will resolve to one of them. A request claiming to be an AI agent from an address outside those ranges is almost always a scraper wearing the name. Your host or CDN can confirm the source IP for any log line you flag.

Why Your Website Is Invisible to AI Agents and Your Analytics Will Never Show It

6 August 2026 ai for business

Why Your Website Is Invisible to AI Agents Even in Greenock

A website is invisible to AI agents when its sitemap, the file listing every URL a site wants crawled, and its robots.txt, the file telling crawlers what they may fetch, never list the pages that actually exist, even though every page still returns a working HTTP 200. AI crawlers fetch raw HTML without running JavaScript, so the visit never fires a tracking tag. Google Analytics, like every JavaScript-based analytics tool, has no way to record a visit that, from its point of view, never happened.

That is worse than a broken page. A broken page eventually throws an error somebody notices. This failure mode never does.

We found it on our own site first. Online Marketing Group runs glasgow.localbusiness.space, a directory of 373 real Glasgow businesses. Auditing it on 2026-08-04 the way we audit a client’s site, we found a sitemap listing 6 URLs, against 373 real business listings.

Beside it sat a robots.txt of 1,248 bytes that turned out, once we actually read it, to be 100% comments. It was Cloudflare’s own boilerplate content-signals text, zero real directives, no Sitemap: line, served because our origin had never served one of its own. Every surface answered with a clean 200. 383 of our own 389 pages sat in no discovery surface an AI agent could find.

If a directory we built to showcase our own methods could hide almost all of its pages behind a clean 200, most sites we audit carry some version of the same blind spot. Get in touch to book a free audit today and we’ll show you exactly what the agents can and can’t see on yours.

AI Crawler Visibility Sits Underneath Your SEO Not Inside It

Search rankings and AI agent visibility get sold as the same skill in different packaging. They are not. Google Search still crawls and indexes pages the traditional way, as its Wikipedia entry notes, and its AI Overviews, the summarised answers now sitting above search results, draw on that index. A purpose-built AI agent like OAI-SearchBot, PerplexityBot or ClaudeBot arrives cold instead, with no ranking history, and reads only what your robots.txt and sitemap hand it directly.

Why Your Website Is Invisible to AI Agents Even in Greenock

Miss that layer and page quality stops mattering. Content going undiscovered is already the norm, not the exception: Ahrefs, in a study of roughly a billion web pages, found 96.55% of them get no organic search traffic from Google at all. And on the AI side, separate Cloudflare research puts real robots.txt adoption among the top 10,000 domains at only around 37%.

How We Made Our Own Directory AI-Readable

Online Marketing Group is a Greenock digital marketing agency first. The Glasgow directory is a side project we run to prove our own methods before we sell them, which is exactly why finding this on it stung.

Keith Malone puts the underlying problem better than any technical write-up: “Many companies spend millions on advertising but leave significant revenue on the table.” He was talking about weak websites and unoptimised profiles, but a sitemap that lists a handful of pages for a 373-business directory is the same sentence written in code.

Getting Found Is Only Half the Job

Discovery gets an agent to your page. Reading it is a second job the usual mention-and-backlink advice skips entirely. An agent that lands on a business listing still has to pull the name, address, opening hours and phone number out of raw HTML, with no browser rendering the layout a person would rely on.

Structured data is what closes that gap. Schema markup, the tagged code that labels each fact on a page so a machine can read it without guessing, is the difference between a page an agent reaches and one it can actually use. An AI discovery file like llms.txt only points an agent at your content; schema tells it what each part of that content means. So the order matters: fix the sitemap and robots.txt first, then add LocalBusiness schema to every page, because a URL an agent can find but not parse is barely better than one it never finds.

Why Your Analytics Will Never Show You This Problem

Google Analytics counts a visit the moment a browser runs a JavaScript snippet. An AI agent fetching your robots.txt or sitemap does neither. It makes a server-to-server request, reads the response, and moves on.

Nothing gets hidden, because a crawl of a discovery file was never a session to begin with. The only place that request is ever recorded is your raw server log, a file most business owners have never opened.

One site owner found this out the hard way, commenting under a popular tutorial on adding an llms.txt file after going looking for proof it had worked: “Is there any evidence that any actual AI uses this file? I’m not seeing it from the web logs. The file is never requested.” They had done everything the checklists recommend and still had to dig through raw logs, not a dashboard, to get an honest answer.

Cloudflare Fills the Gap Your Own Server Leaves Empty

What to Check on Your Own Website This Week

If your site sits behind Cloudflare and your origin has never served a real robots.txt, Cloudflare quietly serves a managed robots.txt for you. It’s legal preamble about content signals, granting nothing and forbidding nothing, sitting at the URL where a real file should be.

The moment your origin serves a genuine file, Cloudflare’s stand-in steps aside entirely. Google’s John Mueller, the company’s long-time search advocate, has said the underlying Content Signals directive currently has no effects whatsoever for any crawler or LLM. A label is not a lock.

What to Check on Your Own Website This Week

The fix on our own directory took one day. Our sitemap went from 6 URLs to 389, covering all 373 listings plus the site’s static pages. Our robots.txt went from 1,248 bytes of comments to 631 bytes carrying 11 allow groups, 10 of them naming AI crawlers directly, GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot, Google-Extended for Gemini and AI Overviews, and CCBot among them, with zero disallows and a Sitemap line pointing straight at the real one.

Check three things on your own site this week. Open yoursite.com/robots.txt and read it end to end; if it’s mostly legal text with no Sitemap: line, your origin is likely serving nothing and a CDN is filling the silence. Open yoursite.com/sitemap.xml and count the URLs against how many pages you actually have.

Then ask whoever hosts your site for a slice of raw server logs and search for GPTBot, ClaudeBot or PerplexityBot. If those lines exist, agents are reaching you, though it’s worth checking a suspicious one resolves to the crawler’s published IP range, since the user agent alone is trivial to fake. If they never appear, contact us today and we’ll audit what’s happening to your enquiries, the same way we audit paid search, because a site returning 200 everywhere can still be functionally empty to the systems now routing a growing share of buying decisions.

Found this useful? Add omgmarketinguk.com as a preferred source on Google so our latest guides show up first for you in Search and AI Overviews.

← Back to the blog

Double Your Money Guarantee At least 200% ROI
Pay Per Lead Only pay for real leads
No Retainers No fat monthly fees
No Lock-In Contracts Stay because it works

How The Guarantee Works

A 200% Return — Or We Keep Working

No vague promises and no walking away when the campaign ends. Here's exactly how it works.

Apply & Qualify

Tell us about your business. We only take on local businesses we're genuinely confident we can grow — that's what lets us stand behind the guarantee.
We Go To Work

Pay per lead, no retainer, no lock-in. We build and run the campaigns that bring qualified local customers to your door — you only pay for real leads.
We Don't Stop Until You've Doubled Your Money

For every £1 you invest, we work to return £2 in revenue. If you haven't hit a 200% return yet, we keep working — at no extra cost — until you do.

Apply & Qualify

See If You Qualify

The Double Your Money Guarantee isn't offered to everybody — only to local businesses we're confident we can grow. Tell us about yours and we'll come back with a straight answer, plus a no-obligation plan showing where your leads are leaking.

Free, no obligation — no card, no contract
A real reply from a real person
Built for local businesses

Questions?

Frequently Asked Questions

Straight answers on how we work, what we guarantee, and what it costs.

A robots.txt with allow rules only controls what a crawler is permitted to fetch, not what it can find. If your sitemap never lists a page, or a CDN like Cloudflare is serving its own managed robots.txt in front of your real one, an agent can be fully allowed and still never discover the page exists. Allow rules and discoverability are two separate problems, and fixing one does not fix the other.
Every AI crawler request appears in a website's raw server logs as an HTTP line with a distinct user agent, such as GPTBot, ClaudeBot or PerplexityBot, whether or not a browser ever ran JavaScript. Searching a log export for those names shows exactly which agents reached your site and when, which is the only reliable way to confirm access, since GA4 never sees a server-to-server request. Most hosts and CDNs will pull a log slice on request.
Check your raw server logs for a request to /llms.txt from a named AI user agent; if the line is never there, nothing has fetched it yet. llms.txt is a voluntary convention proposed in 2024, not a standard every AI system reads, so an unfetched file doesn't mean it's broken, only that nothing you use currently looks for it. Neither Search Console nor GA4 will tell you either way.
It depends which crawler. Training crawlers like GPTBot and CCBot gather content and send nothing back, while retrieval crawlers like OAI-SearchBot and Perplexity-User are the ones that can actually cite and link to your business inside an AI answer. Blocking everything indiscriminately protects you from the first group but closes the door on the second too, which is real referral value you'd otherwise never see in GA4 anyway.
It's optional. No major AI system currently requires it, and it won't fix a sitemap or robots.txt that isn't doing its job. It's worth the low effort as a forward-looking signal, but a working robots.txt and a complete sitemap matter far more right now.
No. Nothing currently consumes llms.txt the way Search Console consumes a submitted sitemap; it simply sits at yoursite.com/llms.txt for any AI system built to look for it. There's no submission step and no confirmation screen telling you it's been read.
Structured data, usually schema markup written as JSON-LD. A sitemap and robots.txt help an agent find and fetch a page; schema labels the facts on it, such as your business name, address, opening hours and phone number, so the agent can lift them without guessing from the layout. LocalBusiness schema is the most useful type for a local business, and an AI discovery file like llms.txt does not replace it. Getting found and being readable are two separate jobs, and schema handles the second one.
Check the request against the crawler's published IP ranges, not just the user agent, which anyone can copy. OpenAI, Anthropic, Perplexity and Google all publish the IP addresses their bots use, and a genuine GPTBot or ClaudeBot line in your server log will resolve to one of them. A request claiming to be an AI agent from an address outside those ranges is almost always a scraper wearing the name. Your host or CDN can confirm the source IP for any log line you flag.