---
title: "Can AI Engines Read Your Site? Run This Fetch Test Before Buying a Visibility Tool"
description: "Test 10 key pages and 5 controls for crawler access, readable HTML, verified visits and live quotes before paying for an AI visibility fix."
url: "https://groas.com/post/can-ai-engines-even-read-your-site-a-one"
image: "https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/c2a8943d-44b8-46f0-b4ce-5df164013c42.png"
published: "2026-10-16T05:49:23.116Z"
modified: "2026-10-16T05:49:23.184Z"
---

October 16, 2026 · 10 min read

# Can AI Engines Read Your Site? Run This Fetch Test Before Buying a Visibility Tool

[Alexander PerelmanHead Of Product @ groas](https://groas.com/author/alexander-perelman)[LinkedIn](https://www.linkedin.com/in/alexander-433793253/)

![Tiny inspectors with keys test doors on a model street—some bricked shut, some empty, one lit—while a giant gift-wrapped tool sits ignored.](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/c2a8943d-44b8-46f0-b4ce-5df164013c42.png)

In this article

1. [Question: is the page blocked, unreadable or simply not quoted?](#question-is-the-page-blocked-unreadable-or-simply-not-quoted)
2. [Setup: choose 10 money pages and 5 pages you will leave alone](#setup-choose-10-money-pages-and-5-pages-you-will-leave-alone)
3. [Test 1: read robots.txt token by token](#test-1-read-robotstxt-token-by-token)
4. [Test 2: inspect the HTML the first fetch returns](#test-2-inspect-the-html-the-first-fetch-returns)
5. [Test 3: verify visits instead of trusting bot costumes](#test-3-verify-visits-instead-of-trusting-bot-costumes)
6. [Test 4: request a live quote, not a brand summary](#test-4-request-a-live-quote-not-a-brand-summary)
7. [Test 5: edit five readable pages, then retest on day 14](#test-5-edit-five-readable-pages-then-retest-on-day-14)
8. [Read the result before choosing the bill](#read-the-result-before-choosing-the-bill)
9. [What a tool must do that this afternoon cannot](#what-a-tool-must-do-that-this-afternoon-cannot)

Most businesses shopping for _AI visibility_ are about to buy the wrong fix. They sign for a monthly tool or a 60-page audit before checking whether ChatGPT, Claude and Perplexity can fetch their key pages, let alone quote them.

So test that first. Pick the pages, write down what you expect, then check what the engines can receive: your robots rules, your raw HTML without JavaScript, your verified bot visits and the words the engines quote back. Keep a control group you leave alone. I used to run ad copy tests this way because, without a control, you will credit the last change you made for whatever happened next.

## Question: is the page blocked, unreadable or simply not quoted?

This protocol separates three outcomes that call for different spending:

- **Blocked:** a crawler cannot get the page.
- **Readable but not quoted:** the page is accessible, but the answer does not appear in the live test.
- **Readable and quoted:** this test has not found a technical access problem.

Do not roll those into one visibility score. A robots rule, a JavaScript-dependent price and a page that buries its answer are different problems. Test which one you have before you buy a fix.

## Setup: choose 10 money pages and 5 pages you will leave alone

Pick **10 money pages** that would cost you something if AI answers skipped them: pricing, core services, top product categories and the two or three blog posts that actually drive pipeline. Then pick 5 stable controls you will not edit during this test. An About page, an old post or a careers page will do. The controls will not be perfect matches for your money pages; they give you a check on whether quotes are shifting even where you changed nothing.

Before opening a tool or an AI chat, make a sheet with a row for each money page and control. For each page, write your prediction on three lines:

1. Can the relevant crawler fetch it?
2. Can it read the key facts without JavaScript?
3. Will a live session quote those facts today?

Write yes or no. Then leave room for the observed result, the exact failing URL, the response you received and the date. If you check results first and decide what you expected afterward, you will find exactly what the tool vendor wanted you to find.

My working expectation for a small business site is that 2 to 3 of the 10 money pages will have an access problem, 4 to 5 will be readable but say little worth quoting, and 1 or 2 will be fine. That is a prediction, not a finding about your site. Config creates blocks; thin or buried answers create a different kind of silence. Prove me wrong on your pages.

## Test 1: read robots.txt token by token

Open `yourdomain.com/robots.txt`. Do not stop when you find one line about AI. Check these user agents individually: `GPTBot`, `OAI-SearchBot`, `ChatGPT-User`, `ClaudeBot`, `Claude-SearchBot`, `Claude-User`, `PerplexityBot` and `Perplexity-User`. Check `Google-Extended` separately as a training-related control. A broad `User-agent: *` rule can matter too; [crawler references still flag blanket disallows in the wild](https://surfacedby.com/blog/ai-bot-crawlers-reference).

The distinction between agents matters. [OpenAI separates GPTBot from its search and user-fetch crawlers](https://geolikeapro.com/blog/openai-crawlers-comparison). If your policy is to restrict training, do not assume the same decision should apply to `OAI-SearchBot` and `ChatGPT-User`. Those are the access paths this citation test needs to examine. What the deck calls “blocking AI scraping,” I call a setting you had better read before flipping.

Record the rule that applies to each agent and page path. For this test, access for the search and user-fetch agents is a pass; a disallow that covers a money page is a fail. A clean robots file is not the end of Test 1, though. [Cloudflare can block AI crawlers independently of robots.txt](https://www.infosecurity-magazine.com/news/cloudflare-blocks-ai-web-scraping/). If a later request gets a 403, check the layer in front of the site before blaming the page.

**Practical takeaway:** write down the exact agent, path and rule. “Robots looks fine” is not a test result.

## Test 2: inspect the HTML the first fetch returns

Robots.txt can allow a request and still deliver a page with no useful answer in it. A pricing page built around JavaScript can look complete in your browser while its initial HTML contains little more than an app shell. [Googlebot has rendered with evergreen Chromium since 2019](https://whitebunnie.com/blog/javascript-seo-checklist/); do not take a Google ranking as proof that every other fetcher receives your rendered prices and specs.

Run three checks on each money page:

1. **View Source.** Search the initial HTML for the price, spec or core claim a visitor needs. Use the page’s actual answer, not a generic keyword.
2. **Disable JavaScript and reload.** Note whether the headings and key facts remain visible. If they vanish, record a readability fail.
3. **Fetch the response body.** [Use `curl` with a normal browser user agent and again with a crawler user agent](https://metricvaultai.com/blog/how-to-check-if-ai-bots-can-actually-crawl-your-site). Compare the status and the full body, not just the status code. A 200 does not tell you whether the answer arrived.

Keep the results separate. A browser-user-agent fetch that succeeds while a crawler-user-agent fetch fails points you back toward access controls. Two successful fetches with an empty answer point toward rendering. **The key fact needs to be in the initial response for this test to pass.** If it exists only after scripts run, ask a developer about server-side rendering or static generation before paying a subscription to watch the problem persist.

## Test 3: verify visits instead of trusting bot costumes

Pull 30 days of server or CDN logs, if you have them, and filter for the agents from Test 1. A user-agent string is a claim, not an identity: anyone can call a request `GPTBot`. Compare the IPs of claimed visits with the providers’ published ranges. OpenAI publishes lists at `openai.com/gptbot.json`, `searchbot.json` and `chatgpt-user.json`; Anthropic lists crawlers at `claude.com/crawling/bots.json`; Perplexity publishes a `perplexitybot.json` file and a Perplexity-User file. Use reverse DNS where it helps you check a claimed identity, rather than treating the user-agent field as verification by itself.

[One 30-day log study found that 6 to 22% of claimed AI-bot hits failed verification](https://surfacedby.com/blog/ai-bot-crawlers-reference). You need not assume the same share on your site. The point is simpler: if a claimed hit is not genuine, counting it as evidence of crawler access can send you after the wrong fix.

Perplexity deserves a separate note. [Perplexity describes Perplexity-User as an agent that generally ignores robots.txt](https://www.anagram.ai/blog/ai-crawlers-explained-gptbot-claudebot-perplexitybot-and-how-to-let-them-in-2026), while reports of undeclared crawler behavior make its footprint harder to infer from a robots rule alone. Check WAF and server behavior as well as the file. For an example of the kind of request to look for, this [AI SEO audit teardown](https://groas.com/post/your-ai-technical-seo-audit-is-60-pages) shows Claude-User robots.txt fetches in server logs.

Record verified visits, rejected requests and any 403s by page and agent. **No verified hit in your 30-day window is a reason to investigate, not proof that access is blocked.** A live fetch in Test 4 can answer the immediate access question; logs can tell you whether the problem has shown up in ordinary visits.

## Test 4: request a live quote, not a brand summary

Choose a money page that passed the access and raw-HTML checks. In fresh ChatGPT, Claude and Perplexity sessions with browsing available, paste its exact URL and use the same prompt each time:

`Read this URL and quote its price, core spec and main claim word for word. If one is absent or you cannot read the URL, say so.`

Replace “price” or “spec” with the relevant fact if the page does not contain one. Save the exact quote, omission or refusal, along with the session date. You are testing whether the live session returns words from that URL, not whether a model has heard of your brand. A plausible paraphrase without the requested words does not pass the quote test.

Expect the access result and quote result to diverge. Access makes a page eligible to be read; it does not make its answer compelling or easy to extract. [Self-contained answers, specific numbers and dates, initial-HTML JSON-LD and third-party corroboration are among the signals discussed for quotable pages](https://www.anagram.ai/blog/ai-crawlers-explained-gptbot-claudebot-perplexitybot-and-how-to-let-them-in-2026). I used to tell clients a clear page was enough. I was wrong. A page can fetch cleanly and still bury the number three scrolls down while another page states it in sentence one.

Note which engine quoted the page and which did not. **A fetch pass followed by a quote fail sends you toward the answer on the page, not back to robots.txt.**

## Test 5: edit five readable pages, then retest on day 14

Do not run a content experiment on pages you have already found to be blocked or empty in the initial HTML. Fix access or rendering first, confirm the pages pass and take a new baseline. Otherwise, a later quote will not tell you which change mattered.

For pages that pass, split the 10 money pages into two groups of five. On Group A, make one defined edit: put a self-contained answer block in the first 100 words of the initial HTML. State the relevant price, number or spec plainly and include matching JSON-LD in that response. Leave Group B and all 5 controls alone. Keep a record of the exact edit and its date.

Re-run Tests 1 through 4 on day 14, using the same URLs, facts and live prompt. Compare the results in this order:

1. Did access or raw HTML change anywhere you did not intend it to?
2. Did untouched controls start or stop getting quoted?
3. Did untouched Group B move?
4. Did edited Group A change differently from both untouched groups?

That last comparison is the useful one. A before-and-after screenshot cannot distinguish your edit from background movement. This is the controlled addition to the [fetch-check sequence](https://metricvaultai.com/blog/how-to-check-if-ai-bots-can-actually-crawl-your-site), not a promise that every page will be quoted within 14 days. If nothing moves, mark the result inconclusive rather than selling yourself a win. Say you spend $20k a month on content and someone proposes a $1,500-a-month visibility tool using an uncontrolled screenshot. This sheet is the reply.

## Read the result before choosing the bill

**Blocked is an access fix, not a citation-software purchase.** If a relevant robots rule disallows the path or your WAF returns a 403 to the fetcher, hand a developer the URL, agent, rule and failed request. Change the setting, then confirm the request succeeds. Do not pay a monthly fee to monitor a door that stays shut.

**Readable but not quoted calls for a page edit before another audit.** If the initial HTML contains the facts and a live session reads the URL but will not quote the answer, look at where and how the page states it. Try the Group A answer block; leave the comparison pages alone and measure again. If the edited pages move while the others do not, you have a reason to roll that edit wider. If they do not, do not pretend another robots check will solve it.

**Readable and quoted means this test has not found a technical access problem.** You may still care how often the page gets cited, but a technical audit should have to identify something more specific than “AI cannot see you.” Keep the working page as a reference when you assess the others.

![Three test outcomes: blocked, readable but not quoted, and readable and quoted](https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/682d8916-4e21-4be4-beaa-fd2697ddff6f.png)

## What a tool must do that this afternoon cannot

A tool earns its fee if it keeps checking verified fetches after this afternoon, tracks live quotes against the pages you left alone and shows what changed on each page and why. It should not charge you to announce that robots disallows a fetcher or that JavaScript hides a price. You just checked both.

If a vendor leads with a 60-page audit and a monthly PDF, pass. I would rather pay for a system that carries out the retest and shows [every action with the reason behind it](https://groas.com/for-businesses). If you want to see the mechanics on a live page first, this [45-minute fetch check](https://groas.com/post/can-chatgpt-claude-and-perplexity-actual) follows the same sequence. That is the appeal of groas on the organic side: continuous execution inside your guardrails, with a human accountable for direction, instead of another report for you to interpret.

Your next step comes from the sheet you wrote before testing. Blocked: send the developer the failed request, fix access and confirm it with a fresh fetch. Readable but not quoted: edit Group A, protect the comparison pages and see what day 14 changes. Readable and quoted: stop shopping for an access fix you cannot show you need. File the sheet either way. The point was never the score. It was knowing whether you needed a tool, a developer for one afternoon, or nothing, before anyone sent you an invoice.

## Frequently Asked Questions

### What is the difference between a page being blocked, readable but not quoted, and readable and quoted?

Blocked means a crawler cannot get the page at all. Readable but not quoted means the page is accessible but the answer does not appear in the live test. Readable and quoted means the test has not found a technical access problem. Each outcome calls for different spending.

### When testing whether AI engines can quote my page, what counts as passing?

The live session must return words from that URL word for word, using a prompt like asking it to read the URL and quote its price, core spec and main claim. A plausible paraphrase without the requested words does not pass the quote test. Save the exact quote, omission or refusal, along with the session date.

### If a page passes the fetch test, does that guarantee AI engines will quote it?

No. Access makes a page eligible to be read; it does not make its answer compelling or easy to extract. A page can fetch cleanly and still bury the number three scrolls down, while another page states it in sentence one. A fetch pass followed by a quote fail points to the answer on the page, not to robots.txt.

### How should I test whether a content edit actually improved AI quoting?

Split 10 money pages into two groups of five. On Group A, add a self-contained answer block in the first 100 words of the initial HTML, stating the relevant price, number or spec plainly with matching JSON-LD. Leave Group B and all 5 controls alone, then re-run the same tests on day 14 and compare all groups in order.

### Why do I need control pages when testing AI visibility?

Without a control, you will credit the last change you made for whatever happened next. A before-and-after screenshot cannot distinguish your edit from background movement. Untouched controls and a comparison group let you see whether only the edited pages changed.

### If AI crawlers are blocked from my site, should I buy a visibility tool?

No. Blocked is an access fix, not a citation-software purchase. Hand a developer the URL, agent, robots rule and failed request, change the setting, then confirm the request succeeds. Do not pay a monthly fee to monitor a door that stays shut.

### My pages are crawlable but AI engines never quote them — what should I do?

Try a page edit before buying another audit. If the initial HTML contains the facts and a live session reads the URL but will not quote the answer, look at where and how the page states it. Add the Group A answer block, leave comparison pages alone, measure again, and roll the edit wider only if the edited pages move while the others do not.

### What should a paid AI visibility tool actually do for me?

A tool earns its fee if it keeps checking verified fetches after your initial test, tracks live quotes against the pages you left alone, and shows what changed on each page and why. It should not charge you to announce that robots disallows a fetcher or that JavaScript hides a price, because you just checked both yourself.

## Pay For Results, Not For Hours

Businesses buy the outcome, agencies resell it, and groas answers for it either way.

[See If You Qualify](https://groas.typeform.com/to/xC1bQNUT)

## Structured data

```json
{"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://graph.groas.com/entity/groas","name":"groas","url":"https://groas.com/","logo":{"@type":"ImageObject","url":"https://groas.com/icon-512.png","width":512,"height":512},"sameAs":["https://www.linkedin.com/company/groas/"]},{"@type":"WebSite","@id":"https://groas.com/#website","name":"groas","url":"https://groas.com/","inLanguage":"en-US","publisher":{"@id":"https://graph.groas.com/entity/groas"}},{"@type":"WebPage","@id":"https://groas.com/post/can-ai-engines-even-read-your-site-a-one#webpage","url":"https://groas.com/post/can-ai-engines-even-read-your-site-a-one","name":"Can AI Engines Read Your Site? Run This Fetch Test Before Buying a Visibility Tool","description":"Test 10 key pages and 5 controls for crawler access, readable HTML, verified visits and live quotes before paying for an AI visibility fix.","inLanguage":"en-US","isPartOf":{"@id":"https://groas.com/#website"},"about":{"@id":"https://graph.groas.com/entity/groas"},"primaryImageOfPage":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/c2a8943d-44b8-46f0-b4ce-5df164013c42.png"},"breadcrumb":{"@id":"https://groas.com/post/can-ai-engines-even-read-your-site-a-one#breadcrumb"},"dateModified":"2026-10-16T05:49:23.184Z","mainEntity":{"@id":"https://groas.com/post/can-ai-engines-even-read-your-site-a-one#article"}},{"@type":"BreadcrumbList","@id":"https://groas.com/post/can-ai-engines-even-read-your-site-a-one#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://groas.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://groas.com/blog"},{"@type":"ListItem","position":3,"name":"Can AI Engines Read Your Site? Run This Fetch Test Before Buying a Visibility Tool","item":"https://groas.com/post/can-ai-engines-even-read-your-site-a-one"}]},{"@type":"BlogPosting","@id":"https://groas.com/post/can-ai-engines-even-read-your-site-a-one#article","url":"https://groas.com/post/can-ai-engines-even-read-your-site-a-one","mainEntityOfPage":{"@id":"https://groas.com/post/can-ai-engines-even-read-your-site-a-one#webpage"},"isPartOf":{"@id":"https://groas.com/blog#blog"},"headline":"Can AI Engines Read Your Site? Run This Fetch Test Before Buying a Visibility Tool","description":"Test 10 key pages and 5 controls for crawler access, readable HTML, verified visits and live quotes before paying for an AI visibility fix.","datePublished":"2026-10-16T05:49:23.116Z","dateModified":"2026-10-16T05:49:23.184Z","image":{"@type":"ImageObject","url":"https://pub-87da24ecbbfc4c3bad6875f3aa013712.r2.dev/generated-images/c2a8943d-44b8-46f0-b4ce-5df164013c42.png"},"author":{"@id":"https://groas.com/author/alexander-perelman#person"},"publisher":{"@id":"https://graph.groas.com/entity/groas"},"keywords":"test, page, pages, before, access, fetch, read, robots","wordCount":2133,"inLanguage":"en-US"},{"@type":"Person","@id":"https://groas.com/author/alexander-perelman#person","name":"Alexander Perelman","jobTitle":"Head Of Product @ groas","description":"Ex Goldman Sachs and Ex Stanford Computer Science","image":"https://groas.com/media/blog/4f17cac3a81acc224cc1b2cfab8f94bbc295a3089d13e8421125771ef8d4064f.jpg","worksFor":{"@id":"https://graph.groas.com/entity/groas"},"sameAs":["https://www.linkedin.com/in/alexander-433793253/"]},{"@type":"FAQPage","@id":"https://groas.com/post/can-ai-engines-even-read-your-site-a-one#faq","mainEntity":[{"@type":"Question","name":"What is the difference between a page being blocked, readable but not quoted, and readable and quoted?","acceptedAnswer":{"@type":"Answer","text":"Blocked means a crawler cannot get the page at all. Readable but not quoted means the page is accessible but the answer does not appear in the live test. Readable and quoted means the test has not found a technical access problem. Each outcome calls for different spending."}},{"@type":"Question","name":"When testing whether AI engines can quote my page, what counts as passing?","acceptedAnswer":{"@type":"Answer","text":"The live session must return words from that URL word for word, using a prompt like asking it to read the URL and quote its price, core spec and main claim. A plausible paraphrase without the requested words does not pass the quote test. Save the exact quote, omission or refusal, along with the session date."}},{"@type":"Question","name":"If a page passes the fetch test, does that guarantee AI engines will quote it?","acceptedAnswer":{"@type":"Answer","text":"No. Access makes a page eligible to be read; it does not make its answer compelling or easy to extract. A page can fetch cleanly and still bury the number three scrolls down, while another page states it in sentence one. A fetch pass followed by a quote fail points to the answer on the page, not to robots.txt."}},{"@type":"Question","name":"How should I test whether a content edit actually improved AI quoting?","acceptedAnswer":{"@type":"Answer","text":"Split 10 money pages into two groups of five. On Group A, add a self-contained answer block in the first 100 words of the initial HTML, stating the relevant price, number or spec plainly with matching JSON-LD. Leave Group B and all 5 controls alone, then re-run the same tests on day 14 and compare all groups in order."}},{"@type":"Question","name":"Why do I need control pages when testing AI visibility?","acceptedAnswer":{"@type":"Answer","text":"Without a control, you will credit the last change you made for whatever happened next. A before-and-after screenshot cannot distinguish your edit from background movement. Untouched controls and a comparison group let you see whether only the edited pages changed."}},{"@type":"Question","name":"If AI crawlers are blocked from my site, should I buy a visibility tool?","acceptedAnswer":{"@type":"Answer","text":"No. Blocked is an access fix, not a citation-software purchase. Hand a developer the URL, agent, robots rule and failed request, change the setting, then confirm the request succeeds. Do not pay a monthly fee to monitor a door that stays shut."}},{"@type":"Question","name":"My pages are crawlable but AI engines never quote them — what should I do?","acceptedAnswer":{"@type":"Answer","text":"Try a page edit before buying another audit. If the initial HTML contains the facts and a live session reads the URL but will not quote the answer, look at where and how the page states it. Add the Group A answer block, leave comparison pages alone, measure again, and roll the edit wider only if the edited pages move while the others do not."}},{"@type":"Question","name":"What should a paid AI visibility tool actually do for me?","acceptedAnswer":{"@type":"Answer","text":"A tool earns its fee if it keeps checking verified fetches after your initial test, tracks live quotes against the pages you left alone, and shows what changed on each page and why. It should not charge you to announce that robots disallows a fetcher or that JavaScript hides a price, because you just checked both yourself."}}]}]}
```