AI Crawlers and GPTBot on Estate Agency Websites: What to Allow

To be cited by ChatGPT, Perplexity, Claude, Copilot or Google's AI Overviews, an estate agency website has to let each company's search crawler in; blocking training crawlers such as GPTBot does not remove it from their search answers, according to those companies. As for llms.txt, none of the companies read for this article says it uses the file, and Google writes that its Search ignores it.

Sources read on 8 November 2026. Written for estate agents in the UK and real estate agents and realtors in the US, it follows our guides to getting your estate agency recommended by ChatGPT and Perplexity and to real estate agency SEO and GEO.

Three kinds of crawlers

Each company runs several crawlers, each named in robots.txt by a "user-agent token", in Google's words. Training crawlers collect pages for AI models. Search crawlers build the index the tool draws its sources from. User fetchers read a page when a person asks for it.

What each company says

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

OpenAI's crawler page sets the principle: "Each setting is independent of the others".

• OAI-SearchBot powers ChatGPT search. Sites that block it "will not be shown in ChatGPT search answers, though can still appear as navigational links". Allow "~24 hours" after a change.
• GPTBot collects for training. Blocking it indicates that the content "should not be used in training generative AI foundation models".
• ChatGPT-User visits a page when a user asks a question: "robots.txt rules may not apply".

Its help centre also asks you to "confirm that the website host or content delivery network allows traffic from OpenAI's published searchbot IP addresses".

Perplexity: PerplexityBot, Perplexity-User

According to its documentation, PerplexityBot is designed to "surface and link websites in search results on Perplexity", and "It is not used to crawl content for AI foundation models". Perplexity-User reads a page for a user and "generally ignores robots.txt rules". A change can take "up to 24 hours".

Anthropic: ClaudeBot, Claude-SearchBot, Claude-User

Anthropic's page describes three. ClaudeBot is for training: restricting it signals that "the site's future materials should be excluded from our AI model training datasets". Claude-SearchBot is for search: disabling it "may reduce your site's visibility and accuracy in user search results". Claude-User handles users' questions: disabling it "prevents our system from retrieving your content in response to a user query".

Google: Googlebot and Google-Extended

Rules addressed to Googlebot "affect Google Search (including Discover and all Google Search features)", according to Google's list of crawlers. AI Overviews and AI Mode are part of it: "AI is built into Search".

Google-Extended is a token, not a crawler: it "doesn't have a separate HTTP request user agent string", so you will not see it in your logs. It decides whether your content trains Gemini models and grounds answers in the Gemini apps. And: "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

Its user-triggered fetchers, such as Google-Agent, "generally ignore robots.txt rules".

Microsoft: Bingbot, and Copilot

Bing's list of crawlers has no crawler reserved for AI: Bingbot "handles most of our crawling needs each day". For Copilot, Microsoft currently reads two meta tags: noarchive ("Do not link in Chat and Copilot") and nocache ("Display only URL/Snippet/Title in Chat or Copilot").

On 30 October 2026, Microsoft announced two preferences, AI Training and AI Grounding, honoured from 1 February 2027; "Both default to Allow". After 1 March 2027, "NoArchive and NoCache will no longer be read as a preference for training or grounding". "Neither choice guarantees inclusion, traffic, or payment."

Apple and Common Crawl

Applebot powers search in "Spotlight, Siri, and Safari". Applebot-Extended "does not crawl webpages": it lets publishers opt out of training Apple's models, and its rules "are not considered in ranking for Search", according to Apple's page of 4 September 2026. CCBot is the crawler of Common Crawl, a "non-profit foundation" that maintains an "open repository of web crawl data"; blocking it stops its crawls.

The crawler table

robots.txt: what it does not guarantee

robots.txt is a text file at the root of the site, standardised by RFC 9309 in September 2022. Three limits:

1. It is not a lock: "These rules are not a form of access authorization."
2. It shows what it hides: "Listing paths in the robots.txt file exposes them publicly".
3. It does not stop indexing: it "is not a mechanism for keeping a web page out of Google" (Google).

To remove a page from the results, use the noindex tag on a page that is not blocked: otherwise "the crawler will never see the noindex rule" (Google). Another trap: a crawler with its own group obeys only that group, and User-agent: * applies only "If no matching group exists" (RFC 9309).

AI Overviews and AI Mode: Google's three levers

• Googlebot in robots.txt: access to all of Search.
• Snippet tags: nosnippet "will also prevent the content from being used as a direct input for AI Overviews and AI Mode" (robots meta tag), but it also removes the snippet from classic results.
• The Search generative AI control in Search Console. Google began testing it with "a subset of website owners in the UK" on 3 June 2026, and it has been rolled out "to all websites worldwide" since 31 August 2026. It covers AI Overviews, AI Mode and generative AI features in Discover. For an excluded site: "You won't receive any traffic or impressions from these features". And "This control doesn't affect AI training" (Search Console Help). Inclusion is the default.

llms.txt: a proposal, not a setting

llms.txt is a proposal by Jeremy Howard, published on 3 September 2024 and revised on 10 August 2026: a Markdown file at the root of the site that summarises it to help agents "use a website".

• Google lists it among the things you can ignore, "as Google Search itself doesn't use them", in its guide to optimising for generative AI features; keeping one "will neither harm nor help your site's visibility or rankings in Google Search". Its page AI features and your website says it again: you don't need "AI text files".
• OpenAI, Perplexity, Anthropic, Microsoft and Apple: their crawler pages do not say they read a site's llms.txt.

Our view: not a priority. If your website platform generates one at no extra effort, keep it; don't pay for it as an SEO extra.

The block the agency never chose

On 1 July 2025, Cloudflare announced that it would block AI crawlers without permission by default, and that "every new domain will now be asked if they want to allow AI crawlers" (press release).

Cloudflare's changelog of 1 July 2026 says that from 15 September 2026, new domains block training and agent bots on "pages that display ads". Its documentation, last updated on 2 November 2026, says that "For new domains, Search, Agents, and Training use Allow (do not block) by default". Assume nothing: check the setting.

Its Content Signals Policy of 24 September 2025 adds a Content-Signal line to robots.txt (search, ai-input, ai-train): preferences, "not technical countermeasures against scraping", which none of the companies read says it reads.

The step: ask your web developer or host, in writing, whether the site sits behind a firewall and which AI crawlers it blocks. An agency can be missing from ChatGPT without its robots.txt being the cause.

What the law says

United Kingdom. Section 29A of the Copyright, Designs and Patents Act 1988, inserted on 1 June 2014, allows a person with "lawful access to the work" to copy it to "carry out a computational analysis of anything recorded in the work for the sole purpose of research for a non-commercial purpose". The government's consultation "Copyright and Artificial Intelligence" ran from 17 December 2024 to 25 February 2025. Its page, updated on 19 March 2026, says "a report and impact assessment have been published".

United States. US copyright law has no written exception for AI training: the question is fair use. The U.S. Copyright Office released a pre-publication version of Part 3 of its report, on generative AI training, on 9 May 2025 (a final version "will be published in the future"). It writes: "The key question, as most commenters agreed, is whether those acts of prima facie infringement can be excused as fair use." It adds that "it is not possible to prejudge the result in any particular case", and speaks of an exception only as a possibility: "Congress may consider an exception or limitation for AI training in the future". It also warns that honouring "platform-level flags, like robots.txt" should not be enough for an opt-out, because copyright owners often "have no control over the platforms where their works appear". That is the case for listings on Rightmove, Zoopla, Zillow or Realtor.com: the portal writes its own robots.txt.

For your listings: section 29A speaks of a "work" without saying whether a listing or its photos are one, and names neither robots.txt nor llms.txt. This is not legal advice.

Our recommendation

1. Allow the search crawlers: Googlebot, Bingbot, Applebot, OAI-SearchBot, PerplexityBot, Claude-SearchBot.
2. Let user fetchers through (ChatGPT-User, Perplexity-User, Claude-User): when a buyer or a tenant asks an assistant to read your listing, one of them comes.
3. Decide on training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot): allowing them means letting your text feed future models. OpenAI, Google and Apple write that refusing them changes nothing in their search; refusing Google-Extended does, however, take the site out of grounding in the Gemini apps.
4. Keep private the client area, account pages and internal search: the password protects them, and robots.txt avoids useless crawling.
5. Stay included in Google's AI features (Search Console), unless you have a precise reason.

A sample robots.txt for a fictional agency

Take a fictional estate agency in Leeds, at www.example.com. It accepts search and refuses training (to allow training, delete group 2).

# robots.txt for www.example.com (fictional estate agency in Leeds)

# Group 1: every crawler without a group of its own
# (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot...).
User-agent: *
Disallow: /client-area/
Disallow: /my-account/
Disallow: /search
# These paths stay readable: the client area requires a password.

# Group 2: training refused (the agency's choice).
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: __usage_ai-training
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Search crawlers, unnamed, follow group 1. The __usage_ai-training line is a shared signal in preparation, described in a draft of 30 October 2026 that "has no formal standing" at the IETF. Anthropic says it began recognising a "no AI training" signal based on this IETF work on 30 October 2026, Microsoft will read it from 1 February 2027, and Google plans to support it "in early 2027".

How to check

1. Open your robots.txt and look for Disallow: / under User-agent: * or under a search crawler. With no file, "Google can crawl all URLs on your site".
2. The robots.txt report in Search Console shows the files found for "the top 20 hosts" on the site and their errors, and lets you request a recrawl.
3. Server logs: ask your hosting provider for last month's visits, filtered on the names in the table. Zero visits from OAI-SearchBot or PerplexityBot may point to a block.
4. Crawler identity: "The HTTP user agent string can be spoofed", Google warns. The companies publish their IP addresses, for example openai.com/searchbot.json or perplexity.ai/perplexitybot.json; Bing offers its "Verify Bingbot" tool.
5. The firewall: an allowed crawler that never arrives is a question for your provider.

What Cleemo offers

With Cleemo's real estate software and AI agents, each agency has its own AI assistant. It finds information in your files, reads your documents and prepares a listing or a draft reply; a listing is published only after your approval, and the assistant's actions are logged. Cleemo App brings sales, property management and CRM together in one application, which you can use without AI; Cleemo AI adds the assistant to it, or works around your current software, depending on its access. The AI employees page describes each role.

Frequently asked questions

Does blocking GPTBot remove my agency from ChatGPT?

No, according to OpenAI: GPTBot is for training, and each setting is independent. Blocking OAI-SearchBot is what removes a site from ChatGPT search answers.

Does blocking Google-Extended remove my site from AI Overviews?

No, according to Google: Google-Extended does not affect inclusion in Google Search, and AI Overviews are part of Search. To opt out, use the Search generative AI control in Search Console, or the nosnippet tag.

Do I need an llms.txt file to be cited by AI?

Not for Google, which writes that its Search ignores these files. None of the other companies read for this article says it reads websites' llms.txt files.

Do AI crawlers respect robots.txt?

The training and search crawlers listed here are controlled through robots.txt, according to their companies. Those that read a page for a user, such as ChatGPT-User, may ignore it; Anthropic lets you block Claude-User.

Can my website be blocked without my knowing?

Yes. A firewall such as Cloudflare's can block AI crawlers whatever your robots.txt says, and its defaults have changed since 2025.

How long does a change take to work?

About 24 hours for ChatGPT, up to 24 hours for Perplexity, and a few days for the Search Console control. A tag read by Google waits for the next crawl, which can take "anywhere from several days to several months".

👉 See how Cleemo's AI agents work for your agency