Applebot-Extended is a robots.txt token, not a crawler. Disallowing it tells Apple not to use your content to train its foundation models (Apple Intelligence, Services and Developer Tools), while the normal Applebot keeps crawling and your pages stay in Spotlight, Siri and Safari results. Apple says Applebot-Extended "does not crawl webpages" and that "webpages that disallow Applebot-Extended can still be included in search results". For AI-generated answers in Siri and Search, Apple uses a different control: the nosnippet meta tag.
Applebot vs Applebot-Extended
Everything here is from Apple's About Applebot page.
| Applebot | Applebot-Extended | |
|---|---|---|
| What it is | Apple's web crawler | A robots.txt token Apple checks before using Applebot's data for model training |
| Crawls? | Yes | No: it "is only used to determine how to use the data crawled by the Applebot user agent" |
| Used for | Search in Spotlight, Siri and Safari; may also feed AI answers and training | Opting out of training Apple's foundation models |
| robots.txt | "Respects standard robots.txt directives" aimed at it; if none, "the Apple robot will follow Googlebot instructions" | Disallow to opt out of training |
| Ranking | Normal search factors | "Site rules for Applebot-Extended are not considered in ranking for Search" |
Three different controls
Apple separates three uses of your content, each with its own switch:
- Search (Spotlight, Siri, Safari): controlled by
Applebotrules in robots.txt and thenoindexmeta tag. - AI answers with sources ("broad world knowledge questions in Siri and Search"): opt out with
nosnippeton specific content. Apple: it "will not use data tagged nosnippet as additional context" when AI models generate output in its products. - Training of Apple's foundation models: opt out by disallowing
Applebot-Extended.
Apple also notes that pages marked isAccessibleForFree: false in structured data can still appear in search, but Applebot "will not use that content as additional context" for AI output.
robots.txt examples
Stay in Apple search, opt out of training
User-agent: Applebot-Extended
Disallow: /
That's all. Applebot keeps crawling under your other rules.
Opt out of training for part of the site
User-agent: Applebot-Extended
Disallow: /private/
Keep a section out of AI answers too
Add to the pages concerned:
<meta name="applebot" content="nosnippet">
or <meta name="robots" content="nosnippet"> to apply it to every crawler that supports it. Apple also accepts X-Robots-Tag: applebot: nosnippet as an HTTP header, useful for PDFs. Note that nosnippet also means Apple "won't generate a description or web answer for the page", and the robots version removes Google's snippet too.
The Googlebot fallback
One Applebot behaviour surprises people: "If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions." If your robots.txt has a strict User-agent: Googlebot group and no Applebot group, Applebot uses the Googlebot rules, not *. To give Applebot its own rules, name it:
User-agent: Applebot
Allow: /
The robots.txt precedence guide covers how groups are chosen in general.
Rendering and blocked resources
Apple says Applebot "may render the content of your website within a browser" and that if JavaScript, CSS or XHR resources are blocked in robots.txt, "it may not be able to render the content properly". Don't block your theme or script folders. Applebot "does not follow crawl-delay".
Verifying Applebot
Apple says Applebot traffic is "generally identified by using reverse DNS in the *.applebot.apple.com domain", or by matching the IP against its published Applebot IP CIDRs. Applebot-Extended never appears in logs, since it doesn't make requests.
Tools that add Applebot-Extended
The Squarespace "Block known artificial intelligence crawlers" setting (Squarespace) and Cloudflare's managed robots.txt (Cloudflare) both add an Applebot-Extended group. Neither blocks Applebot itself in the lists they publish.
Applebot-Extended and AdSense
Unrelated: Apple's crawler and opt-outs don't affect Google's AdSense crawler. A free Approvalens scan checks the AdSense side, and the should I block AI training bots guide compares the training opt-outs across companies.
Check your setup
The free AI crawler checker shows whether your robots.txt disallows Applebot-Extended and the other training tokens, separately from the search bots, and flags nosnippet or noai directives on the page. Because Applebot-Extended makes no requests, its check is robots.txt only; page requests for the other bots are simulated with their user agents from our server, so a firewall that verifies IPs may treat the real bots differently. The robots.txt tester shows how Google reads the same file.
FAQ
Will disallowing Applebot-Extended remove me from Siri or Spotlight?
No. Apple says pages that disallow Applebot-Extended can still be included in search results, and your content "will remain discoverable through Spotlight, Siri, and Safari".
Does Apple honour the noai tag?
Apple's Applebot page doesn't mention it. Apple documents Applebot-Extended for training and nosnippet for AI answers.
Is blocking Applebot-Extended retroactive?
Apple's page doesn't say. It describes the token as controlling how Applebot's data is used; it says nothing about data already used.
Who do I contact at Apple about Applebot?
Apple lists applebot@apple.com on the same page.
Spotted something out of date or wrong? Tell us and we'll correct it.
Read this guide in Turkish →Check it on your own site. Free, no sign-up.
Free tools for this
Free scan
Check your own site
Free scan: readiness score and every issue, usually in a few minutes.