Sitemaps, robots.txt and AI crawlers (LLMs.txt and OKF)

This guide covers how search engines and AI engines find and read your site: the XML sitemap, robots.txt with per-crawler AI controls, llms.txt, and the OKF bundle. All of these work without a Xagio account, and they link to each other automatically when you turn them on.

XML sitemaps

Xagio builds an XML sitemap index at /sitemap-xag.xml, with one sitemap for each post type and taxonomy you include. It rebuilds when your content changes and is listed in your robots.txt automatically.

Turn on sitemaps

In WordPress, go to Xagio > Sitemaps.
On the Overview tab, turn on Enable Sitemaps.
Open the Sitemaps Content tab and pick what to include (see Choose what’s included).

Each setting saves when you change it. The Overview tab lists the URLs of your generated sitemaps.

When Xagio sitemaps are on, WordPress’s built-in sitemap (/wp-sitemap.xml) is turned off, and Xagio adds Sitemap: <your site>/sitemap-xag.xml to your robots.txt.

Submit your sitemap to Google

Submit https://<your site>/sitemap-xag.xml in Google Search Console under Sitemaps. Do the same in Bing Webmaster Tools.

Sitemap settings

Setting	What it does
Enable Sitemaps	Builds and serves the sitemaps.
Use Compression when possible	Sends compressed sitemaps to browsers and crawlers that support it.
Cache Sitemaps in Webroot	Writes the sitemap files (`sitemap-xag*.xml`) into your site’s root folder instead of serving them from the database. This can be faster. Your root folder must be writable.

Make Xagio the primary /sitemap.xml

Most crawlers look for /sitemap.xml first. Under Crawler Compatibility:

If no other plugin owns /sitemap.xml, Xagio already redirects it to your Xagio index. No action needed.
If another plugin controls it, you see “Xagio detected an existing sitemap at /sitemap.xml (controlled by [plugin]).” Turn on Make Xagio the primary /sitemap.xml to 301-redirect /sitemap.xml to /sitemap-xag.xml anyway.

With Enable Sitemaps off, Xagio does not touch /sitemap.xml at all.

Choose what’s included

On the Sitemaps Content tab, each post type and taxonomy has:

An on/off switch to include it.
Priority: 0.0 to 1.0, a hint about how important these URLs are relative to the rest of your site.
Change Frequency: Always, Hourly, Daily, Weekly, Monthly, Yearly or Never, a hint about how often these pages change.
Exclusions (exclude specific content by ID): a comma-separated list of post or term IDs to leave out, for example 102,333,40.

Only published content is listed.

When the OKF bundle is on, the index also includes an OKF sitemap. See OKF bundle.

Robots.txt and AI crawlers

The Robots.txt tab edits the robots.txt file WordPress serves for your site. It also lets you allow or block AI crawlers one by one, say how AI may use your content, and compare your settings with the live file to catch blocks added by a CDN.

Before you start:

Xagio edits WordPress’s virtual robots.txt. If a real robots.txt file exists in your site’s root folder, your web server serves that file and Xagio’s rules are ignored. The tab warns you: “A static robots.txt file exists on disk. WordPress serves that file directly, so the robots.txt rules managed by Xagio will not apply until the static file is removed.” Delete or rename that file first.
If Settings > Reading > Discourage search engines from indexing this site is ticked in WordPress, WordPress serves Disallow: / and Xagio’s rules are not applied.

Edit robots.txt

In WordPress, go to Xagio > SEO Settings.
Open the Robots.txt tab.
Edit the rules in Robots.txt Editor.
Select Save. You see “Robots.txt saved successfully.”
Select Open /robots.txt to see the live file.

If your rules contain Disallow: /, Xagio warns that this “blocks search engines from crawling your entire site” and asks you to confirm.

Reset to Default restores the WordPress default (User-agent: *, Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php), sets every AI crawler back to allowed, and turns Content-Signal off.

You don’t need to type these into the editor. Xagio adds them for you:

A Sitemap: line for /sitemap-xag.xml, when Xagio sitemaps are on.
Comment lines pointing to /llms.txt and /okf/, when those are on in AI Settings.
A Disallow: / block for each AI crawler you switch off.
The Content-Signal line, when it is on.

Allow or block AI crawlers

Under AI Crawlers, every crawler is allowed by default, so AI engines can read and cite your content. Switch a crawler off to block it. Xagio writes a User-agent: block with Disallow: / for that bot.

The crawlers are: OpenAI (GPTBot), OpenAI (ChatGPT-User), OpenAI (OAI-SearchBot), Anthropic (ClaudeBot), Anthropic (Claude-User), Google (Google-Extended), Perplexity (PerplexityBot), Perplexity (Perplexity-User), Meta (External Agent), Apple (Applebot-Extended), Common Crawl (CCBot) and ByteDance (Bytespider).

The editor previews the change as you switch. Select Save to apply it.

Add a Content-Signal line

Content-Signal lets you say how AI may use the content it is allowed to read, separately from whether it can crawl.

Under Content Signal, turn on Add Content-Signal to robots.txt.
Set each signal:
- search: appear in AI-powered search results and be linked.
- ai-input: be used as input or grounding for AI answers, with citation.
- ai-train: be used to train AI models.
Select Save.

The defaults are search on, ai-input on, ai-train off. Xagio writes one Content-Signal line into the User-agent: * group.

Check the live robots.txt

Some CDNs (for example Cloudflare’s “Block AI bots” setting) add their own rules above WordPress.

Under CDN / Upstream Diagnostic, select Check live robots.txt.
Read the result:
- “Live /robots.txt matches your Xagio settings.” Nothing to do.
- A list of mismatches. “[bot] is blocked upstream (likely CDN), but Xagio says allow.” means your CDN is blocking that crawler. “Xagio disallows [bot] but the live robots.txt does not — your block is not being served.” means your block is not reaching visitors, usually because of a static file or cache.

Xagio cannot override rules your CDN adds. Turn off the CDN feature, or change it in your CDN dashboard.

LLMs.txt

llms.txt is a Markdown file served at /llms.txt. It gives AI engines such as ChatGPT, Claude, Perplexity and Gemini a short map of your site: what it is about and which pages matter. Xagio builds it for you from your site title, tagline and the SEO titles and descriptions of your pages.

Turn on llms.txt

In WordPress, go to Xagio > AI Settings.
Open the LLMs.txt tab.
Turn on Enable /llms.txt.
Adjust the options below.
Select Save. You see “llms.txt settings saved.”
Select Open /llms.txt to see the live file.

The Preview box shows the file as it will be served.

Option	What it does	Default
Enable /llms.txt	Serves the file. When off, `/llms.txt` returns a 404.	Off
Link to the Xagio sitemap	Adds a Key resources section linking to `/sitemap-xag.xml`. Only added when Xagio sitemaps are on.	On
Intro / description	The short summary under your site title. Leave it blank to use your WordPress tagline.	Blank
Max items per post type	How many items to list for each post type. 1 to 1,000.	100
Included post types	Which post types to list.	Pages and Posts

What the file contains

Your site title as the heading, then your intro (or tagline).
Core Services: your pages, except company and policy pages.
One section per other included post type, for example Posts.
Key resources: a link to your sitemap, if that option is on.
Knowledge Bundle: a link to /okf/, if the OKF bundle is on.
Optional: company and policy pages (titles containing words such as about, contact, team, privacy, terms or disclaimer), as plain links.

Each item uses the page’s Xagio SEO title and description. If those are empty, Xagio uses the post title, then the excerpt or the first 30 words of the content. Only published items are listed.

The file is sent with a noindex header, so it does not show in search results itself. When llms.txt is on, Xagio also adds a comment line pointing to it in your robots.txt.

Reset to Default turns llms.txt off and restores the default options. It asks you to confirm first.

OKF bundle

The OKF (Open Knowledge Format) bundle publishes each of your pages as clean Markdown under /okf/, with links between pages, so AI engines can read your content without parsing HTML. The index lives at /okf/ and each page at /okf/<slug>.md.

Turn on the OKF bundle

In WordPress, go to Xagio > AI Settings.
Open the OKF tab.
Turn on Enable /okf/.
Under Included post types, pick the post types to publish. Pages and Posts are on by default.
Select Save. You see “OKF settings saved.” and Xagio builds the bundle.
Select Open /okf/ to see the index.

What gets published

Up to 1,000 of your most recent published items from the post types you picked.
Each page uses its Xagio SEO title and description, or the excerpt if those are empty.
The bundle rebuilds by itself when you publish, update, trash, restore or delete content.
The /okf/ index can be indexed by search engines. The individual .md pages are sent with a noindex header, so they don’t compete with your real pages in search.

When OKF is on, Xagio also links it from /llms.txt, from a comment in your robots.txt, and from an OKF sitemap in your Xagio sitemap index.

Check the bundle status

Bundle status shows Documents in bundle and Last built, plus one of these:

All good. “The bundle passed validation with no content issues.”
Content quality notices (the bundle is still published). For example, “Near-duplicate content: [page] and [page] (N% similar).” or a page with no source URL. Fix the pages if you want, but nothing is blocked.
Validation failed. The new bundle broke a hard check, such as an empty page, a missing header block or a broken internal link. If an earlier good bundle exists, “The new bundle was not published — the previous version is still served.” Otherwise the new bundle is published with the errors listed.

Select Rebuild now to rebuild the bundle after you fix content. You see “OKF bundle rebuilt.”

Reset to Default turns /okf/ off and restores the default post types. It asks you to confirm first.

Troubleshooting

Sitemaps

/sitemap.xml still shows another plugin’s sitemap. Turn on Make Xagio the primary /sitemap.xml, or turn off the other plugin’s sitemap feature.
The sitemap is out of date. Clear your page cache and CDN cache. If Cache Sitemaps in Webroot is on, check that the root folder is writable.
Sitemap URLs return 404. Go to Settings > Permalinks in WordPress and select Save Changes.

Robots.txt

“Could not fetch /robots.txt: …” or “Live /robots.txt returned HTTP [code].” The check could not load your live file. Open /robots.txt in a browser to see what is served.
Changes don’t show on /robots.txt. Clear your page cache and CDN cache, and check for a static robots.txt file.

LLMs.txt

The file Xagio serves is not the one you see. If a real llms.txt file exists in your site’s root folder, your web server serves it instead and the tab says so. Select Save to have Xagio remove that file, or delete it yourself.
A page has a poor description in the file. Fill in that page’s SEO description in the On-Page SEO box. See The On-Page SEO box.

OKF

/okf/ returns a 404. Check that Enable /okf/ is on and saved. Then go to Settings > Permalinks in WordPress and select Save Changes to refresh the rewrite rules.
A page is missing. Check that its post type is included, that it is published, and that it is within the 1,000 most recent items.