How we fixed AI crawler access on a Cloudflare site

A website can look completely ordinary in a browser and still be closed to the systems that are meant to read it. That was GattyWorks on 24 August. An agent-readiness scan gave us 5 out of 100, even though the homepage was up and our robots.txt allowed every crawler.

The first instinct with a score like that is to rewrite pages, add a new manifest, or start guessing at metadata. We did not. The scan's error had a more useful clue: a Cloudflare challenge page was arriving before the site did.

The page was public. The front door was still locked.

We checked the actual zone settings with our Cloudflare audit tooling. Direct AI-crawler controls were already off. There were no custom firewall rules, IP blocks, or lockdowns aimed at those crawlers. The setting doing the work was Bot Fight Mode, enabled across the whole domain.

Bot Fight Mode is useful when a site needs broad automated-traffic protection. It can also challenge a legitimate AI crawler before that crawler reaches the HTML. On this domain, that policy no longer matched the job. GattyWorks is meant to be readable by AI systems, so there should be no crawler restriction at that layer.

curl -A "ChatGPT-User" -I https://gattyworks.com/
curl -A "ClaudeBot" -I https://gattyworks.com/
curl -A "GPTBot" -I https://gattyworks.com/

Before the change, the scanner saw the challenge instead of the site. We turned Bot Fight Mode off, then repeated the checks with GPTBot, ClaudeBot, ChatGPT-User, Google-Extended, DeepSeekBot, and ora-agent user agents. Each received a 200 response and the rendered homepage, not an interstitial.

The second scan could finally see the site

The rescan reached 78 out of 100. That is a 73-point change without changing the homepage copy or adding a special crawler allowlist. The difference was access: the evaluator could fetch the server-rendered content, follow navigation, and identify the site correctly.

That second result mattered more than the number. It showed us what to fix next. The crawler explored `/services`, `/audits`, and `/how-it-works`, then found that those routes did not exist. They were reasonable guesses from our navigation and the way we describe the studio, but a visitor or crawler that guessed them hit a dead end.

We added the pages the scan went looking for

We shipped three lightweight discovery pages: /services for the work we offer, /audits for our audit entry point, and /how-it-works for the delivery process. Each page points to the detailed service routes and the brief form, so it is useful on its own instead of acting as a redirect in disguise.

We also added all three routes to the sitemap and deployed them to production. The live version now serves each route with a 200 response. We did not add `/docs`, because we do not have product documentation to publish there. A clean 404 is better than a fake documentation section built to satisfy a scanner.

Open access still needs boundaries

Disabling Bot Fight Mode does not make every bot trustworthy. User-agent names can be copied, and a site-wide bot challenge is one of the simpler ways to reduce abusive traffic. We kept the rest of the security posture in place, including Cloudflare's managed protections and the existing per-IP rate limit of 300 requests per 10 seconds.

Check the response before you rewrite the site

The practical lesson is small. When an AI crawler, search tool, or audit bot cannot understand a site, first inspect what it is actually receiving. Check a few real user agents, inspect the WAF settings, and look for a challenge or block page. Only then decide whether the content or information architecture needs work.

In our case, it was both, in order: one Cloudflare switch opened the door, and the second scan showed the three pages worth adding. That is the whole change log. No crawler exceptions, no pretend API, and no rewrite for a problem at the edge.