Web & tools5 min read
Making a Website Readable for Google and AI Search Tools
A step-by-step check from robots.txt to the Rich Results Test shows whether Google and AI assistants can actually read and correctly interpret a website.
- #robots.txt
- #Google Search Console
- #KI-Suche
- #Core Web Vitals
- #strukturierte Daten
- #Website-Technik
AI-assisted · editorially reviewed Details
Every website has a robots.txt file at the exact same address, yourdomain.com/robots.txt — and that address is where this check starts, because it decides whether Google and AI assistants are even allowed to read a page. Most owners who built their site through an agency have never opened that file themselves. The order below is the order that actually makes sense in practice: first see what machines are permitted to look at, then whether they can do it fast enough, and only after that whether they understand what they find.
1. Open robots.txt directly in a browser
Tool: any browser, no login required. The address is always [domain]/robots.txt. It lists in plain text which areas are blocked for which bots. The line to watch for is Disallow: / with no bot specified above it — if it sits at the top without a matching User-agent line, the entire site is blocked for everyone. This happens more often than it should: staging environments are deliberately locked down so test versions never reach Google. If that line survives the move to the live site, the finished website becomes invisible with no error message anywhere. Skip this step and the problem usually surfaces months later, once inquiries quietly dry up.
2. Identify the AI crawlers separately
Inside the same file, individual AI providers increasingly get their own entries, separate from the classic search engine bots. OpenAI's own documentation distinguishes GPTBot, which gathers content for model training, from OAI-SearchBot, which handles citations inside ChatGPT's search feature; the two can be blocked independently. Google separates the regular Googlebot from Google-Extended, which according to Google's crawler overview only controls whether content feeds Gemini and similar AI features, not whether the page shows up in regular Google Search. Perplexity lists its own crawler under PerplexityBot. Anyone who skips this distinction tends to block everything with a single blanket line, including the bot that was supposed to make the site findable in the first place.
3. Set up Google Search Console and inspect the URL
Tool: Google Search Console, free, a Google account is enough. Once ownership is verified, the URL inspection tool shows for any single page whether Google knows it, when it was last read, and whether there were access problems. Without this step, any claim about visibility stays exactly that: a claim. A furniture workshop whose site was built by an agency three years ago and never touched since will often find a handful of pages marked "Crawled — currently not indexed" here. That is not a total failure, just a sign that something in the content or the internal linking isn't convincing.
4. Check and submit the XML sitemap
The sitemap usually sits at [domain]/sitemap.xml and most content management systems generate it automatically. It can be registered under "Sitemaps" inside Search Console. If it is missing or lists outdated addresses, bots have to discover every new page through internal links instead — that works, but it takes longer and is especially unreliable on small sites with few internal references. A business that skipped clean redirects after a relaunch runs into exactly the problem described in the post on a relaunch without redirects: a working sitemap does not fix missing redirects.
5. Measure load time with PageSpeed Insights
Tool: PageSpeed Insights, free, no login. It measures the Core Web Vitals (load time, visual stability, and responsiveness) separately for mobile and desktop. Google's page experience documentation states that these values feed into search as a signal. For crawlers, there is also the so-called crawl budget: according to Google's crawl budget documentation, bots favor servers that respond quickly, since they can read more pages in the same amount of time. On a twenty-page site this barely matters. On a catalog with several hundred product pages, a slow server response can mean new pages aren't picked up for weeks.
6. Check structured data with the Rich Results Test
Tool: Google's Rich Results Test, free. It shows whether a page carries machine-readable schema.org markup, for example opening hours, address, or reviews for a local business, as documented in Google's guide to LocalBusiness markup. Without that markup, every system has to interpret the plain text itself, which goes wrong on vaguely worded contact pages more often than owners expect. With clean markup, a tradesperson's address can be read reliably straight from the source code. No guessing required.
7. Run a manual test inside an AI chat
The final test happens outside any toolbox: ask Perplexity or ChatGPT, with web search switched on, about the business and watch whether and how the site gets cited. That proves nothing universal, only a snapshot on one day with one model. Still, it shows whether the previous six steps changed anything at all.
The step that usually goes wrong
What usually fails isn't the technology, it's the sequence. Someone blocks every AI bot in robots.txt as a precaution, reasoning that their text "isn't meant for training" — and overlooks that OAI-SearchBot and PerplexityBot don't train anything, they only cite a page when someone actively asks about it. Blocking GPTBot also has zero effect on ranking position in ordinary Google Search; anyone hoping for more visibility from that move is confusing two unrelated systems.
None of these tools check accessibility. That stays a separate job.
Businesses that would rather skip clicking through the first six steps themselves can use Log-Studio's free website check as a shortcut: without any sign-up, it measures load time, Google findability, security, technical setup, and whether AI assistants can read the page at all. Log-Studio, the web design brand of Log-System Development, also builds websites for small businesses afterward, with one direct contact person instead of an agency ticket queue — useful for owners who want to know what's actually broken before spending money on a relaunch.
Read next
Web & toolsAdvertisement from Log-Studio (https://log-studio.de)
When It's Worth Opening Your Browser's Developer Tools for Your Own Website — and When It Isn't
Pressing F12 opens free developer tools in any browser — useful for quick troubleshooting, but no substitute for a systematic site review or checking whether AI assistants can read the page.
Web & toolsAdvertisement from Log-Studio (log-studio.de)
Running Your Own Website Check From Start to Finish
Six concrete steps for small-business owners to check their own site's Core Web Vitals, mobile usability, link structure and contact channels — including a free website check as a shortcut.
Web & toolsAdvertisement from Log-Studio (log-studio.de)
Running a Website Check Yourself, Tool by Tool
Nine free tools, one fixed order: from the SSL grade through Search Console and PageSpeed Insights to matching the Google Business Profile — how a website check actually runs without an agency.