What Googlebot Actually Sees on Your Page (and Why It Is Not What You See)

There is a specific kind of SEO problem that survives every audit: the page looks perfect. Content is there, headings are sensible, internal links are in place, the client can see all of it. Google ranks it for nothing.

Almost always, the crawler received a different page than the one on the screen. Not a slightly different page. Sometimes an empty one.

This gap between what a browser renders and what a crawler is handed causes more unexplained ranking failures than any other technical issue we deal with, and it is invisible until you go and look.

---

Why the Two Versions Differ

Your browser does a great deal of work on your behalf. It runs JavaScript, waits for network requests to settle, executes framework code that injects content into the DOM, and only then paints something. What you see is the result of several seconds of processing.

A crawler makes an HTTP request and receives a document. For Googlebot, rendering happens in a second pass, on Google's schedule and Google's budget, which may be minutes or days after the initial fetch. Some crawlers never render at all. Most AI answer crawlers fall into that group.

So there are really three versions of your page:

1. The raw HTML response — what every crawler gets immediately
2. The rendered DOM — what Googlebot builds after executing JavaScript, eventually
3. What you see — the rendered DOM plus your cookies, your location, your session, your ad blocker

Ranking decisions are made mostly from versions one and two. Nobody audits either.

The Five Gaps That Cost Rankings

These are the failures we find most often, roughly in order of how expensive they are.

Body content that only exists after JavaScript runs. Common in React, Vue and Angular sites without server rendering, and in any page loading its main content through an API call. The raw HTML contains a nav, a footer and an empty div. Google may render it later and index it fine. Or it may not, and every AI crawler certainly will not.

Internal links that are not links. A <div onclick="navigate()"> styled to look like a link is invisible to a crawler. So is a <button>. We audited a 400-page catalogue last year where the entire category navigation was JavaScript click handlers, and 340 of those pages had never been discovered by Google in four years of the site being live.

Content behind an interaction. Tabs, accordions and "read more" toggles are fine when the content is present in the HTML and hidden with CSS. They are not fine when the content is fetched on click. The distinction is invisible in a browser and total from a crawler's side.

A different response for a different user-agent. Sometimes deliberate, sometimes a firewall rule, sometimes a bot-protection service being helpful. A page returning 200 to Chrome and 403 to Googlebot will be dropped from the index without warning, and nothing in Search Console spells out the cause.

Geo or consent walls that intercept the crawler. Googlebot generally crawls from US IPs and accepts no cookies. If your consent banner blocks content until accepted, or your site redirects by region, the crawler may only ever receive the interstitial.

How to See the Crawler's Version

Three approaches, in ascending order of effort.

View source, not inspect element. This is the single most useful habit in technical SEO and almost nobody has it. Ctrl+U shows the raw HTML response. The Inspect panel shows the rendered DOM, which is a different document. Search the source for a sentence from the middle of your main content. If it is not there, the crawler did not get it on the first pass.

Fetch the page as a crawler. Our Crawler View requests any URL with the user-agent of Googlebot, Bingbot, OAI-SearchBot, PerplexityBot or ClaudeBot and shows exactly what came back: the HTML, the heading structure the crawler can parse, the links it can follow, the structured data it found, and the indexability signals in the response. Running the same URL as Googlebot and then as ClaudeBot is the fastest way to see whether your page depends on rendering.

URL Inspection in Search Console. Google's own view, and the authoritative one for Googlebot specifically. Use "Test live URL" then "View crawled page" to see the rendered HTML. It is slower and limited to properties you own, which is why we start with the first two and use this to confirm.

Ranking well on some pages and nowhere on others, with no obvious difference between them? Get a Free Visibility Audit →

Reading the Result

Once you have the crawler's HTML in front of you, work through it in this order.

Check that your primary content is present. Not the nav, not the footer, the actual body text. If the page is 40KB of script tags and 200 words of boilerplate, that is the whole finding and nothing further matters until it is fixed.

Check the heading structure. Crawlers build a topical understanding partly from the H1 and H2s, and headings injected by JavaScript are frequently missing from the raw response even when body text is present. A page whose H1 is absent from the source is being read as topically vague.

Count the internal links. Compare that number to what you can click in a browser. A large gap tells you where crawl paths are dying.

Look for the structured data. JSON-LD injected by a tag manager will often not appear in the initial response, which is why schema that validates perfectly in a testing tool can still produce no rich results.

Check the response headers and meta tags for noindex, canonical tags pointing elsewhere, and unexpected redirects. A canonical pointing at the homepage is a surprisingly common CMS default and it silently deindexes everything.

When the Page Depends on Rendering

If your raw HTML is empty, you have three options and they differ enormously in cost.

Server-side rendering sends complete HTML on the first response. Every crawler gets everything, immediately. This is the correct answer and usually means a framework migration.

Static generation pre-builds pages to HTML at deploy time. Same crawler outcome as SSR, cheaper to host, appropriate when content changes on a schedule rather than per request. This site is built that way.

Pre-rendering keeps the client-side app for users and serves a rendered snapshot to crawlers. It works, and it is the most common retrofit, but it introduces its own failure mode: the snapshot silently going stale or diverging from the user version, which is where the line into cloaking sits. If you run pre-rendering, verify it rather than trusting it. The Pre-rendering Tester fetches your page both ways and reports whether crawlers are getting the pre-rendered copy, an empty shell, or something different enough from the user version to be a problem.

There is a fourth option nobody sells you, which is often correct for small sites: stop building marketing pages in a single-page app. A service page has no interactivity requirement. The framework is solving a problem the page does not have.

The Mobile Version Is the Version That Counts

Google indexes the smartphone crawl of your site. Not the desktop one. If your mobile template drops content, collapses navigation into JavaScript, or lazy-loads sections the desktop version renders immediately, then the reduced version is your indexed site.

This still catches people out years after mobile-first indexing became universal, because the desktop page is the one everyone reviews. The Mobile-First Index Tool fetches as Googlebot Smartphone and as desktop Googlebot, then diffs the two: content, headings, internal links, images, structured data and robots directives. The output you want is a boring one. Any meaningful difference is a finding.

Do This Once a Quarter

Pick five URLs: your homepage, your highest-value service page, a blog post, a location or category page, and whatever page you have most recently rebuilt. Fetch each as Googlebot and as one AI crawler. Confirm the body content, H1 and internal links are present in the raw response.

It takes fifteen minutes and it catches deployment regressions before they turn into a quarter of lost traffic. Frameworks change rendering behaviour between major versions, plugins update, and a bot-protection rule added by an IT team on a Friday can deindex a site by Monday. None of this shows up in a rank tracker until the damage is a month old.

For the underlying mechanics, Google Search Central's JavaScript SEO documentation is the primary source and it is clearer than most third-party writing on the subject. Our technical SEO guide covers where this fits alongside the rest of the crawl and index stack.

What This Is Really About

Every technical SEO problem is eventually a question of whether the machine got what the human got. Speed, structured data, internal linking and indexability are all versions of the same question.

The reason this one is worth its own check is that it fails silently and completely. A slow page still ranks, a bit lower. A page whose content never reached the crawler does not rank at all, and produces no error, no warning, and no clue in any report you are likely to read.

Suspect Google is seeing a different site than your visitors are?

A Free Visibility Audit includes a crawler-level fetch of your key pages, a rendered-versus-raw comparison, and a prioritised list of what is not reaching the index.

Get a Free Growth Audit →

---

Related reading: The Complete Technical SEO Guide | Website Not Showing Up on Google | On-Page SEO Checklist