Why AI Crawlers Do Not Run Your JavaScript (and What It Costs You)
Googlebot renders JavaScript. Most AI crawlers do not. If your content only exists after a script runs, assistants see an empty page while Google ranks you fine.
A SaaS company we audited this spring ranked on page one for eleven of its fifteen target terms. Ask ChatGPT, Perplexity or Claude about the same category and it was never mentioned. The founder assumed the assistants were biased towards larger competitors.
Their marketing site was a React application with no server rendering. Googlebot rendered it in a second pass and indexed everything. Every AI crawler that fetched it received a page containing a navigation bar, a footer, and an empty div where the content belonged.
They were not being ignored. There was nothing to read.
Googlebot Is the Exception, Not the Standard
Google spent years building a rendering pipeline. Googlebot fetches your HTML, queues the page, and later runs a headless browser against it to see what JavaScript produces. It is expensive infrastructure and Google is one of very few organisations that has bothered.
Most AI crawlers make an HTTP request and parse what comes back. No browser, no script execution, no waiting for an API call to resolve. This is a reasonable engineering decision on their side: rendering at web scale costs orders of magnitude more than fetching, and the majority of the useful text on the web is in the initial HTML response.
Behaviour differs between operators, changes without announcement, and is rarely documented. What holds consistently is that rendering support ranges from partial to none, while Googlebot's is complete. Building on the assumption that everything renders like Googlebot is what produces the failure above.
The gap is widening in practical terms, because the proportion of buyer research happening through assistants keeps climbing while the rendering capability on that side stays roughly flat.
What an Unrendered Page Looks Like
Open your site and press Ctrl+U. That is the raw HTML response, and it is what a non-rendering crawler receives.
For a client-side rendered application, you typically get a <head> with some meta tags, a <div id="root"></div>, and several script tags. The word count of actual content is close to zero. Everything a reader sees is constructed in the browser afterwards.
A crawler reading that has your title tag, your meta description, and nothing else. It cannot summarise your service, cannot extract your pricing, cannot follow your internal links to find your other pages, and has no basis on which to recommend you for anything.
Partial versions are more common and harder to spot. The hero section is in the HTML because it was server rendered, and the rest of the page loads client-side. Or the content is present but the internal navigation is JavaScript click handlers, so the crawler reads one page and finds no route to the other forty.
How to Check Yours in Five Minutes
Three checks, in order.
View source and search for a sentence from the middle of your main content. If it is not in the source, it does not exist for a non-rendering crawler. This one test answers the question for most sites.
Fetch the page as an AI crawler. Our Crawler View requests a URL with the user-agent of OAI-SearchBot, PerplexityBot or ClaudeBot and shows the HTML that came back, the headings, the links and the structured data that survived. Running the same URL as Googlebot and then as ClaudeBot puts the rendering dependency in front of you as two different documents.
Test the pre-rendering path if you have one. Sites that already retrofitted a pre-render service usually assume it works. Our Pre-rendering Tester fetches your page both as a crawler and as a browser, then reports whether crawlers are receiving the pre-rendered copy, an empty shell, or a version different enough from the user's to look like cloaking. Snapshots go stale silently, and nothing in your analytics will tell you.
Ranking well on Google and never mentioned by AI assistants? Get a Free Visibility Audit →
What This Costs in Practice
Three losses, of increasing severity.
You are absent from answers in your own category. The assistant recommends whoever it could read. In most B2B and service categories, that is a competitor whose site is a WordPress install or a static build, regardless of who has the better product.
Your facts get sourced from elsewhere. When a model cannot read your site, it assembles what it knows from directories, aggregators, review platforms and old press coverage. Those sources are frequently out of date on pricing, service areas and positioning, and you have no way to correct them.
Your best content never counts. The detailed comparison page, the pricing breakdown, the technical documentation, the work that would earn a citation. All of it sits behind a rendering step that never happens.
For the SaaS company above, the entire content programme was invisible on the surface where their buyers had started researching. Eleven page-one rankings and no presence in the channel that was growing.
Fixing It
Four options, ordered by cost and by how well they hold up.
Static generation. Pages are built to HTML at deploy time. Every crawler gets everything immediately, hosting is cheap, and there is no rendering step to fail. Correct for marketing sites, blogs, documentation and anything that changes on a schedule rather than per request. This site is built this way.
Server-side rendering. The server produces complete HTML per request. Same crawler outcome, higher hosting cost, appropriate when content is genuinely dynamic or personalised.
Pre-rendering. Keep the client-side app for users, serve a rendered snapshot to crawlers. The standard retrofit when rewriting is not on the table. It works, and it introduces a maintenance burden most teams underestimate: the snapshot can drift from the live version, and a large enough divergence is cloaking rather than an optimisation. If you run this, test it on a schedule rather than trusting it.
Hybrid rendering, where the critical content is server rendered and interactive components hydrate afterwards. Available in every major framework now and usually the right answer for an application that also has marketing pages.
There is a fifth option that nobody sells and that is often correct: stop building marketing pages in a single-page application. A pricing page has no interactivity requirement. The framework is solving a problem that page does not have, and moving those routes to static generation is frequently a week of work rather than a rewrite.
What Else Has to Be in the HTML
Rendering is the biggest failure, and three smaller ones travel with it.
Structured data injected by a tag manager is absent from the raw response. It validates perfectly in tools that execute JavaScript, and non-rendering crawlers never see it. Put JSON-LD in the page source.
Internal links built as click handlers cannot be followed. A crawler needs <a href="">. If your navigation is <div onClick={...}>, your site is a set of orphan pages from the crawler's point of view, however good the internal linking looks in a browser.
Content behind an interaction that fetches on click is unreachable. Tabs and accordions are fine when the content is in the HTML and hidden with CSS. They are not fine when the content arrives on demand.
Rendering Is Necessary, Not Sufficient
Fixing this makes you readable. It does not make you citable.
Once the content is in the HTML, what determines whether an assistant names you is the same set of properties that earn a featured snippet: direct statements of fact, prices you are willing to publish, service areas stated in plain text, clear answers to the questions buyers actually ask. Marketing copy that hedges everything is extractable and useless in equal measure, which is a different problem with the same symptom.
We covered that substance layer in how to get your business recommended by ChatGPT and in what is AIEO. This article covers the layer beneath both, and it is worth checking first, because no amount of content work matters if the fetch returns an empty div.
While you are checking, confirm the crawlers are allowed in at all. A rendering fix on a site that blocks OAI-SearchBot in robots.txt changes nothing, and that combination is more common than either problem alone. Our guide to checking AI crawler access covers that in ten minutes.
The Check Worth Running This Week
Take your homepage, your main service or product page, and your best piece of content. View source on each and search for a sentence from the middle of the visible copy.
If all three are present, this is not your problem and you can stop. If any of them come back empty, you have found the reason your AI visibility does not match your search visibility, and it is an engineering fix rather than a content one.
That distinction matters, because most companies in this position respond by writing more content. It goes into the same empty div.
Getting search traffic while your competitors get the AI recommendations?
A Free Visibility Audit includes a crawler-level fetch of your key pages as both search and AI agents, so you can see exactly what each one receives.
Related reading: What Googlebot Actually Sees | Is ChatGPT Allowed to Crawl Your Website? | What Is AIEO
Search Beyond Google
Search Beyond Google is a digital marketing growth agency helping ambitious businesses in the GTA and across North America build compounding visibility across SEO, Local SEO, AEO, AIEO, Google Ads, and Social Media. Every article is researched and written by the SBG team — practitioners who build and test these strategies daily across real client campaigns.
Ready to apply this to your business?
A free 45-minute Visibility Audit maps these frameworks to your specific market, competitors, and current digital presence.
Get Your Free Visibility Audit →Related Articles
Is ChatGPT Allowed to Crawl Your Website? How to Check and Control It
Most sites blocking ChatGPT never meant to. Here is how to check which AI crawlers your robots.txt allows, and which ones you actually want to keep out.
How to Rank on Google Gemini and Microsoft Copilot
Gemini and Copilot pull from different indexes and reward different signals. Here's how each one picks sources and what to change to get your business cited.
How to Get Your Business Featured on Perplexity AI
Perplexity AI is growing 15% month-over-month and is the go-to research tool for professional B2B buyers. Here is exactly how to get your business cited in Perplexity responses.