A recent study has revealed the problem of websites relying on JavaScript for structured data (micro-markup). This data, often in JSON-LD format, is difficult for AI crawlers to access if it is not contained in the initial HTML response.
Crawlers such as GPTBot (used in ChatGPT), CloudeBot, and PerplexityBot cannot execute JavaScript and skip any structured data output using it.
This creates problems for sites using tools such as Google Tag Manager (GTM) to insert JSON-LD on the client side, as many AI crawlers cannot read dynamically generated content.
Recommendation of the month: Telega.in — native integration platform in Telegram channels. Manually verified channels and bots. We help you find your target audience on Telegram and launch effective integrations.
- What webmasters need to know about JSON-LD and AI crawlers
- What is the difference between traditional search engines
- Google’s warning about excessive use of JavaScript
- And what should I do?
- Why is this important
What webmasters need to know about JSON-LD and AI crawlers
Elie Berrebi, founder of SEM King, studied what happens when JSON-LD is added using Google Tag Manager without server-side rendering (SSR).
He found out why this type of structured data is often not perceived by AI crawlers.:
- Initial HTML download. When the crawler requests a web page, the server returns the first HTML version. If structured data is added using JavaScript, it will not be in this initial response.
- Executing JavaScript on the client side. JavaScript runs in the browser and modifies the document object model (DOM) for users. At this stage, GTM can add JSON-LD to the DOM.
- Crawlers without JavaScript rendering. AI crawlers that can’t run JavaScript don’t see any changes in the DOM. This means that they will skip any JSON-LD added after the page loads.
In general, structured data added only using JavaScript on the client side is invisible to most AI crawlers.
What is the difference between traditional search engines
Traditional search crawlers such as Googlebot can read JavaScript and process changes made to a web page after it is loaded, including JSON-LD data entered by Google Tag Manager.
Unlike them, many AI crawlers cannot read JavaScript and only see the raw HTML from the server. As a result, they skip dynamically added content, such as JSON-LD.
Google’s warning about excessive JavaScript usage
This issue is related to Google’s broader warning about excessive JavaScript usage.
In a recent podcast, the Google team discussed the growing reliance on JavaScript. Although it allows you to create dynamic functions, JS is not always ideal for important SEO elements such as structured data.
Martin Splitt, Google’s developer relations specialist, explained that websites range from simple pages to complex applications. It is important to balance the use of JavaScript and ensure that key content is available in the source HTML.
John Mueller, another Google Search representative, agreed with him, noting that developers often resort to JavaScript when simpler options such as static HTML would be more effective.
And what should I do?
Developers and SEO specialists should ensure that structured data is available to all crawlers in order to avoid problems with AI search crawlers.
Here are some solutions:
- Server-side rendering (SSR). Rendering pages on the server to include structured data in the initial HTML response.
- Static HTML. Use markup directly in HTML to limit dependence on JavaScript.
- Pre-rendering. Offer pre-rendered pages where JavaScript has already been executed, providing crawlers with fully rendered HTML.
These approaches are in line with Google’s recommendations to prioritize HTML-based development and the inclusion of important content, such as structured data, in the initial server response.
Why this is important
The importance of AI crawlers will only grow over time, and they play by different rules than traditional search engines.
If your site depends on GTM or other client-side JavaScript for structured data, you’re missing out on AI-based search results ranking opportunities.
By switching to server-side or static solutions, you can protect your site from future problems and ensure its visibility in traditional and AI search.