Google’s Zoe Clifford Discusses JavaScript SEO & Best Practices

In a recent episode of the “Search Off the Record” podcast, Martin Splitt and John Mueller, members of the Google Search team, sat down with Zoe Clifford from the rendering team to discuss the evolution of JavaScript and its impact on Google Search. 

The conversation provided a deep dive into the challenges and solutions of rendering JavaScript, a crucial aspect of modern web development and search engine optimization (SEO).

The Importance of Rendering in SEO

As Zoe Clifford explained, rendering is the process by which Googlebot executes JavaScript to produce a final view of a webpage, akin to how it appears to users. This view, the Document Object Model (DOM), is essential for search engines to index dynamic content effectively.

In the early days, JavaScript was primarily used for simple tasks like form validation and basic interactivity. However, as web applications grew more complex, JavaScript’s role expanded, leading to the rise of single-page applications (SPAs) and client-side rendering. This shift posed a significant challenge for search engines, which traditionally struggled to execute JavaScript and access dynamic content.

Free SEO Audit: Uncover Hidden SEO Opportunities Before Your Competitors Do

Gain early access to a tailored SEO audit that reveals untapped SEO opportunities and gaps in your website.

Analyze My Site

Google’s Advancements in JavaScript Rendering

Google introduced the “evergreen” Googlebot in 2019, recognizing the need to improve its rendering capabilities. This new version continuously updates to the latest stable version of Chromium, ensuring that Googlebot can handle modern JavaScript features and SPAs.

Zoe highlighted that Googlebot now incorporates a headless browser, enabling it to execute JavaScript and render pages accurately.

A headless web browser is usually executed via a command-line interface without a graphical user interface.

Despite these advancements, rendering JavaScript remains resource-intensive. Zoe noted that Google renders all HTML pages, regardless of their JavaScript usage, balancing the high cost with the necessity to ensure comprehensive indexing of web content. 

Google manages this by optimizing the rendering process and leveraging efficient resource allocation.

Challenges and Solutions in JavaScript Rendering

The podcast discussion shed light on several key challenges and solutions related to JavaScript rendering:

JavaScript Errors and Debugging:

JavaScript can introduce errors that affect rendering. Zoe shared examples of how such errors can lead to incomplete or broken pages, impacting indexing. Developers are advised to handle JavaScript errors gracefully to ensure core content remains accessible even if some scripts fail.

User Agent Shenanigans:

Some websites serve different content to Googlebot than to regular users, a practice known as “user agent shenanigans” in SEO terms Dynamic Rendering. This can lead to inconsistencies and indexing issues. Zoe emphasized the importance of ensuring the content served to Googlebot is the same as what users see.

JavaScript Redirects:

While supported by Google, JavaScript redirects are processed differently than server-side redirects, occurring at render time rather than crawl time. Developers should implement these redirects correctly to avoid indexing and ranking issues.

Handling Cookies and Stateful Sessions:

Googlebot’s rendering is stateless, meaning each session starts fresh without stored cookies or session data. Developers should design websites to ensure essential content is accessible without relying on cookies or session-specific data.

Response and Page Fragility:

Zoe discussed the importance of handling responses gracefully to avoid page fragility. If a web page relies heavily on external API calls and these calls fail or return errors, the page content can break, leading to a poor user experience and indexing issues. Developers should ensure their pages can handle such errors without compromising the core content.

Handling API Failures:

Developers should implement robust error handling to ensure that the failure of an API call does not render the entire page unusable. This can include displaying fallback content, showing error messages, or using cached data when the API is unavailable.

Resources Blocked by robots.txt:

If critical resources like JavaScript, CSS, or API endpoints are disallowed in the robots.txt file, Googlebot cannot fetch and render them. This can lead to incomplete or incorrect indexing of the page content.

Ensuring Resource Accessibility:

Ensure that essential resources are not blocked by robots.txt. Use the Robots.txt Tester in Google Search Console to check the accessibility of these resources and update the robots.txt file as necessary.

More About Dynamic Rendering and SEO Impact

One of the solutions discussed in the podcast for handling JavaScript-heavy websites is dynamic rendering. This approach involves serving a static HTML version of your content to search engines while providing a fully interactive version to users. Dynamic rendering can be particularly useful for websites that rely heavily on JavaScript and want to ensure their content is indexed effectively.

Advantages of Dynamic Rendering:

Improved Indexing: By serving static HTML to search engines, you ensure that all content is accessible and indexable without requiring JavaScript execution.

Better Performance: Reducing the load on Googlebot by serving pre-rendered pages can speed up the indexing process and improve overall performance.

Implementing Dynamic Rendering:

• Use tools like Rendertron or Prerender.io to generate static HTML snapshots of your pages.
• Set up your server to detect user agents (like Googlebot) and serve the static version of your page to them while serving the dynamic version to regular users. 

The Role of Structured Data in SEO

Martin Splitt and Zoe Clifford during Google I/O 2019

During the podcast, John asked Zoe about the implementation of structured data and its impact on SEO for JavaScript-heavy websites. Zoe said structured data helps search engines understand a webpage’s content and context, enabling richer and more relevant search results. Here is what he recommends

Implementing Structured Data:

JSON-LD: The recommended format for structured data is JSON-LD (JavaScript Object Notation for Linked Data). It allows developers to embed structured data within a script tag in the HTML, which is easier for search engines to parse.

Schema.org Vocabulary: Using the Schema.org vocabulary helps define specific types of content, such as articles, products, events, and more. This standardization aids search engines in recognizing and categorizing information accurately.

Benefits of Structured Data:

Enhanced SERP Features: Properly implemented structured data can lead to enhanced search engine result page (SERP) features, such as rich snippets, knowledge panels, and other interactive elements that improve visibility and click-through rates.

Better Content Understanding: Structured data provides additional context, helping search engines understand the relationships between different pieces of content on a page. This can be particularly beneficial for complex web applications relying on JavaScript.

Best Practices for JavaScript SEO

To ensure that JavaScript content is indexed correctly by Google, the podcast highlighted several best practices for developers:

Server-Side Rendering (SSR):

Implement SSR to generate the full HTML content on the server, reducing reliance on client-side rendering and ensuring search engines can index the content effectively.

Progressive Enhancement:

Build web pages with basic HTML content that functions without JavaScript. Enhance the experience with JavaScript to ensure that even if JavaScript fails, the core content remains accessible.

Dynamic Rendering as Last Resort:

Serve a static HTML version of your content to search engines while providing a fully interactive version to users. However, dynamic rendering is a workaround and not a recommended solution.

Testing with Google’s Tools:

Utilize Google Search Console’s URL Inspection tool and Mobile-Friendly Test to check how Googlebot renders your pages and address any issues that prevent proper indexing.

Using Google’s URL Inspection Tool

The URL Inspection tool in Google Search Console is an invaluable resource for developers to ensure their pages are rendered and indexed correctly. Here’s how to use it effectively:

Step 1: Access Google Search Console

Log in to your Google Search Console account and select the property (website) you want to inspect.

Step 2: Enter the URL

In the URL Inspection tool, enter the complete URL of the page you want to test and press Enter.

Step 3: View the Results

The tool will provide detailed information about the URL, including its index status, any crawl errors, and the last time it was crawled.

Step 4: Test Live URL

Click on the “Test Live URL” button to see how Googlebot renders the page in real-time. This test will execute JavaScript and provide a rendered view of the page.

Step 5: Review Rendered Page

Examine the rendered page to ensure all critical content is visible and functioning correctly. Pay attention to any errors or issues that might prevent proper indexing.

Step 6: Fix Issues

If any issues are detected, such as missing content or JavaScript errors, address them promptly. Re-test the URL after making corrections to ensure the problems are resolved.

The Role of Robots.txt in Rendering

The robots.txt file is a crucial component in controlling how search engines crawl and index your site. During the podcast, the team emphasized the importance of properly configuring the robots.txt file to ensure that critical resources are accessible to Googlebot.

Key Points about Robots.txt:

Allowing Access to Resources: Ensure that your robots.txt file does not block important JavaScript, CSS, or API endpoints that are necessary for rendering your pages.

Testing Robots.txt: Use the Robots.txt Tester in Google Search Console to check whether your robots.txt file is correctly configured and make adjustments if necessary.

Best Practices for Robots.txt:

Grant Access to Important Resources: Allow Googlebot to access resources required for rendering, such as JavaScript and CSS files, by not disallowing them in your robots.txt file.

Review Regularly: Review your robots.txt file regularly to ensure it aligns with your site’s current structure and needs.

JavaScript has transformed the way we access web content. Google’s advancements in rendering technology, particularly with the evergreen Googlebot, have bridged the gap between dynamic content and search indexing. 

By following best practices, leveraging structured data, and using tools like the URL Inspection tool, developers can ensure their JavaScript content is user-friendly and search engine-friendly, maximizing visibility and effectiveness in search results.

This episode of “Search Off the Record” offers invaluable insights for developers aiming to optimize their JavaScript-driven websites for Google Search. As JavaScript evolves, staying informed and adapting to best practices will be crucial for maintaining and improving search visibility.