Technical SEO: The Complete Guide to Crawling, Rendering, Indexing and Speed

Last updated: October 2026.

You published the page, it’s in the sitemap, and Search Console says “Crawled, currently not indexed”. Or the page is indexed, but ChatGPT and Perplexity never cite it, because the copy they fetched was an empty JavaScript shell. Neither problem shows up in a content review. Both are technical SEO.

Technical SEO is the work that gets the right pages through a pipeline: crawled, rendered, indexed, then ranked or cited. Google describes its own processing of JavaScript pages in three phases, crawling, rendering and indexing (JavaScript SEO basics, updated 4 Mar 2026). Every AI answer engine now runs its own crawler in front of the same pipeline, and most of them skip the rendering step. A page that fails at any stage never reaches the next one.

This guide follows the pipeline stage by stage: how crawlers work, which problems cost traffic at which site size, then crawling, AI crawlers, rendering, indexing, architecture, Core Web Vitals and mobile, structured data, international sites, migrations, log files, WordPress, and a five-step audit. It means search-facing work only. Conversion work on landing pages is a separate job and is not covered here.

How search engines and AI crawlers process a site: crawl, render, index

A crawler fetches a URL, a renderer runs its JavaScript, an indexer decides whether to keep it. Googlebot first fetches the URL and parses the HTML for links. Pages that return a 200 status go into a rendering queue, where a headless Chromium executes the JavaScript. Google then indexes the rendered HTML (JavaScript SEO basics, updated 4 Mar 2026). Ranking, and now citation in AI answers, only happens to what made it into an index.

AI engines add crawlers, and most of them stop at the first stage. In Vercel and MERJ’s analysis of crawler traffic across its network (December 2024, GPTBot alone made 569 million fetches in a month), OpenAI’s, Anthropic’s and Perplexity’s crawlers fetched JavaScript files but did not execute them. Gemini uses Google’s rendering. That is late-2024 data, but a 2026 test pointed the same way (see the rendering section).

Each engine has its own crawlers, and each crawler has one job. Blocking the wrong one is the most common technical mistake of the AI era, because the names don’t say what they do.

Crawler or token	Company	What it does	Block it and you lose
Googlebot	Google	Crawls for Google Search, which includes AI Overviews and AI Mode	Google Search, all of it
Google-Extended	Google	A robots.txt token, not a separate crawler, for Gemini training and grounding	Gemini use only. Google says it does not affect inclusion in Search (common crawlers doc, 15 Oct 2026)
Bingbot	Microsoft	Fills Bing’s index, which grounds Copilot	Bing and Copilot
OAI-SearchBot	OpenAI	Builds the index for ChatGPT search	Appearing in ChatGPT search answers
GPTBot	OpenAI	Collects content for model training	Training use only
ChatGPT-User	OpenAI	Fetches a page when a user asks. OpenAI says “robots.txt rules may not apply”	Little, by design
Claude-SearchBot	Anthropic	Indexes content for Claude’s search results	Visibility in Claude search answers
ClaudeBot	Anthropic	Collects content that may be used for training	Training use only
Claude-User	Anthropic	Fetches a page when a Claude user asks	Retrieval for user questions
PerplexityBot	Perplexity	Surfaces and links sites, not used for training	Perplexity answers
Applebot-Extended	Apple	A robots.txt token, not a crawler, for Apple’s AI model training	Training use only. Apple says rules for it “are not considered in ranking for Search” (4 Sep 2026)

Sources: OpenAI crawler docs, Anthropic help centre (updated 7 Apr 2026), Perplexity crawler docs, Apple support (4 Sep 2026), Google common crawlers (15 Oct 2026), all checked 29 Oct 2026.

My read: technical SEO used to have one customer, Googlebot. It now has a dozen, and only one of them renders the way Google does. Every decision below is made for the weakest reader in that table.

Which technical problems cost traffic, by site size

The technical work worth doing depends on how many URLs you have and how they’re built. Google’s crawl budget guide is written for three kinds of site: 1 million+ unique pages changing about weekly, 10,000+ pages changing daily, or sites where a large share of URLs sits in “Discovered, currently not indexed”. Google calls those numbers rough estimates, not thresholds (updated 22 Jul 2026). Search Console’s Crawl Stats report says a site with fewer than a thousand pages should not need it at all.

So split the work by condition:

• Under about 1,000 pages (a blog, a services site, a B2B marketing site): crawl budget is not your problem. What breaks is rendering, accidental noindex, robots.txt mistakes, blocked AI crawlers, orphan pages and slow templates.
• 10,000 to 1 million pages (ecommerce, marketplaces, publishers, programmatic pages): add index bloat from filters, parameters and thin templates, canonical conflicts, and crawl spent on URLs that should not exist.
• 1 million+ pages, or any site mid-migration: everything above, plus crawl capacity, server response time and log file analysis, because you can’t see crawl waste at this size any other way.

One example from the middle-to-large band. At Zolvit (Vakilsearch), a legal services site with 500K+ pages where I led SEO from July to December 2024 [FIRST-PARTY, n=1], the work after a Google update started with index hygiene at scale: we audited 10,000+ blog posts, deleted thin pages first, and moved a large judgments section off the main domain to a separate one, which was later shut. Organic traffic went from about 400K to 700K monthly visits in six months. It is one site, the exact update is unknown (a core or a spam update), and content work ran at the same time, so this does not show which change did what. The full story is on my portfolio.

Size first, tactics second.

Here is how to work through the pipeline on your own site, one stage at a time.

Crawling: robots.txt, sitemaps and crawl budget

robots.txt decides what a crawler may fetch. It does not decide what gets indexed. Google’s robots.txt introduction says a disallowed URL can still be indexed if other sites link to it, and shows up without a description. To keep a page out of Google, use a noindex tag or put it behind a password (updated 10 Dec 2025).

First action: open yoursite.com/robots.txt and read every Disallow line aloud. For each one, say which crawler it applies to and why. Rules nobody can explain are the ones that hide pages.

What robots.txt is for:

• Keeping crawlers out of infinite or useless URL spaces: internal search results, faceted filter combinations, cart and session URLs.
• Saving crawl on a large site. Google’s crawl budget guide says robots.txt, not noindex, is the tool for this, because Google still requests a noindexed page before dropping it (updated 22 Jul 2026).
• Per-crawler decisions for AI bots, covered in the next section.

What it is not for: de-indexing pages, hiding private content, or blocking pages you also tagged noindex. A page blocked in robots.txt can’t show Googlebot its noindex tag, so the two together keep a URL half-alive in the index.

Sitemaps list the URLs you want indexed, nothing else. Put only canonical, indexable, 200-status URLs in them. Google calls a sitemap a weak canonical signal and a redirect or rel=canonical a strong one (canonicalization doc, updated 10 Jul 2026), so a sitemap full of redirected or parameter URLs sends mixed signals for no gain. Submit it in Search Console and reference it in robots.txt.

Crawl budget, if you’re in the large-site band. Google splits it into crawl capacity (how much server time Google is willing to use, which rises and falls with your site’s health) and crawl demand (how much it wants your URLs, based on size, freshness, quality and popularity). The levers are the crawl budget guide’s own: consolidate duplicates, block URLs that should never be crawled, return 404 or 410 for removed pages, fix soft 404s, keep sitemaps current, avoid long redirect chains, and keep the server fast. Don’t use robots.txt to shuffle budget temporarily between sections, which the guide warns against.

• Can someone on the team explain every Disallow rule?
• Does any sitemap contain a URL that redirects, 404s or carries noindex?
• Is any page both disallowed and tagged noindex?

Read the file before you touch it.

AI crawlers and robots.txt: GPTBot, Google-Extended and the rest

Decide per crawler, by purpose: training or search. Every major AI company now separates the crawler that feeds model training from the one that feeds its answer engine. Blocking a training crawler costs you nothing in that company’s search product. Blocking its search crawler removes you from it.

First action: list every user-agent your robots.txt names. Then mark each as training, search or user fetch using the crawler table above, and confirm that every block matches what the business decided.

If you want to	Disallow	Leave open
Be cited in ChatGPT search but stay out of OpenAI training	GPTBot	OAI-SearchBot
Be cited in Claude but stay out of Anthropic training	ClaudeBot	Claude-SearchBot, Claude-User
Stay out of Gemini training and grounding, keep Google Search	Google-Extended	Googlebot
Stay out of Apple AI training, keep Apple search features	Applebot-Extended	Applebot
Appear in Perplexity answers	nothing	PerplexityBot
Appear in Copilot answers	nothing	Bingbot

Three facts change most robots.txt decisions:

• Google-Extended does not remove you from AI Overviews or Search. Google’s common crawlers page says it does not affect inclusion in Google Search and is not a ranking signal (updated 15 Oct 2026). To limit what Google’s AI features show from a page, Google points to snippet controls such as nosnippet and max-snippet.
• OAI-SearchBot, not GPTBot, controls ChatGPT search. OpenAI’s crawler documentation says blocking OAI-SearchBot removes a site from ChatGPT search answers, while GPTBot covers training (checked 29 Oct 2026).
• Fetchers acting on a user’s request may ignore the file. OpenAI says robots.txt “may not apply” to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it.

The condition. If you sell the content itself (a publisher, a data provider, a paywalled course), blocking the training crawlers is a reasonable business call. If you’re a B2B company or a brand that wants to be named in AI answers, keep every search crawler open and put your effort into the next section. How AI visibility is earned and measured is in the GEO guide.

• Who in the business decided our AI crawler policy, and when?
• Does any rule block a search crawler we want citations from?
• Did a CDN or security plugin add bot blocking that robots.txt doesn’t show?

Block training if you must. Never block search by accident.

Rendering JavaScript: put content and links in the initial HTML

If your main copy or internal links only exist after JavaScript runs, assume most AI crawlers never see them. Google renders JavaScript, though rendering is a separate, later step. Most AI crawlers didn’t render it at all in late-2024 data. Google’s own guide still recommends server-side rendering or pre-rendering, because it’s faster for users and crawlers and not every bot runs JavaScript.

First action: open a key page, switch JavaScript off in Chrome DevTools (or view the raw page source, not the inspected DOM), and check two things: is the main copy there, and are the internal links real a elements with an href? What you see now is roughly what GPTBot sees.

The evidence that this matters now. In a controlled test on one 2,400-page directory site (Search Engine Land, August 2026), half the internal links were hard-coded HTML and half were injected by JavaScript. GPTBot crawled all 759 HTML-linked pages on day one and none of the JavaScript-linked hierarchy. ClaudeBot, PerplexityBot and Bingbot found no JavaScript-linked pages either. It’s one site over 27 days, so read it as a direction, not a rate.

What to fix, in order:

1. Render the main content and navigation on the server (SSR, static generation or pre-rendering) for every page you want found. Most modern frameworks offer this as a setting.

2. Use real links. Google’s link documentation says a link is crawlable when it’s an a element with an href. Click handlers on spans and buttons are not links.

3. Don’t let JavaScript change indexing signals. Google’s JavaScript SEO guide warns that Google may skip rendering when it meets a noindex tag, so removing noindex with JavaScript may not work. Ship the right robots tag, canonical and title in the initial HTML.

4. Return real status codes. A single-page app that shows “not found” with a 200 status creates soft 404s. Google suggests a noindex robots tag added by JavaScript on such error views if a real 404 is impossible.

When to skip it: logged-in app screens, dashboards and checkout steps have no search value. Leave them client-side.

• Does the main copy of our top 20 pages appear with JavaScript off?
• Are our menus, breadcrumbs and related-post links real a href links?
• Is any noindex, canonical or title set only by JavaScript?

View source before you buy a rendering tool.

Indexing and canonicals: decide which URL represents each page

Indexing is Google’s decision, made per URL, and the Page indexing report shows you every decision. Google states it doesn’t index every crawlable page. “Crawled, currently not indexed” means Google fetched the page and chose not to keep it, and resubmitting changes nothing. “Discovered, currently not indexed” means the crawl was postponed (Search Console Help, checked 29 Oct 2026).

First action: in Search Console, open Indexing, then Pages. Sort the “Why pages aren’t indexed” table by count and open the top three reasons. For each, sample ten URLs and sort them into “should be indexed” and “should not exist”.

Read the reasons as three groups:

• You told Google not to index it (excluded by noindex, blocked by robots.txt, alternate page with proper canonical). Fine, if intended. Check the sample.
• Google picked a different URL (duplicate without user-selected canonical, Google chose a different canonical than the user). Your canonical signals disagree with each other.
• Google chose not to (crawled or discovered, currently not indexed). Usually a quality or duplication judgment on thin or near-identical pages, sometimes a crawl demand issue on large sites.

Canonicals are a strong hint, not a command. Google calls a redirect and rel=canonical strong signals, and a sitemap a weak one, and says a small site will likely do fine without declaring any canonical preference (updated 10 Jul 2026). On a large or templated site, declare one and make every signal agree: the canonical tag, internal links, the sitemap and any hreflang all point at the same URL. Common conflicts are http and https, www and non-www, trailing slashes, uppercase paths, tracking parameters and filter combinations.

Before you noindex or delete, decide by condition. On a large site with many low-value templated pages, removing or consolidating them is often the fix. On a small site with thin but unique posts, improving or merging them is usually better than deleting. The page-level side of this is in the on-page SEO guide.

• What share of our indexed pages would we be happy to show a buyer?
• Do our canonical tag, internal links and sitemap point at the same URL for each page?
• Which “not indexed” reason has grown fastest in the last three months?

Sample ten URLs before deciding anything.

Site architecture and internal linking: every page you care about, a few clicks from home

Internal links are how crawlers find pages and how you tell them which pages matter. Google’s link documentation asks that every page you care about is linked from at least one other page on the site. Orphan pages, reachable only from a sitemap, get discovered late and crawled rarely. AI crawlers are less patient than Google, as the rendering test showed.

First action: crawl the site with any desktop crawler, export pages by click depth and by inlink count, then compare that list with your sitemap. Every sitemap URL with zero internal links is an orphan. Every money page deeper than four clicks is a candidate for a better link.

What a sound architecture looks like:

• Shallow and grouped by topic. Home, then category or hub pages, then detail pages. A hub that links to every page in a topic cluster, and pages in the cluster that link back and to each other, is the structure both crawlers and readers follow.
• Readable, stable URLs. Google recommends words over IDs and hyphens over underscores (URL structure doc, updated 21 Oct 2026). Change URLs only when you have to, because every change is a small migration.
• Breadcrumbs that are real links, on every page below the top level.
• Descriptive anchor text in body copy, not “click here”. Navigation and footer links count, but contextual links say more about the target.
• Pagination with plain linked pages, so items on page five are reachable without JavaScript.

The condition: if you run a 50-page site, this is an afternoon’s work. If you run a 50,000-page catalogue, it’s a template decision: fix the category template and the related-items module, not individual links.

• Which of our ten most valuable pages have the fewest internal links?
• Can a crawler reach page five of every listing without running JavaScript?
• Does each topic we want to be known for have a hub page?

Link to what earns money first.

Core Web Vitals and mobile: measured on real users, judged on the phone

Core Web Vitals count, but as part of page experience, not a single switch. Google’s page experience documentation says Core Web Vitals are used by its ranking systems and that there is no single page experience signal (updated 22 Sep 2026). Treat them as a floor to clear, especially when competing pages are equally good, not as a growth lever.

The three metrics and their “good” thresholds, measured at the 75th percentile of page loads, split by mobile and desktop (web.dev):

Metric	Measures	Good	What usually breaks it
Largest Contentful Paint (LCP)	Loading: when the main content appears	2.5 seconds or less	Slow server response, a large unoptimised hero image, render-blocking CSS and fonts
Interaction to Next Paint (INP)	Responsiveness: delay after every tap, click or key press	200 milliseconds or less	Heavy JavaScript on the main thread: tag managers, chat widgets, sliders, third-party scripts
Cumulative Layout Shift (CLS)	Visual stability: how much the page jumps	0.1 or less	Images and ads without set dimensions, late-loading banners, font swaps

INP replaced First Input Delay as a Core Web Vital on 12 March 2024 (web.dev). FID measured only the first interaction. INP measures every one, so sites that passed easily before can fail now.

First action: in Search Console, open Core Web Vitals and read the mobile report first. It groups URLs by template, so one failing group usually means one template to fix. Then run PageSpeed Insights on one URL from each failing group. Field data (from real Chrome users) is what counts. Lighthouse lab scores are for diagnosis.

Mobile is the version Google indexes. Google uses the mobile version of a site’s content, crawled with the smartphone agent, for indexing and ranking (mobile-first indexing doc, updated 10 Dec 2025). So:

• Same primary content on mobile and desktop. Use tabs or accordions instead of cutting copy on mobile.
• Same structured data and the same robots meta tags on both versions. A noindex on mobile only is a noindex.
• Don’t lazy-load primary content on user interaction. Content that waits for a tap may never be seen.

The condition: for most sites, fixing the worst mobile template group is the whole job. Chasing a perfect lab score on a page that already passes in the field is time better spent elsewhere, unless that page competes with near-identical pages, which is where Google says page experience counts most.

• Which template group fails on mobile, and which metric fails it?
• Which third-party scripts load on every page, and who owns each?
• Does the mobile page carry the same content, schema and robots tags as desktop?

Fix the template, not the page.

Structured data that still earns rich results

Structured data earns a place in rich results only for the types Google still supports, and it guarantees nothing. Google’s introduction says markup can make results more engaging, recommends JSON-LD, and warns that markup which breaks its guidelines may be ineligible for rich result display. It does not call structured data a ranking signal. Google’s AI optimization guide adds that structured data isn’t required for generative AI search (updated 10 Jul 2026).

First action: open Google’s search gallery, list the types that match what your pages genuinely contain, then test one URL per template in the Rich Results Test.

Types worth implementing, by site type:

• Every site: Organization (logo and name), WebSite, BreadcrumbList.
• Ecommerce: Product with offers, price, availability and shipping, plus Review and AggregateRating where the reviews are real and on the page.
• Publishers and blogs: Article, with author and dates that match the visible page.
• Local businesses: LocalBusiness with the same name, address and hours as the Business Profile.
• Video, events, recipes, jobs, software apps: their own types, where the page is about that thing.

Rules that keep markup eligible: mark up only what’s visible on the page, keep values in sync with the page (a price in schema that differs from the page is a guideline breach), and fix errors in Search Console’s enhancement reports after each template release.

Skip anything Google has retired. FAQ and HowTo rich results no longer show (details in the table near the end). Adding their markup now is harmless and earns nothing in Google.

• Does each template’s markup match what a visitor sees on the page?
• Which supported type fits our main page type that we haven’t implemented?
• Who checks the enhancement reports after a template release?

Mark up what’s true, then stop.

International SEO and hreflang

hreflang tells Google which language or regional version of a page to show which searcher. It only works when every version lists every other version, and itself. Google’s localized versions guide accepts the annotations in HTML, in HTTP headers or in a sitemap, and treats the three as equivalent (updated 21 Sep 2026).

First action: pick one page that exists in several languages or countries. Open each version and check that each lists itself and all the others, with the same set of URLs, and that every listed URL returns 200 and is canonical.

The rules that catch most sites:

• Return links are mandatory. If page A lists page B and B doesn’t list A, Google ignores the pair.
• Codes are ISO 639-1 for language and ISO 3166-1 Alpha 2 for region: en-gb, not en-uk. A region alone is not valid.
• Use x-default for a language selector or an auto-redirecting home page. It applies when no listed language or region matches the user.
• hreflang and canonical must agree. Each version is canonical to itself. A French page canonicalised to the English one tells Google to drop the French page.
• Google doesn’t detect language from hreflang or the lang attribute. It reads the content. A “German” page with English body copy is an English page.

Pick the structure before the tags. Country domains (example.de), subdirectories (example.com/de/) and subdomains (de.example.com) all work. Subdirectories are the easiest to run on one CMS, and country domains send the clearest country signal at the cost of separate sites to maintain.

The condition: if you serve several countries in one language with the same content, you usually need no hreflang at all. Add it when you change the content, currency or language per market.

• Does every version of our top page list itself and all the others?
• Is every hreflang URL canonical to itself and returning 200?
• Do we have an x-default for visitors outside our listed markets?

Test one page fully before templating it.

Migrations and redirects: moving without losing the index

A migration is the one technical project that can lose a site’s traffic in a week. Domain changes, URL restructures, HTTPS moves, CMS changes and platform rebuilds all ask Google to re-learn every URL. Google’s site move guide recommends permanent redirects (301 or 308), keeping them at least a year, and submitting Change of Address in Search Console for domain or subdomain moves. It says a small to medium site can take a few weeks for most pages to move, larger sites longer, and that visibility may fluctuate during the move (updated 20 Aug 2026).

First action, before anything is built: export every URL that has had a click or a backlink in the last 16 months (Search Console Performance plus your backlink tool), and put it in a spreadsheet with one column: its new URL. That redirect map is the migration. Everything else is execution.

The sequence:

1. Benchmark. Save Search Console clicks and impressions by page, the Page indexing counts, Core Web Vitals and a full crawl of the old site.

2. Map one-to-one. Each old URL redirects to its closest new equivalent, in one hop. Redirecting everything to the home page throws away that mapping, and Google may treat such redirects as soft 404s.

3. Carry the signals. Titles, headings, body copy, canonical tags, hreflang, structured data, internal links and image URLs come across, or change on purpose.

4. Test on staging, behind a password, not a robots.txt block that might ship to production.

5. Launch and verify. Crawl the old URL list and confirm every one returns a single 301 to a 200 page. Submit the new sitemap.

6. Watch for weeks, not days. Compare the Page indexing report and clicks by page against the benchmark weekly.

One thing I’d check that most migration lists miss. When I migrated this site on my own [FIRST-PARTY, n=1], most posts came across into WordPress with the same publish date, 18 February 2022, the migration date, instead of their original dates. Nothing broke, but every date a reader, an Article schema field or an AI engine reads was now wrong. Map dates, authors and other metadata fields in the redirect sheet, not only URLs.

The condition: a URL-preserving CMS change on a small site is low risk if the templates keep their HTML. A domain change plus URL restructure on a large site is the highest-risk case, so do them as two separate moves if you can.

• Does every old URL with clicks or links have exactly one new URL in the map?
• What is our rollback plan if indexing drops in week two?
• Which metadata fields (dates, authors, schema) did we check after the move?

The map is the migration.

Log files: what each crawler requested

Server logs are the only record of what every crawler requested, including AI crawlers Search Console never shows. Search Console’s Crawl Stats report covers Google’s side: total requests, download size, average response time, host status over the last 90 days, and breakdowns by response, file type, purpose (discovery or refresh) and Googlebot type. It’s available only for root-level properties, and Google says sites under a thousand pages shouldn’t need it.

First action: open Settings, then Crawl Stats in Search Console. If host status isn’t green, or average response time is climbing, fix the server before anything else on this page. Then ask your host or CDN for 30 days of raw access logs.

What logs answer that nothing else can:

• Where crawl goes. Count requests by URL pattern. If parameter and filter URLs take most of Googlebot’s requests, that’s the crawl waste to block.
• What never gets crawled. Compare crawled URLs against your sitemap. Money pages not requested in 30 days have a linking or demand problem.
• Which AI crawlers visit, and how often. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and the rest appear by user-agent. This is where you confirm a robots.txt or CDN change worked.
• Status codes as crawlers saw them. Redirect chains, 5xx spikes and soft errors that a browser test misses.

Verify before you trust a user-agent. Anyone can claim to be Googlebot. Google’s verification guide gives two methods: a reverse DNS lookup that must resolve to googlebot.com, google.com or googleusercontent.com, followed by a forward lookup that returns the same IP, or matching the IP against Google’s published IP range files (updated 20 Mar 2026). Anthropic doesn’t publish IP ranges, so its bots can only be identified by user-agent.

The condition: if you have fewer than a few thousand pages, the Crawl Stats report is enough. Above 100,000 pages, or during a migration, logs are the only way to see crawl waste.

• What share of Googlebot’s requests last month went to URLs we don’t want indexed?
• Which important pages weren’t crawled at all in 30 days?
• Which AI crawlers visit us, and does that match our robots.txt policy?

Thirty days of logs settle most crawl arguments.

Technical SEO on WordPress: the settings that cause most problems

WordPress handles most technical basics out of the box, and most WordPress technical problems come from one setting, one plugin or one theme. W3Techs counts WordPress on about 40% of all websites (October 2026), so these are the issues an SEO meets most often.

First action: in WordPress, open Settings, then Reading, and confirm “Discourage search engines from indexing this site” is unticked on the live site.

The WordPress-specific checks:

• The visibility checkbox. Since WordPress 5.3 (September 2019), that setting adds a noindex, nofollow robots meta tag instead of a robots.txt Disallow (WordPress core announcement, 2 Sep 2019). It also turns off WordPress’s own sitemaps. Ticked on staging, then copied to production, it de-indexes the whole site.
• One sitemap source. WordPress core has published a sitemap index at /wp-sitemap.xml since version 5.5 (July 2020). SEO plugins usually replace it with their own. Submit whichever one is live, and only one.
• Thin archive pages. Tag archives, author archives on single-author sites, date archives and attachment pages create indexable URLs with little content. Set them to noindex in your SEO plugin unless they earn traffic.
• Speed is mostly theme and plugins. HTTP Archive’s Web Almanac put WordPress’s mobile Core Web Vitals pass rate at 45% in its 2025 CMS chapter (published 15 Jan 2026), among the lowest of the platforms compared, up from 40% in its 2024 edition. The average hides a wide spread: a lean theme, fewer plugins, server caching and compressed images move most sites past the thresholds.
• Page builders and JavaScript. Check rendered output for builder-made menus and tabs with JavaScript off, as in the rendering section.

What this page doesn’t cover: plugin security, updates and hacked-site cleanup are in Is WordPress secure?, and choosing WordPress at all is in advantages of WordPress.

• Is the visibility checkbox unticked on production, and ticked anywhere it shouldn’t be?
• Which sitemap URL is live, and is it the one submitted in Search Console?
• Which archive types are indexed, and does any of them earn clicks?

Check the checkbox after every launch.

How to run a technical SEO audit: five steps in pipeline order

An audit works in the same order as the pipeline: access, rendering, indexing, experience, then monitoring. Fixing speed on a page that isn’t indexed is wasted effort. Each step below starts with a first action in a named tool. The full checklist with every item is the SEO audit checklist.

1. Check that every crawler you want can get in

First action: read robots.txt, then test your five most valuable URLs in Search Console’s URL Inspection tool and confirm “Crawl allowed: Yes” and “Indexing allowed: Yes”.

Then check the CDN or firewall bot settings and the last 30 days of logs for 403s to Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot.

• Does any rule, file or firewall block a search crawler we want?
• Is any production page carrying a leftover staging noindex?
• Does URL Inspection show the canonical we expect?

Start with access. Everything else depends on it.

2. Compare the raw HTML with the rendered page

First action: for one URL per template, compare view source with URL Inspection’s rendered HTML (View crawled page). Anything that only appears in the rendered version is invisible to most AI crawlers.

• Is the main copy in the raw HTML?
• Are navigation and related links plain a href links?
• Are title, canonical and robots tags in the raw HTML?

One URL per template is enough.

3. Sort the Page indexing report into keep and remove

First action: export Indexing, then Pages, sample ten URLs from each of the top three “not indexed” reasons, and label each URL keep, merge, or remove.

• Are any pages we want indexed sitting in “Crawled, currently not indexed”?
• Do canonical tags, internal links and the sitemap agree?
• Which indexed URLs should not exist at all?

Decide on samples, then fix templates.

4. Fix the worst Core Web Vitals template on mobile

First action: open the Core Web Vitals mobile report, pick the URL group with the most poor URLs, and run PageSpeed Insights on one example to find which metric fails and why.

• Which single metric fails across most poor URLs?
• Which third-party scripts can be removed or delayed?
• Does the mobile version carry the same content and tags as desktop?

One template fix beats fifty page fixes.

5. Put the checks on a schedule

First action: set a monthly calendar entry to re-run steps 1 to 4 and annotate every release, migration and Google update next to your Search Console data.

At Forage AI I run technical audits through Claude agents connected over MCP and APIs [FIRST-PARTY, n=1], alongside internal linking and reporting jobs, so the checks run on a schedule instead of when someone remembers. Agents read the same reports a person would. They don’t replace deciding what to fix.

The condition: if you run a 50-page site, a quarterly check is enough. A large site that ships code weekly needs automated checks on every release, because one template change can noindex thousands of URLs overnight.

• Who owns the technical check each month?
• Does every release start a crawl of the changed templates?
• Where do we log what changed and when?

Schedule it, or it won’t happen.

What stopped being true

Technical advice that was once correct and isn’t now. Each row has a primary source you can check.

What	When	Primary source
First Input Delay as a Core Web Vital	Replaced by INP on 12 March 2024	web.dev, INP launch post, 12 Mar 2024
Google’s Mobile-Friendly Test, its API and the Mobile Usability report	Retired from 1 December 2023 (mobile-first indexing itself still applies)	Google Search Central blog, 19 Apr 2023
Search Console’s URL Parameters tool for parameter handling	Deprecated, announced 28 March 2022 and removed a month later	Google Search Central blog, 28 Mar 2022
HowTo rich results	Stopped showing from 13 September 2023	Google Search Central blog, Aug 2023
FAQ rich results	No longer shown in Google Search from 7 May 2026	Google Search Central documentation updates
WordPress’s “Discourage search engines” adding Disallow: / to robots.txt	Switched to a noindex robots meta tag in WordPress 5.3	Make WordPress Core, 2 Sep 2019
Blocking Google-Extended to leave AI Overviews	Google-Extended does not affect Search inclusion or ranking	Google common crawlers doc, updated 15 Oct 2026
Blocking GPTBot to leave ChatGPT search	OAI-SearchBot controls ChatGPT search, GPTBot only training	OpenAI crawler documentation, checked 29 Oct 2026
Using robots.txt Disallow to remove a page from Google	Never worked as a removal method: use noindex or a password	Google robots.txt introduction, updated 10 Dec 2025

Pulling it all together

Access first: this week, read robots.txt and your CDN bot rules, and confirm Googlebot, Bingbot and the AI search crawlers you want can reach your key pages.

Rendering second: view source on one page per template and move anything that only appears after JavaScript into the server-rendered HTML.

Indexing third: sample the top three “not indexed” reasons in Search Console and decide keep, merge or remove for each sample.

Structure fourth: find orphan and deep money pages in a crawl and link to them from strong pages.

Experience fifth: fix the worst mobile Core Web Vitals template group.

Then the specialist jobs, only if they apply to you: hreflang if you serve several languages or countries, a redirect map before any migration, log files above a few thousand pages, and structured data for the types your pages genuinely contain.

FAQ

What is technical SEO?

Technical SEO is the work that lets search engines and AI crawlers fetch, render, index and understand a site: robots.txt and sitemaps, JavaScript rendering, indexing and canonical signals, site architecture, Core Web Vitals and mobile, structured data, hreflang, redirects and migrations, and log analysis.

What’s the difference between technical SEO and on-page SEO?

Technical SEO is sitewide and template-level: whether pages can be crawled, rendered and indexed, and how fast they are. On-page SEO is what you change on one page, such as content, titles and headings. The two overlap on indexing tags and canonicals, which are set per page but usually fixed per template.

Do small sites need to worry about crawl budget?

Usually not. Google’s crawl budget guide is written for sites with over a million pages changing weekly or over 10,000 changing daily, and the Crawl Stats report says sites under a thousand pages shouldn’t need it. Small sites lose far more to rendering and indexing mistakes.

Does blocking AI crawlers hurt my Google rankings?

Blocking Google-Extended, GPTBot, ClaudeBot or Applebot-Extended doesn’t affect Google Search, according to Google’s and Apple’s documentation. Blocking Googlebot does. Blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes you from those AI search products, not from Google.

Is page speed a ranking factor?

Google says Core Web Vitals are used by its ranking systems, within page experience, and that there is no single page experience signal (Sep 2026). A faster page won’t outrank a more useful one. A slow one can lose close contests and loses visitors either way.

Does Google index JavaScript websites?

Yes. Google renders JavaScript with headless Chromium in a step after crawling. The risk is the other crawlers: most AI crawlers didn’t execute JavaScript in Vercel’s 2024 data, so content and links that need JavaScript are invisible to them. Server-side rendering covers both.

In a nutshell

The honest timeline: access and rendering fixes can show in Search Console within days of a re-crawl, but indexing changes on a large site take weeks, and Google says even a clean migration can fluctuate for weeks. Technical work removes what’s holding a site back. It doesn’t make thin content rank, so judge it by indexed pages and crawl health first, and by traffic over months.

When this guide doesn’t apply: if your site is a few dozen server-rendered pages on a well-maintained platform, most of the work here is a one-time check of robots.txt, indexing and mobile speed. Your time is better spent on content and links, which the on-page SEO guide above covers.

If you’d rather have someone run a technical audit or plan a migration with you, that’s what I do in a consultation.

About this article

How this one was checked. Every number traces to a named source with its dataset and period, and every platform feature, policy or limit cites the platform’s own documentation with the date it was verified. Where the author’s own results appear, they are labelled as one account’s results, with the budget and context, not as a rule.

No affiliate links and no paid placements. Tools are named because they are what practitioners use, not because anyone paid to appear here.

Reviewed: October 2026. Ad platforms and search features change every few months; anything we could not verify at the time of writing is marked as unverified rather than turned into a number.

Corrections. If something here is wrong or has changed since, we would rather know. Corrections are made in place, with a note saying what changed.