Solved problems. Proven solutions.

Preview URLs should be treated as nonpublic workflow resources unless there is a deliberate reason to expose them. 1. Require authentication or signed expiring tokens for private previews when feasible. 2. Send noindex via meta robots or X-Robots-Tag on any preview that can be reached without authentication. 3. Keep preview URLs out of sitemaps, navigation,…

“Discovered - currently not indexed” usually means Google knows the URL but has not prioritized fetching it yet. The cause can be crawl demand, site quality patterns, or an oversized set of low-value URLs. 1. Confirm the pages are linked internally and not only present in the sitemap. 2. Check whether the site launched a large batch of near-duplicate locati…

Most internal search-result URLs should not be treated as indexable landing pages because they are unstable, low-control, and can create large crawl spaces. Curated filter pages are different. 1. Separate user-generated search URLs from intentionally built landing pages. 2. Keep internal search pages out of XML sitemaps and generally use noindex when they s…

A pre-launch checklist should prove that the public site is crawlable, indexable, canonical, measurable, and mapped correctly from any previous URLs. 1. Remove staging noindex directives, HTTP authentication, and robots rules that should not reach production. 2. Verify canonical URLs, HTTPS, preferred hostname, trailing-slash rules, and redirect behavior. 3…

Pagination should preserve crawlable access to deeper items while avoiding misleading canonicalization. Page 2 is not usually a duplicate of page 1 because it contains different items. 1. Give each pagination URL a stable crawlable URL and generally let it self-canonical when it represents a real page in the sequence. 2. Do not canonical every paginated pag…

Find orphan pages by comparing multiple URL sources rather than relying on one crawler. 1. Crawl the site from the homepage and export every discovered internal URL. 2. Export XML sitemap URLs, CMS content records, Search Console landing pages, analytics landing pages, and backlink-target URLs. 3. Normalize protocol, hostname, trailing slashes, and paramete…

A 200 status only says the server delivered a document. Search engines can still decide the document looks like an error, empty result, unavailable item, or low-value placeholder. 1. Check the visible content. Pages saying “not found,” “no results,” “product unavailable,” or showing only navigation can be classified as soft 404s. 2. Compare the title, H1, a…

Promote facets based on durable demand, inventory stability, uniqueness, and business value. Start with query research and internal search/filter usage to identify combinations people actually seek. Then check whether the facet produces enough stable inventory and unique context to function as a standalone landing page. Strategic facets should receive clean…

Handle the URL according to the product's expected future and whether a real replacement exists. For temporary stockouts, keep the product URL live if the product is likely to return. Clearly show availability, preserve useful specifications/reviews, offer notifications, and link to relevant alternatives. Keep structured data availability accurate. For perm…

Classify variants by whether they represent independently useful landing pages. If a variant has unique search demand, distinct content or specifications, a stable SKU, and can be purchased directly, it may deserve an indexable self-canonical URL. Examples can include materially different models, capacities, or product configurations. If color/size variants…

Each data source answers a different question, so disagreements are expected. A crawler tells you what your current site architecture exposes when requested under the crawler's configuration. It is excellent for status codes, internal links, canonicals, robots directives, rendered HTML comparisons, and template-wide patterns. Server logs tell you what bots…

Design sitemap segmentation around diagnostic cohorts, not arbitrary file-size boundaries alone. Create separate sitemap groups for meaningful templates and business states: products, categories, articles, locations, service areas, documentation, high-priority evergreen pages, and other major content types. If inventory is large, subdivide further by stable…

Create an automated "SEO render diff" for representative URLs on every important template. Capture two versions of each page: the raw HTTP response body and a browser-rendered DOM after the normal application settles. Extract a focused set of SEO-critical fields from both: title, meta robots, canonical, hreflang, H1, primary text, internal links, structured…

A soft 404 is usually a content/state problem, not a status-code mystery. First determine whether the URL represents a real, useful resource that should exist independently. Inspect the rendered main content. Common triggers include near-empty location pages, out-of-stock product pages with no replacement path, search-result pages with zero results, thin ca…

Do not manage Search Console as an error inbox. Start by defining the intended index state for each template and URL class, then compare Google's reported state with that expectation. Build a URL inventory with at least: canonical URL, template, HTTP status, robots state, canonical target, sitemap membership, internal-link count, last meaningful update, and…

Treat redirects as a data-mapping problem and test them before DNS or routing changes go live. Export all meaningful legacy URLs from analytics, Search Console, XML sitemaps, crawls, backlink tools, server logs, and important campaign lists. Deduplicate them into one migration inventory. Assign each old URL one explicit destination based on content equivale…

Choose the control based on the desired outcome. Use robots.txt when you want to reduce crawling of URL patterns that search engines do not need to fetch, but understand that blocked URLs can still appear in search if discovered through links. Robots.txt is a crawl directive, not a reliable deindexing mechanism. Use noindex when a URL may be crawled but sho…

Use XML sitemaps as controlled cohorts of URLs. Their value is not the submit button; it is the ability to compare indexing behavior across known groups. Create separate sitemap files for meaningful templates or business groups such as products, categories, articles, locations, service areas, and priority evergreen pages. Keep every sitemap limited to canon…

Use a fixed diagnostic order so you do not chase symptoms in random tools. 1. Start with the exact URL and establish the expected state. Decide whether the URL should be crawlable, indexable, canonical, and internally discoverable. Many audits become confused because nobody defines the intended state first. 2. Test raw HTTP behavior. Check status code, redi…

Diagnose indexing as a funnel: URL existence -> crawl access -> renderability -> canonical eligibility -> content value -> discovery/signals. 1. Confirm the exact URL returns a stable 200 response. Check redirects, soft-404 behavior, authentication, geofencing, and intermittent server errors. 2. Check robots controls at every layer: robots.txt, meta robots,…

Treat canonicalization as a cluster of signals. The HTML canonical tag is one input, and it can be outweighed by stronger contradictory signals. Debug in this order: 1. Compare the candidate URLs. Check content similarity, status codes, redirect behavior, trailing slashes, protocol/host variants, parameters, pagination, print views, and localized versions.…

Validate what the server returns, what the rendered DOM contains, and what search engines are actually indexing. Those are three different checkpoints. 1. Fetch the raw HTML without executing JavaScript. Confirm the response status, canonical, title, meta robots, structured data, primary heading, internal links, and ideally the primary body content. If the…