Faceted navigation and crawl budget: how to stop wasting it

Faceted navigation is good for shoppers and hostile to crawlers. Every combination of colour, size, brand, price band and sort order is a distinct URL serving a near-identical page, and the number of them grows multiplicatively.

The result is a crawler spending its time on ?colour=blue&size=m&sort=price_asc instead of on your new products, and a set of thin, competing pages diluting the category page you actually want to rank.

Decide which facets deserve a page at all

A small number of facet combinations genuinely have search demand. Most do not. The test is whether people search for the combination as a phrase.

• Worth indexing: facets that match real queries — "black leather boots", "waterproof walking jacket". These deserve a proper landing page with its own title, its own copy and an internal link.
• Not worth indexing: price ranges, sort orders, pagination beyond the first page, availability toggles, and any combination of three or more facets.
• The rule of thumb: one facet, sometimes two, never three.

Write the list down. The decision is a content decision, not a technical one, and it should survive the next theme change.

The four controls, and what each actually does

1. Canonical tags — tell search engines which version is the real one. The page is still crawled. Use for near-duplicates you are happy to have visited.
2. noindex — keeps a page out of the index but the crawler must fetch it to see the tag. Use for pages that must not rank.
3. robots.txt disallow — genuinely stops the crawl, which is what saves budget. But a disallowed page cannot be read, so its noindex is never seen. Never combine the two on the same URL.
4. Not linking to it — the most effective control of all. Facet links that are not in the HTML as crawlable anchors are largely not discovered.

A configuration that works

1. Indexable facet pages: clean URLs, self-referencing canonical, real internal links from the category page, unique title and short intro copy.
2. Everything else: parameter URLs, canonical pointing at the clean category page, and rel="nofollow" on the filter links or filters applied without creating crawlable links at all.
3. Sort orders and view toggles: never indexable, never linked as anchors.
4. Pagination: page 2 onward self-canonical and indexable if the content is genuinely different, and never canonicalised back to page 1 — that hides the products on later pages entirely.
5. robots.txt: disallow the parameter patterns you have already excluded by other means, once you are confident nothing you want indexed matches them.

Internal linking is the other half

Crawl budget is spent according to what you link to. Most shops link heavily to filters and barely at all to the category pages they want to rank.

• Link to your indexable facet pages from the category page, in the body, as ordinary anchors.
• Link from products back up to the most relevant category and facet page via breadcrumbs.
• Audit your internal link graph once a year. The page with the most internal links is what you are telling search engines is most important, and it is often a filter nobody searches for.

How to tell whether it is working

• Search Console crawl stats. Look at what is actually being fetched. If parameter URLs dominate, nothing above is configured correctly.
• Indexed page count against your real product and category count. Many times larger means facets are in the index.
• Time to index for a new product. The practical symptom of a wasted budget, and the one a merchant will actually notice.