When should you use robots.txt, noindex, canonicals, or authentication to control search access?

Choose the control based on the desired outcome.

Use robots.txt when you want to reduce crawling of URL patterns that search engines do not need to fetch, but understand that blocked URLs can still appear in search if discovered through links. Robots.txt is a crawl directive, not a reliable deindexing mechanism.

Use noindex when a URL may be crawled but should not remain in the search index. Google generally needs to fetch the page to see the noindex directive, so do not simultaneously block it in robots.txt while expecting the noindex to be processed.

Use rel=canonical when duplicate or near-duplicate URLs must remain accessible but you want ranking signals consolidated to a preferred version. Canonical is not a security or removal tool and may be ignored when the pages are not equivalent.

Use redirects when the old URL no longer needs to exist independently and users/search engines should permanently reach another URL.

Use authentication or network-level access controls for genuinely private content. Search directives are not security controls.

A practical decision tree:
1. Private/sensitive? Require authentication.
2. Gone/replaced? Redirect or return 404/410.
3. Accessible but should not index? noindex.
4. Duplicate but needed? canonical.
5. Large low-value crawl space that does not need fetching? robots.txt after confirming you are not hiding directives Google must see.

Audit combinations because conflicts are common. A canonical target should be indexable. A noindexed page should not usually be in the XML sitemap. A robots-blocked page cannot reliably communicate updated meta directives.

The goal is not maximum blocking; it is an intentional crawl/index state for every URL class.