How do you use server log files to find crawl-budget waste instead of guessing from a crawler?

Treat log analysis as behavioral evidence, not a replacement for a site crawl.

First isolate verified search-engine bot requests and normalize URLs before analysis. Remove obvious static assets unless they are part of the investigation, separate Googlebot Smartphone from other user agents, and group requests by template or URL pattern.

Build four views:

1. Crawl distribution by template. Compare bot requests across product pages, category pages, location pages, parameters, search URLs, feeds, APIs, and legacy paths. A large share of requests going to low-value parameter combinations is a strong waste signal.

2. Response-code distribution. Count 200s, 3xx, 404/410, and 5xx responses by bot and by path. Repeated crawling of long redirect chains or dead URLs usually means internal links, sitemaps, external references, or historical signals still expose them.

3. Recrawl frequency. Measure how often important URLs are requested versus low-value URLs. If stale filters are crawled daily while strategic pages are revisited rarely, look for differences in internal linking, sitemap freshness, canonical signals, and update patterns.

4. Crawl-to-index comparison. Join log data with crawl/index data. URLs that receive frequent bot hits but remain noncanonical or nonindexable are prime candidates for cleanup.

Do not use a single “crawl budget score.” Prioritize patterns that consume real requests without contributing to discovery or freshness. Typical fixes include parameter controls, cleaner internal links, removal of crawlable faceted combinations, eliminating redirect hops, retiring obsolete XML sitemap URLs, and consolidating duplicate endpoints.

After changes, compare the next 2–4 weeks of logs. The success signal is not merely fewer requests; it is a larger proportion of bot activity landing on URLs you actually want discovered, refreshed, and indexed.