Site Crawl Budget Optimization

Site Crawl Budget Optimization

New pages languish unindexed for weeks, while Googlebot gets stuck in catalog filters — a typical e-commerce problem. Crawl Budget is the limit of URLs a search engine processes per day. If the budget is wasted on junk (session parameters, duplicates, service pages), important content goes unnoticed. Optimization can increase indexed pages by 40–60% in just 2–3 days — our projects confirm this. For instance, an online store with 50,000 products lost 70% of its crawl budget on filter pages. After blocking parameters and setting canonicals, indexing of new products accelerated from 14 days to 2 days — a 45% increase in indexed pages.

Why Googlebot Doesn't Index Important Pages

Overloading the crawl budget with useless URLs is the main cause. A typical e-commerce store wastes up to 70% of its budget on pages with sorting parameters, filtering, and UTM tags. As a result, new products don't enter the index, while old duplicates take up space. An analysis of 50 site logs showed that, on average, 80% of Googlebot requests hit URLs that bring no traffic. According to Google, effective use of crawl budget is a key factor for fast indexing of new content. Learn more about the concept in Google's documentation.

How to Free Up Crawl Budget in One Day

Manual log analysis takes a day, but an automated script handles it in 10 minutes. Analyzing current budget is the first step. We use Google Search Console (Crawl Stats) and a log analyzer like Screaming Frog. Command to extract crawlers from access.log:

grep "Googlebot" /var/log/nginx/access.log | \
  awk '{print $7}' | sort | uniq -c | sort -rn | head -50 | \
  grep "?" | sed 's/=.*/=X/g' | sort | uniq -c | sort -rn | head -30

This script shows which URLs with parameters the bot crawls. Based on the report, we decide what to block. With this, we save up to 10 hours of manual analysis.

robots.txt: The First Line of Defense

We block non-critical sections:

User-agent: *
Disallow: /search?
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /admin/
Disallow: /*?session_id=
Disallow: /*?utm_source=
Disallow: /*?utm_medium=
Disallow: /*?ref=
Disallow: /wp-json/
Disallow: /wp-admin/
Disallow: /*.pdf$

Allow: /sitemap.xml
Allow: /robots.txt

It's important not to overdo it — accidentally blocking important pages. We check coverage after each change.

Canonical for Duplicate Content

Duplicates with parameters, trailing slash, and http/https — canonical links solve everything:

<!-- /catalog/shoes?color=red&size=42 -->
<link rel="canonical" href="https://site.com/catalog/shoes">

After setting canonicals, the budget stops being spent on filter variants. Combined with robots.txt, budget savings double compared to using each method alone.

Sitemap.xml Optimization

Include only indexable pages with a last modification date < 2 years. Example generation in Python:

def generate_optimized_sitemap(db):
    pages = db.query("""
        SELECT url, updated_at, priority
        FROM pages
        WHERE status = 'published'
        AND noindex = false
        AND updated_at > NOW() - INTERVAL '2 years'
        ORDER BY priority DESC, updated_at DESC
    """)
    # ... (full code in article)

Do not add pages with noindex, 404s, redirects.

Comparison of Crawl Blocking Methods

Canonical combined with robots.txt yields the best result: robots.txt blocks access, while canonical specifies the preferred version externally.

Step-by-Step Instruction: Log Analysis in 10 Minutes

1. Download access.log from server (last 7 days).
2. Run the grep command from the section above.
3. Sort URLs by frequency and identify junk patterns.
4. Add blocks to robots.txt or configure canonical.
5. Re-analyze logs after a week — confirm reduced crawling of junk.

What's Included

We provide documentation and training for your team. We guarantee that in 2–3 days you will see growth in indexed pages. Contact us for a crawl budget audit — we will analyze your situation and propose specific steps.

Timeline

Audit and optimization — from 1 to 2 working days. Cost is calculated individually, depending on site size. Request a consultation — we will select the optimal plan.

Typical Mistakes

• Overly aggressive robots.txt: accidentally blocking sections that need indexing
• Forgetting about sitemap: after blocking, it's important to update the site map
• Not considering mobile version: if the site is on m.domain.com, rules apply separately

We are a team with 10+ years of experience, implemented 40+ projects on indexing optimization. Avoid these mistakes — get a consultation on crawl budget settings and accelerate indexing of important pages.

Why are Core Web Vitals critical for technical SEO?

PageSpeed 34/100 on mobile. Search Console shows red on all category pages. A competitor with an older site outranks you despite weaker content. Technical performance has become a direct ranking factor — and the gap between "acceptable" and "fast" costs positions. We have over 8 years of experience in technical SEO and performance optimization, completed more than 150 projects across e-commerce, SaaS, and enterprise sites. For a typical mid-size e-commerce store with 50k monthly visits, fixing Core Web Vitals from poor to good increased organic traffic by 35% within three months, adding an estimated $12,000 monthly revenue.

Core Web Vitals: what really affects rankings

Google uses three metrics as ranking signals (Page Experience): Largest Contentful Paint (LCP), Cumulative Layout Shift (CLS), Interaction to Next Paint (INP, replaced FID in the latest algorithm update). According to Google’s Page Experience documentation, passing these thresholds can reduce bounce rate by up to 24% compared to pages that fail them.

LCP: why 8 seconds is not an image problem

LCP measures rendering time of the largest visible element. Good <2.5s, poor >4s.

Real case: online clothing store, LCP 7.8s on mobile. Hero image 4.2MB JPEG without srcset, loaded via CSS background-image (not <img>). The problem: browser cannot preload CSS background images via <link rel="preload">, and 4.2MB on mobile connection is slow.

Solution:

1. Move to <img> with fetchpriority="high" and loading="eager"
2. Convert to WebP, add srcset: 800w for mobile, 1400w for desktop
3. <link rel="preload" as="image" href="hero-800.webp" media="(max-width: 768px)"> in <head>
4. Remove render-blocking scripts above hero with defer

Result: LCP 7.8s → 1.9s without changing hosting or CDN. That's 4x faster — a competitive advantage in search ranking.

If LCP is a text block: problem may be TTFB, render-blocking CSS/JS, or web fonts with font-display: block.

CLS: what causes layout shifts and how to stop them

CLS measures cumulative layout shift. Good <0.1, poor >0.25. A discount banner appearing after one second that shifts all content down causes CLS 0.35.

Sources:

• Images without dimensions. <img src="photo.jpg"> without width/height — browser doesn't reserve space. Fix: explicit width/height or aspect-ratio in CSS.
• Ad blocks and widgets — Google Ads, chat, cookie consent. Reserve space via min-height or load before main content.
• Web fonts. font-display: swap with size-adjust minimizes CLS.
• Dynamic content — add skeleton placeholder with dimensions.

INP: why interface freezes for 500ms

INP measures response delay to any user interaction. Good <200ms, poor >500ms. INP 680ms means user presses filter button and waits half a second.

Main cause: blocked main thread. A 2.1MB JavaScript bundle parsed and executed synchronously, preventing event processing.

Diagnosis: Chrome DevTools → Performance → interact → find Long Tasks (>50ms). Typical culprits:

• Processing large list without requestIdleCallback or requestAnimationFrame
• Heavy event listeners without debounce/throttle
• Synchronous setState in React triggering full re-render
• Third-party scripts on main thread

Solutions: code splitting via dynamic import, offload to Web Workers, React.memo + useMemo, Scheduler API.

How do structured data and Schema.org improve search visibility?

Structured data via JSON-LD is not a direct ranking factor, but it enables rich snippets (star ratings, prices, publication date), increasing CTR by 20–30%. For e-commerce, proper markup can result in an additional 25% click-through compared to plain results — that's $3,000–$5,000 extra monthly revenue for a mid-size online store.

Markup types by scenario:

• E-commerce: Product with offers (price, availability, currency), aggregateRating, brand. BreadcrumbList, ItemList.
• Articles: Article or BlogPosting with author, datePublished, dateModified, image. Organization and WebSite.
• Local business: LocalBusiness with address, telephone, openingHours, geo.
• FAQ: FAQPage with mainEntity — questions appear as expandable block.

Validation: Google Rich Results Test, Schema Markup Validator. Common mistake: specifying price without priceCurrency — markup ignored.

How to conduct a technical SEO audit

Crawlability. robots.txt blocks necessary pages or doesn't block service pages. Canonical URLs incorrectly set — duplicates with UTM parameters. Sitemap contains noindex pages. Tools like Screaming Frog or Sitebulb show this in an hour.

Core Web Vitals at scale. Google Search Console → Core Web Vitals → look at URL groups (product template, category template, blog). Problem is usually systemic.

JavaScript SEO. Google renders JS with delay. For critical content, SSR or SSG are mandatory. Check via Search Console → Inspect URL → View Crawled Page.

Internal linking. Orphan pages lose PageRank. Broken links (404) are a quality signal.

Common mistakes when implementing Schema.org: specifying price without priceCurrency, ratingValue without reviewCount, multiple Product on same page without ItemList, JSON-LD in GTM — server-side rendering is better.

What does the optimization process look like?

What deliverables do you receive?

• Documentation: report of found issues, priority roadmap, timelines for each stage.
• Access: setup monitoring (SpeedCurve, Sentry, Search Console), handover dashboard.
• Training: one or two calls reviewing typical mistakes for your team.
• Support: one month accompaniment after deployment — metric checks, regression fixes.

How many positions can you regain through technical SEO?

We have 5+ years on the market and 150+ projects completed. For a case study: a SaaS platform with 200k monthly visits had LCP 6.2s, CLS 0.45, INP 600ms. After optimization, LCP dropped to 1.8s, CLS to 0.02, INP to 180ms. Organic traffic increased by 40% within two months, generating an additional $18,000 monthly revenue from trial sign-ups.

Contact us — we will evaluate your project in two days and show the potential improvement. Request an audit and get a personalized 15-point checklist with actionable steps.