Crawl budget

Click for sound

“Fresh index, good coverage, not get blocked. Choose two.”
– Dmytro Gerasymenko, Founder & CEO of Ahrefs

When building a big index, there’s always a trade-off between freshness and good coverage.
Freshness implies running regular crawls to keep information up-to-date. Good coverage implies crawling as many pages as possible. Yet, you can’t have both running at full capacity, otherwise you’d get blocked by webmasters and hosting companies.

The answer to this is implementing a crawl budget, which refers to the number of URLs a crawler can and wants to crawl.

Crawl budget is composed of two parts: crawl rate and crawl demand.

Crawl rate

Crawl rate refers to the number of requests a crawler can make to a site when crawling it.

Crawling a website too fast can add too much load to a server. Since this can lead to poor user experience or result in our crawler getting blocked, our crawl rate takes into account:

• Page speed – Faster-loading pages are preferred to slower-loading ones.
• Website size – Small websites with high-quality links will most likely be crawled in full as compared to larger websites with low-quality links that might get only partially crawled.

Crawl demand

Crawl demand, or crawl priority, represents the level of importance attached to crawling and recrawling pages on a website.

This is done by our scheduler, which determines the crawl demand by:

• URL popularity (URL Rating) – The higher the quality of the backlinks pointing to a page, the higher the priority.
• Website popularity (Domain Rating) – The higher the strength of a website’s backlink profile, the higher the priority.

KEEP LEARNING

Check out some of the articles on the Ahrefs blog to learn about crawl budget:

• What is Crawl Budget?
• When Should You Worry About Crawl Budget?
• Crawl Me Maybe? How Website Crawlers Work

About this course How to Use Ahrefs

62 lessons

3h 18m

Learn practical ways to use Ahrefs' SEO tools and reports to improve SEO.

What you’ll learn

• How to analyze yours and your competitors’ websites with Site Explorer
• How to increase your AI visibility in LLMs with Brand Radar
• How to master keyword research with Keywords Explorer
• How to improve your on-page and technical SEO with Site Audit
• How to track and improve your Google rankings with Rank Tracker
• How to find untapped keyword and link building opportunities with Content Explorer

Course by

Andrei Țiț
Andrei Țiț is a product marketer at Ahrefs. He develops content to help you make the most out of Ahrefs.

Sam Oh
Sam Oh is VP of Marketing at Ahrefs. He incorporates his commitment to education and love for entrepreneurship into actionable and easy-to-digest tutorials.

Certification Exam67 questions

Take the course exam to test what you’ve learned. Pass it and you’ll get a personal certificate signed by our educators and Ahrefs.

Take exam

An Ahrefs account is required to take this exam. Sign up for an Ahrefs Free account or paid subscription if you don’t have one.